Measuring Excellence: 

Planning and Managing Evaluations 

of Law Enforcement Initiatives

U.S. Department of Justice

Office of Community Oriented Policing Services

Kristin Ward | Susan Chibnall | Robyn Harris
ICF International, a global professional services firm, 

partners with government and commercial clients to deliver 

consulting services and technology solutions in energy, 

climate change, environment, transportation, social programs, 

health, defense, and emergency management. In October 

2005, ICF acquired Caliber Associates, an organization that 

provides full life-cycle program consulting and support 

in research and evaluation, communications, training and 

technical assistance, and information technology. Together, 

ICF and Caliber have expanded capabilities to assist both 

civilian and national security organizations to develop more 

effective and innovative human resource, education, and 

social service programs.

Prior to the acquisition, Caliber had been awarded a 2-year 

contract from the U.S. Department of Justice Office of 

Community Oriented Policing Services (the COPS Office) 

to develop Measuring Excellence: Planning and Managing 

Evaluations of Law Enforcement Initiatives. Tailored specifically 

for the law enforcement community, the guide emphasizes 

the importance of program evaluation to law enforcement. 

It is in a user-friendly format that assists law enforcement 

administrators throughout the evaluation management 

process—from the initial stages of thinking about and 

planning a program evaluation, through selecting, designing, 

and implementing program evaluations, and using the results 

to meet the needs of their communities and stakeholders. 
Kristin Ward | Susan Chibnall | Robyn Harris

June 2007

This project was supported by cooperative agreement 

#2003-CK-WX-K052 awarded by the Office of 

Community Oriented Policing Services, U.S. Department 

of Justice. The opinions contained herein are those of 

the authors and do not necessarily represent the official 

position of the U.S. Department of Justice. References 

to specific companies, products, or services should not 

be considered an endorsement by the authors or the 

U.S. Department of Justice. Rather, the references are 

illustrations to supplement discussion of the issues.

Measuring Excellence: 

Planning and Managing 

Evaluations of Law 

Enforcement Initiatives

iii

Table Of Contents

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Chapter I: Why Evaluate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Purpose of the Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Why Evaluate? The Rationale for Program Evaluation . . . . . . . . . . . . . . 3

Real-World Examples of Evaluation in Policing . . . . . . . . . . . . . . . . . . 10

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter II: Key Concepts in Evaluation . . . . . . . . . . . . . . . . . . 15

The Different Types of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Evaluation Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

The Continuous Feedback Loop of Program Planning and

Evaluation and the Key Steps in the Evaluation Process . . . . . . . . . . . . 40

Chapter Summar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Chapter III: Conducting an Internal Evaluation . . . . . . . . . . 45 

Steps in the Evaluation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Step One: Specify—Evaluation Planning . . . . . . . . . . . . . . . . . . . . . . . 50

Step Two: Collect—Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . 58

Step Three: Compare—Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 69

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Chapter IV: Managing Evaluations . . . . . . . . . . . . . . . . . . . . . 77

How Do I Manage an Evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

What Do I Need to Do During the Evaluation? . . . . . . . . . . . . . . . . . 88

What Do I Need to Do After the Evaluation? . . . . . . . . . . . . . . . . . . . 89

Challenges to Managing Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Should I Hire an External Evaluator? . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Chapter V: Emerging Topics in Evaluation Management 101

Collaborative Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Participatory Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Performance Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
iv

Exhibit I-1: Six Myths About Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 6

Exhibit II-1: Examples of Questions Addressed in a 

Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Exhibit II-2: Examples of Questions Addressed in a 

Process Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Exhibit II-3: Examples of Questions Addressed in an 

Outcome Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Exhibit II-4: Examples of Questions Addressed in an 

Impact Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Exhibit II-5: Examples of Questions Addressed in a 

Cost-Benefit Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Exhibit II-6: Overview of Evaluation Types . . . . . . . . . . . . . . . . . . . . . 29

Exhibit II-7: Strengths and Weaknesses of Different 

Types of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Exhibit II-8: Common Threats to the Validity of 

Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Exhibit II-9: The Program Planning and Implementation Cycle . . . . . . 40

Exhibit II-10: The Evaluation Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Exhibit II-11: Integration of the Planning and Evaluation Cycles . . . . . 42

Exhibit III-1: Difference Between Academic Research and 

Internal Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Exhibit III-2: Tips for Developing Goals and Objectives . . . . . . . . . . . 54

Exhibit III-3: Sample Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Exhibit III-4: Tips for Developing Surveys and Survey Questions . . . . 62

Exhibit III-5: Tips for Conducting Surveys . . . . . . . . . . . . . . . . . . . . . 64

Exhibit III-6: Tips for Planning and Facilitating Focus Groups . . . . . . 66

Exhibit III-7: Data Collection Methods at a Glance . . . . . . . . . . . . . . . 67

Exhibit III-8: Options for Organizing and Reporting 

Qualitative Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Exhibits
Appendixes

Appendix A: Sample Needs Assessment Action Plan Template . . . . . . 117

Appendix B: Preparing Measurable Goals and Objectives . . . . . . . . . . 118

Appendix C: Planning Model Template . . . . . . . . . . . . . . . . . . . . . . . 121

Appendix D: Community Satisfaction Survey . . . . . . . . . . . . . . . . . . 123

Appendix E: Sample Focus Group Checklist . . . . . . . . . . . . . . . . . . . 126

Appendix F: Instrument Development Checklist . . . . . . . . . . . . . . . . 128

Appendix G: Topics to Cover in an Informed Consent Statement 

for Research Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Appendix H: Sample Evaluation Plan Template . . . . . . . . . . . . . . . . . 132

Appendix I: Sample Interview Questions Checklist . . . . . . . . . . . . . . 133

Appendix J: Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Appendix K: Sample Job Announcement . . . . . . . . . . . . . . . . . . . . . . 139

Appendix L: Evaluation Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Exhibit IV-1: Assessing Evaluator Capabilities . . . . . . . . . . . . . . . . . . . 94

Exhibit IV-2: Advantages of Hiring an External Evaluator 

Compared to Using Internal Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Exhibit IV-3: Sources for Finding an External Evaluator . . . . . . . . . . . . 98

Exhibit V-1: Difference between Participatory Evaluation 

and Traditional Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Exhibit V-2: Differences between Performance Measurement 

and Program Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

Exhibits continued

Measuring Excellence vii

Foreword

The primary purpose of this guide is to provide you with information for conducting and 

managing program evaluations in your law enforcement agency. However, during the 

needs assessment conducted to develop this guide, during which input was solicited from 

law enforcement personnel nationwide through focus groups and telephone interviews, it 

became clear that many police administrators and managers would benefit from an enhanced 

understanding of evaluation. From the needs assessment, five key factors arose that shaped the 

development of this guide:

Many of you would benefit from exploring the value, importance, and potential uses 

of evaluation information • Chapter I: Why Evaluate? 

Many of you also would benefit from learning more about evaluation basics, so you 

could better manage and conduct a program evaluation • Chapter II: Key Concepts 

in Evaluation. 

In many law enforcement agencies, the person responsible for managing evaluation 

activities also conducts them • Chapter III: Conducting an Internal Evaluation. 

Law enforcement agencies around the country are conducting their own program 

evaluations of program processes, outcomes, and impacts. In addition, for large or 

complex community initiatives, law enforcement agencies are conducting external 

evaluations with the help of evaluation experts outside their department. Many of 

you could use tips on managing both types of evaluative endeavors • Chapter IV: 

Managing Evaluations. 

Law enforcement agencies are increasingly engaging in new types of evaluation 

methodologies, including participatory evaluations, collaborative evaluations, 

and performance measurement activities (i.e., monitoring and reporting program 

accomplishments) • Chapter V: Emerging Topics in Evaluation Management.

This guide presents the basic concepts and guidance for planning, implementing, and 

managing an evaluation, including understanding evaluation concepts and activities. 

Where more information might be needed or desired, an appendix is provided with references 

to other, more detailed, sources. 

This guide is designed to help law enforcement agencies that plan to begin conducting evaluations 

as well as those that want to refine and improve their evaluation management techniques. 

The authors thank Bonnie Bucqueroux for reviewing the report. Through her suggestions, she 

made important contributions to the work.

1.

2.

3.

4.

5.

Chapter I: 

Why Evaluate?

Measuring Excellence 

Why Evaluate?

Purpose of the Guide

Modern policing continues to place new demands on police leadership. Policing 

today not only addresses crime, fear of crime, and social and physical disorder, 

but also requires the police to be involved in collaborative problem solving—the 

only government agency patrolling neighborhoods around the clock. 

As the scope of police work broadens, so too does its measurement, extending 

beyond numbers of arrests made and response time and making defining success 

an increasingly complex challenge. Whose views matter? The police themselves? 

Elected officials, the business community, schools, the faith community, 

the media, average citizens? Particularly in an environment of strict budget 

considerations, police are compelled to move beyond mere opinion and justify 

their work through comprehensive, quality evaluations. 

Persuading police to embrace evaluation prompts the question, “What’s in it for 

us?” Also, for evaluation to gain widespread acceptance, police administrators 

need the tools to conduct evaluations and do them well. Therefore, this guide 

begins with an explanation of why evaluation must become an integral part of 

police work, especially as police administrators repeatedly find themselves asked 

to do more with less. 

The guide also provides practical information and advice on conducting quality 

evaluations that recognize the unique challenges inherent in police work—

evaluations that can withstand rigorous scrutiny and help build community trust 

and support. This guide is a user-friendly resource that is supplemented with 

appendixes that take you through several practical exercises and direct you to 

more detailed information from other sources.

Why Evaluate? The Rationale for Program Evaluation

There are many kinds of evaluation, but the most common is program evaluation, 

in which the goal is to discover how well a particular program is working. 

Program evaluation is defined as a systematic process of gathering and analyzing 

information for the purposes of program assessment, program improvement, 

and, in a broader sense, strategic management. It provides a structured way to 

verify, document, and quantify program activities and their effects. 

Done well, evaluation provides meaningful results to help you make informed 

judgments about the effectiveness and efficiency of your strategies and programs. 
 Why Evaluate?

You should be able to demonstrate that you are committing resources to efforts 

that produce tangible benefits for the entire community. Evaluation can help you 

in these ways:

Compare alternatives. What works best? Framed correctly, evaluation can let 

you compare programs and strategies so you can prioritize where to put scarce 

resources.

Determine appropriateness. Is the program the right way to address the problem? 

Does it enjoy not only the support but also the participation of the community?

Policing is about both means and ends and evaluation can help you determine 

whether the program is the best and most inclusive way to deal with the problem. 

Clarify program objectives. What are you trying to accomplish? How will you 

define success? A well-planned evaluation requires you to clarify assumptions 

about the links between your target population, the program activities, and the 

immediate, intermediate, and long-term outcomes you expect. There is wisdom in 

the saying: “If you don’t know where you are going, any path will take you there.” 

Evaluation helps you decide where you are going and how to determine whether 

you are taking the best path to get there. 

Advocate for your efforts. How can you energize supporters and persuade critics 

to come on board? Perhaps the biggest virtue of an evaluation is that it has 

more validity than mere opinion. It is easy to say that you know something is 

working and provide anecdotes that support that view. However, as we grow 

more sophisticated in our decision-making, it becomes increasingly important to 

conduct outcome-based evaluations.

Describe the program and educate the community. Do key stakeholders 

understand the initiative and how it works? Do average citizens know what the police 

are doing and why? In the rush to do good things, police often fail to explain their 

actions. Evaluation can require documenting not only the program but also the 

underlying philosophy and the results. People outside police circles often believe 

in the myth of the easy answer or the quick fix to crime and social disorder 

problems. A thoughtful evaluation can help stakeholders and citizens understand 

better the challenges of policing.

Solidify support. How can we keep supporters engaged? Positive results from a 

well-planned evaluation can be used to justify a program’s existence, maintain 

commitments from program administrators or the community, and leverage 

additional resources from funders, community partners, and other stakeholders.

Address program cost. Are you operating within budget? Are there opportunities 

for cost savings? Do the results justify the investment? Evaluation is a process 

that provides feedback to help you make adjustments and can give valuable 

information about ways to save money and whether the program delivered 

sufficient “bang for the buck.”
Measuring Excellence 

Make informed program improvements. Do you need to make midcourse 

corrections? Are there changes that should be made before the program is allowed to 

continue or is replicated elsewhere? The information collected during an evaluation 

can provide the information you need to make intelligent changes likely to 

improve opportunities for success.

Monitor program integrity. Are you really doing what you said you would do? 

Tracking the number and type of activities you offer, the number and type of 

participants involved, and your activity-related expenses can help determine if you 

actually are implementing the program as intended and promised.

Provide a blueprint for peers. How will newcomers learn what to do? Will others 

inside the agency be able to implement the program or strategy? A quality evaluation 

provides a road map for others to follow, with the potential barriers and pitfalls 

identified.

Recognize and capture unanticipated or unintended program 

effects. Were there “side effects” that need to be documented and understood?

Evaluation often uncovers positive or negative program effects that were neither 

anticipated nor intended—consequences that can be lost if no evaluation process 

is in place.

Inform internal systems. Can the findings be used to improve functioning inside 

the department? Evaluation also can be used more holistically and broadly as a 

tool for internal strategic planning and management. If, for example, you find 

that a new initiative’s success correlates to the communication skills of the officers 

implementing the program, this has implications for your human resources 

department and its recruiting guidelines. 

Assess overall effectiveness. Did the program or effort work as intended? Data 

collected during an evaluation will help you document and demonstrate what is 

working and what is not. 

Pilot test innovations. How can we identify new ideas for the future? Policing must 

never be static and managers must be willing to take reasonable risks on new 

ideas. But the challenge is to conduct quality evaluations to determine whether 

promising ideas might pan out in practice.

Contribute to the field. Does this evaluation help our understanding of future 

directions? The lessons that you learn from your evaluation can be shared with 

other law enforcement agencies and communities. Sharing evaluation findings 

regularly will contribute to creating a much-needed body of verifiable law 

enforcement knowledge.
 Why Evaluate?

Exhibit I-1: 

Six Myths About Evaluation

Evaluation in the Law Enforcement Context

Formal evaluation is rarely, if ever, required to maintain general funding of police 

department operations. Often, it is not even required of state and federal law 

enforcement grants. Consequently, many police chiefs and other law enforcement 

administrators often do not make evaluation a priority. In part, this reluctance 

to integrate and implement evaluation might stem from not wanting to invest in 

learning bad news. Common myths and misconceptions about evaluation, as well 

as a lack of understanding about the place of evaluation in the policing context, 

play a role in the scarcity of evaluation activity in police departments across the 

country. Six myths about evaluation are presented in Exhibit I-1.

The Myth The Reality

Police leaders don’t value 

evaluation; lack of internal 

support.

Today’s police leaders must be able to prove that they are 

spending tax dollars wisely. Many police departments today 

are conducting and managing meaningful evaluations of 

their efforts.

Police departments do not 

need to do evaluation (not 

required by law or grants).

Quality evaluations are for YOU, not THEM. Evaluation 

provides a framework that assists you in doing the best job 

possible within existing resources. 

Police departments lack 

the resources to do proper 

or meaningful evaluation 

studies.

Program evaluation need not be costly or time-consuming. 

Valuable results can be obtained by asking targeted 

questions or using existing data. Do what you can with what 

you have now and build from there.

Police staff lack the skills 

for evaluation design and 

analysis.

Program evaluation is not just for academics. The best 

evaluations are based on a common-sense approach to 

gathering and analyzing information.

Police departments lack 

relevant data to evaluate 

their efforts.

The data will be there if you weave your evaluation strategy 

into program planning. If you did not capture appropriate 

data this time, make sure to craft a better plan for the future.

Police departments fear 

negative results.

The politics of policing can make top command skittish 

about investing in efforts that may generate bad news. But 

evaluation is really about improving police effectiveness 

and prioritizing scarce resources. An evaluation is not a 

“gotcha,” but a blueprint for improvement.
Measuring Excellence 

Evaluation informs spending decision

In addition to the reasons listed in Exhibit I-1, police officials need to embrace 

evaluation to maintain public trust. As stewards of public funds, police departments 

are accountable for ensuring that taxpayer dollars are used appropriately, effectively, 

and efficiently. Program evaluation not only addresses those concerns, but also can 

provide a cost-benefit analysis, which is a tool that can help administrators decide 

where to invest limited funds. While there can be risks involved in evaluating 

programs and strategies (e.g., discovering that a popular 

or expensive strategy is not working), most administrators 

would rather know the truth as soon as possible. With that 

knowledge, decision-makers can make adjustments based 

on informed feedback, or abandon a misguided effort before 

more money is wasted.

Evaluation supports community policing and 

problem-oriented policing

Program evaluation is essential for, and compatible with, cutting-edge policing 

approaches such as community policing and problem-oriented policing. 

For instance, those familiar with the SARA (Scanning, Analysis, Response—and 

Assessment) model will recognize that evaluation is an integral component. 

Problem-oriented policing requires recognition (scanning) and analysis of a problem, 

a strategically planned response to address it, and an assessment (i.e., an evaluation) of 

how well the strategy solved or reduced it. Evaluations—especially those that include 

stakeholder participation—can further the objectives of community policing. By using 

strategies that capture and analyze findings from the community and participating 

professionals, evaluation offers insights into opportunities for enhancement and 

improvement. Also, evaluating community-based initiatives with the help of citizenparticipants can bolster credibility, buy-in, partnership, and accountability, while 

contributing to long-term citizen satisfaction and neighborhood development and 

improvement.

Moreover, law enforcement already is data-driven. In fact, it could be said that 

informal evaluation occurs all the time, as police administrators consult a variety 

of data to make decisions—crime data, calls for service, clearance rates, discussions 

with officers. The challenge is to move from an informal approach that might not 

be rigorous or comprehensive, or that has gaps, to a more 

informed and institutionalized process that contributes 

to better decision-making. As problem-oriented policing 

and community policing increasingly become the ways in 

which most police agencies deliver service, comprehensive 

evaluation provides a way to move beyond the narrow 

and incomplete measures of police performance of the 

past, such as crime rates and response times. This guide 

provides both a rationale for quality evaluation and the 

tools you will need to do the job.

“We need to evaluate the 

short-term temporary gains 

associated with reactive policing 

so that we can effectively plan 

for long-term neighborhood 

improvements.” 

– Chief Harry Dolan, Grand Rapids (Minnesota) 

Police Department

“What you don’t know CAN hurt 

you. Evaluation lets you know 

what works and what doesn’t.” 

– Chief Tony Kleibecker, Muskegon 

(Michigan) Police Department
 Why Evaluate?

Examples of How Law Enforcement Agencies Nationwide

Use Evaluation

Despite the obstacles to conducting formal evaluation in the policing context, 

many police agencies are successfully incorporating program evaluation into their 

daily practice. The Community Policing Consortium offered advice regarding 

what to evaluate in the policing context: 

Internal structures and systems: 

Departmental organization

Recruiting

Training

Performance evaluation

Promotion

Formal and informal rewards

Internal communication

Morale

Strategic plans:

Assess the process of developing a strategic plan

Monitor implementation of the strategic plan

Resource allocation: 

Assess the decision-making process for resource allocation

Monitor resources in terms of what is being spent, where, why, and 

whether money could be spent more wisely

Pilot projects: 

Determine the benefits and drawbacks of wider implementation

Effectiveness of community policing efforts/strategies

Assess outcomes from the perspectives of different stakeholders

Quality of police services:

Customer satisfaction

Community perceptions of the police

(Community Policing Consortium Resources: Strategic Planning for Community 

Policing, n.d.)

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

°

“We don’t have 

enough resources, 

time, or energy to 

waste on things that 

don’t work ... How 

can we prove that 

the millions of dollars 

spent on us are 

actually having some 

effect?”

– Chief Jim Bueerman, 

Redlands (California) Police 

Department 
Measuring Excellence 

Evaluations vary from large-scale, expensive, external, and experimental efforts to 

small-scale, low-budget, internal assessments of local problem-solving initiatives. 

Their purposes range from discrete, one-time evaluations of specific activities to 

using evaluation as a long-term strategic development and management tool. 

The following are examples of successful evaluations that law enforcement 

agencies across the country have conducted:

Evaluation of the effectiveness of a blended housing and recreation 

initiative to lower risk factors for, and incidence of, crime in 

neighborhoods, using a citywide experimental design (Redlands, 

California: 2004).

Evaluation of a 12-officer bike patrol program, using neighborhood 

satisfaction surveys and process/implementation evaluation to understand 

whether the program was being implemented as intended and whether 

funds were well spent (Muskegon, Michigan: 2003).

Evaluation of the effectiveness of multidisciplinary collaboration among 

city groups and agencies to identify drug dealers and increase the number 

of narcotics raids (Muskegon, Michigan: 2003). 

Evaluation of youth violence-prevention efforts (e.g., partnerships 

between police departments and schools to prevent and decrease 

bullying), using pre- and post-program surveys with students, parents, 

and teachers, as well as observations (Chula Vista, California: 2003).

Evaluation of collaborative efforts to reduce residential burglaries in a 

new housing development area, using interviews with victims, street-view 

environmental assessments, review of incident reports, and tracking and 

comparing trends in theft data in various neighborhoods 

(Chula Vista, California: 2003).

Evaluation of efforts to address disorder in budget motels using 

pre- and post-program interviews with motel managers (Chula Vista, 

California: 2003).

°

°

°

°

°

°
10 Why Evaluate?

Real-World Examples of Evaluation in Policing

This guide will discuss how you can manage and use evaluation to measure and 

improve programs and direct your strategic vision. Throughout the guide, we will 

reference real-world examples, describing how your colleagues in law enforcement 

have planned, implemented, and ultimately learned from evaluation in their 

work. We begin in this chapter with two examples that illustrate the range of 

evaluation activity and purpose in the policing context. We hope they will inspire 

and encourage you to learn more about what your department can achieve 

through evaluation.

The first example is the Minneapolis Police Department’s Hawthorne Huddle, 

which clarifies how evaluation can help you gauge the effectiveness of community 

policing strategies and collaborations with community agencies and citizens to 

improve neighborhoods. The Hawthorne Huddle also illustrates how evaluation 

can assist in improving or complementing the strategies used to build police/

community partnerships. 

The second case study, the Colorado Springs Police Department’s Police 

Accountability and Service Standards (PASS) model, exemplifies how evaluation 

can be utilized for developing and assessing the strategic vision, goals, and 

objectives of law enforcement agencies.
Measuring Excellence 11

The Hawthorne Huddle

A collaborative and action-oriented program evaluation model

Conditions in the Hawthorne neighborhood of North Minneapolis declined into the 

mid-1990s. The area was plagued by high crime, poor rental property management, 

and a deteriorating housing stock. Drug dealing and other crime contributed to 

the disorder in the neighborhood, and residents felt fearful, disenfranchised, and 

abandoned by what they perceived as a lack of effort on the part of city officials to 

improve the situation. Police responsible for patrolling the area felt frustrated by the 

fact that they saw no sign of the bad times abating in the area. 

The Minneapolis Police Department (MPD) decided to collaborate with other 

city and county agencies, nonprofit organizations, a major corporate sponsor 

(the General Mills Foundation), and community residents to develop a plan to 

deal with crime and disorder in the area. MPD used crime mapping and analysis 

of police reports to define and target the biggest problems in the area. Next, 

they organized neighborhood meetings and encouraged citizen participation 

in block clubs to garner support and assistance with problem-solving efforts. 

Residents quickly accepted the block club meetings, which grew over time. To 

this day, agency and community members gather at monthly meetings called the 

Hawthorne Huddle.

Additional strategies were aimed at reducing the number of criminal narcotics 

arrests at specific addresses, as well as the number of vandalism incidents and 

burglaries in the neighborhood. To assess the effectiveness of these strategies, 

several evaluation techniques were ultimately used. 

Evaluation strategies were collaborative and participatory in that MPD was not 

solely, or even predominantly, responsible for evaluation of the Hawthorne Huddle. 

Instead, the agencies involved in the project worked together to define goals and 

objectives and decided how success would be measured. Participating agencies and 

organizations volunteered to collect various kinds of relevant data. MPD gathered 

and analyzed crime statistics for the area while other organizations conducted 

surveys and focus groups to measure residents’ feelings of safety and satisfaction, 

and gauge ongoing concerns. This collaboration helped keep Hawthorne Huddle 

program activities grounded in real community issues and concerns.

This collaborative and action-oriented evaluation model relies on stakeholders 

working together to identify the problem, develop goals and objectives, define 

success, decide how and what to measure, and determine how to use results 

to continue improving the community. Evaluation results are released to the 

community at monthly Huddle meetings. Stakeholders use the information to 

make decisions about future efforts (reflecting the values of community policing 

and its emphasis on building connections to local leaders), work together to identify 

community needs, and leverage evaluation results to improve the community.
12 Why Evaluate?

The Police Accountability and Service 

Standards (PASS) Model

A strategic process, program, and performance evaluation model

Under the leadership of Chief Luis Velez, the Colorado Springs Police 

Department (CSPD) developed the PASS model as a structured way to gauge the 

department’s progress in infusing the community policing philosophy throughout 

the organization. PASS is an ambitious effort that includes a commitment 

to evaluate how new programs are designed and implemented, recognizing 

that community policing cares about means and ends. It includes qualitative 

evaluation strategies that go beyond traditional quantitative measures of police 

activity such as response time and numbers of arrests. PASS serves as a model of 

how police agencies can develop a comprehensive evaluation strategy to ensure 

that all police activities reflect the philosophy, vision, mission, and values of the 

department. 

Colorado Springs is one of a handful of jurisdictions nationwide that have 

designed a prototype for identifying and measuring police service standards. The 

PASS model is a holistic strategic planning and evaluation design that requires an 

annual assessment of the past year’s performance, development of a strategic plan 

in partnership with citizens, implementation of that strategic vision, measurement 

of both processes and outcomes associated with police service delivery, assessment 

of overall organizational performance, and demonstration of police accountability 

and stewardship to community. It involves conducting individual, unit, and 

division performance evaluations to enhance the alignment of goals and objectives 

throughout the police organization (which are linked to the city’s strategic plan). 

Evaluation expert Mora L. Fiedler assisted in developing the user-friendly PASS 

model, which calls for collecting both process and outcome data. Analysis of 

process data answers questions about how efforts are conducted, with what 

resources, with whom, and how the police department partners with others in 

the community. Outcome data are collected to measure the effectiveness of the 

service standards that have been defined with citizen input to the PASS process. 

For example, CSPD collects measurement data about response times, officer 

deployment, traffic, drug and vice activity, neighborhood policing, and citizen 

satisfaction with police services. CSPD follows various evaluation methods to 

collect and analyze these data. Examples will be cited throughout this guide. 

Source: Fiedler, M. The Colorado Springs Police Department’s police accountability and service standard (PASS) 

model. Colorado Springs: Management Services Division, Planning Section, Colorado Springs Police 

Department, 2002.
Measuring Excellence 13

Chapter Summary

Evaluation is important in modern policing, particularly in light of limited 

budgets and the adoption of problem-oriented policing and community policing 

strategies. Police leaders need to know what works and what does not, both inside 

and outside of the department. This chapter outlined the benefits of evaluation 

and discussed several common myths that can prevent law enforcement agencies 

from pursuing evaluations. However, despite both real and imagined barriers, 

police departments nationwide are beginning to incorporate evaluation into their 

practice. Suggested areas for evaluation within policing and several contemporary 

examples of successful evaluations in United States police departments were 

presented. 

Evaluation in law enforcement holds tremendous potential and is not a dangerous 

process designed to document failure or generate bad news. Instead, it is a 

valuable tool that agencies can use to enhance, prioritize, and justify activities. 

The following chapters go further in dispelling the myths that evaluation is too 

demanding, technical, and expensive for police departments to implement. This 

guide will help demystify the process of program evaluation and will provide 

practical advice for starting or enhancing evaluation activities. Refer to this guide 

often for ideas and inspiration. 

Chapter II: 

Key Concepts in Evaluation

Measuring Excellence 17

Key Concepts in Evaluation

The previous chapter explained the reasons why program evaluation is pertinent 

to policing and how police departments across the country have used evaluation 

to improve program and agency performance. Grasping the fundamentals of 

evaluation requires learning the terminology, understanding the purposes and 

requirements of various models, and thinking through how the advantages and 

disadvantages of certain approaches apply in specific situations. 

This chapter introduces several key concepts, with which you should be familiar, 

to effectively manage evaluations in your police department. The different types of 

evaluations are defined, followed by an overview of common evaluation designs. 

Information about evaluation types and designs can help you understand how to 

select the best and most appropriate approach for your purposes. Different types of 

evaluation data that can be collected, the benefits and drawbacks of each, and the 

importance of data validity and reliability also are discussed. The chapter concludes 

by introducing the continuous feedback loop of program planning and evaluation, 

and outlining key steps in the evaluation process.

Managing or conducting evaluations in your police department requires a 

basic knowledge of evaluation types, designs, and methodologies and a general 

understanding of what certain types of evaluation can and cannot accomplish. This 

is not to say that you have to be an expert, know how to conduct such evaluations 

yourself, or that evaluations have to be complicated or academic. Rather, conducting 

and managing practical evaluations successfully depends on keeping the process 

organized, of reasonable scope, and directly related to evaluative questions. In other 

words, do not try to do too much and keep things straightforward and realistic. This 

section will educate you about the core concepts and strategies of evaluative inquiry.

The Different Types of Evaluation

Different types of evaluation will be more or less applicable, depending on your 

evaluation questions and needs. The type of evaluation you are going to manage 

or conduct will guide your design and the data you need to collect. 

The first step in deciding which type of evaluation is best suited for your needs is 

to ask yourself what it is you want to know as a result of the evaluation. Generally, 

you can ask one of these four questions: 

What is the nature and extent of the problem of interest? 

What is my program doing? 

Is my program achieving its goals and objectives for program participants?

Are our efforts effecting change in the population in which the problem 

initially documented?

The question you choose will direct you to data-collection methods and sources 

that can determine the answer.

1.

2.

3.

4.
18 Key Concepts in Evaluation

There are four main types of evaluative inquiry: needs assessment, process 

evaluation, outcome evaluation, and impact evaluation. The first is associated 

with assessing the nature and extent of the problems that your community (or 

geographic area or jurisdiction) is experiencing. The second focuses on improving 

a program and its activities, and the latter two are linked more closely with 

proving that a program works (or achieves the desired and anticipated results). 

Following are brief descriptions of the four kinds of evaluation as well as a 

cost analysis. Each is illustrated by an example that shows how these kinds 

of evaluations could be applied to a new police department initiative aimed 

at domestic violence. We chose a domestic violence example because this is a 

complex and often hidden crime, and one that demands a sophisticated and 

comprehensive response. Showing how evaluations could be constructed to 

deal with these challenges highlights the options that can be applied to a broad 

spectrum of situations. 

Needs Assessment 

This type of evaluation helps an agency, organization, or coalition determine the 

nature and extent of specific problems in the community and how a problem 

is perceived among diverse groups. A careful, thorough needs assessment is 

the road map for change that provides the basis for a strategic plan to address 

specific problems in the community. Needs assessment includes reviewing local 

archival data to establish baseline measures of problem behaviors (e.g., breakins, vandalism, violent crime) and comparing those data to state or national 

data to determine the extent to which a community’s problems are above or 

below state or national levels. Data from surrounding communities also can be 

used for comparison. Needs assessments also should include data from surveys 

and questionnaires and key informant interviews. The goal is to prioritize those 

behaviors that are most problematic in your community and make them the focus 

of your efforts. Examples of questions addressed in a needs assessment are shown 

in Exhibit II-1.
Measuring Excellence 19

Domestic Violence Needs Assessment

A well-designed and comprehensive needs assessment that focuses on domestic 

violence in your jurisdiction could include gathering the following information:

Determine the extent of the problem. You can use official police 

department data (calls for service, arrests) to establish a baseline to 

monitor over time. Also work with prosecutors, courts, and corrections 

to gather data on how many cases go to trial, average sentences, and 

number of repeat offenders. Although only a fraction of domestic 

violence incidents are reported to police, a thorough needs assessment 

should explore other strategies to get a truer picture of the problem. Such 

strategies might include gathering data from focus groups, surveys, and 

interviews of beat cops who are closest to the community, professionals 

and volunteers who work with victims at community hot lines, personnel 

at domestic violence shelters and hospital emergency rooms, and those 

who work with batterers. Strategies also might include targeted outreach 

to professional women’s groups because women from middle- and 

upper-income families often are more reluctant to report problems 

to police. Other data that can aid in understanding the extent of the 

problem would be total deaths and injuries, hospital costs, and workdays 

lost that are attributable to domestic violence. 

Compare rates to other jurisdictions. In addition to comparing 

your data to state and local statistics, you can compare them to nearby 

jurisdictions.

°

°

Exhibit II-1

Examples of Questions Addressed in a Needs 

Assessment

What is the nature and extent of the problem(s) in our community? 

How does our community compare with other local communities 

and state and national rates of the problem? 

According to the data, which behaviors are most problematic in our 

community?

What resources are already in place to deal with these problems? 

What strategies or activities are available that have demonstrated 

effectiveness in changing these problem behaviors? 

What is the scope of our efforts to work on this problem? 

°

°

°

°

°

°
20 Key Concepts in Evaluation

Assess trends over time. Do the data show whether the problem is 

getting better or worse? What factors could be influencing those changes 

over time? 

Explore the underlying dynamics. Domestic violence advocates 

might have demographic data about victims and offenders, such as 

age, education, income, cultural and language barriers, and geographic 

incidence. You may also want to gather data on the prevalence of 

substance abuse, including alcohol and illicit drugs. In addition to 

statistics, interviews with experts will increase understanding of prevailing 

theories, such as how the cycle of power and control results in eruptions 

of domestic violence and why victims are often so reluctant to leave. 

Individual focus groups with victims and offenders also could paint a 

better portrait of the problem of domestic violence in your community. 

Describe current responses. Document ongoing police and community 

efforts to deal with the problem, analyzing gaps, shortfalls, and obstacles 

to success. This step allows you to document the policies, practices, 

and procedures within the department that influence or hinder police 

response to domestic violence, and the prevailing laws that might dictate 

arrest policies.

Identify potential partners. In looking for data and candidates to 

interview or involve in focus groups, you are likely to identify community 

members who might be willing to work with you to implement solutions. 

Some to approach are domestic violence shelters, women’s groups, 

schools, the faith community, hospitals, physicians and nurses groups, 

and the business community (who suffer economic consequences 

associated with lost employee productivity of domestic violence victims). 

Gather information about opportunities to make a positive 

difference. While a needs assessment is not designed to explore solutions, 

nevertheless it makes sense to provide a mechanism for capturing good 

ideas as you proceed. 

To help you organize a needs assessment in your department, see Appendix A for 

a Sample Needs Assessment Action Plan Template.

Process Evaluation

This type of evaluation answers the question, “What does my program look like?” 

Data about the activities and tasks that characterize your program or strategy 

are collected primarily through observations of program activities, document 

review (e.g., meeting minutes, call logs, attendance records), and interviews with 

program staff, program participants, and other relevant stakeholders. See Exhibit 

II-2 for some questions that might be included in a process evaluation.

°

°

°

°

°
Measuring Excellence 21

Domestic Violence Process Evaluation 

Imagine that one of the top findings from the needs assessment outlined above 

was that many domestic violence victims said they did not know where and how 

to apply for Personal Protection Orders (PPO) or they felt too intimidated by 

the system to ask for help. The focus groups uncovered victims who did not call 

police because past experience or community gossip made them worry about the 

risk of dual arrest. Domestic violence victims, particularly those with children, 

hesitate to reach out to police if they fear they could be removed from the home. 

The police department decided to launch the first phase of an initiative to 

improve its response to domestic violence, including a public awareness and 

assistance campaign to educate victims about PPOs and make it easier for them 

to apply. The plan included developing brochures and videos about the benefits of 

PPOs and partnering with emergency rooms, schools, Neighborhood Watch, and 

local women’s groups to spread the word. The department also would recruit and 

train volunteers to help domestic violence victims complete the paperwork, with 

volunteers working from various community locations so victims being stalked 

would not be easily identified as seeking a court order. 

Exhibit II-2

Examples of Questions Addressed in a Process 

Evaluation

What are the critical components/activities of this project

 (both explicit and implicit)?

What aspects of the implementation process are facilitating success 

or acting as stumbling blocks for the project?

To what extent does the project look and act like the one originally 

planned?

Are the differences between planned and actual implementation 

based on what made sense for the stakeholders and goals of the 

project?

What project strengths can be built upon to improve the program?

Where are the gaps in services/program activities, and how can the 

program be improved to meet those unmet needs?

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 

W.K. Kellogg Foundation, 1998.

°

°

°

°

°

°
22 Key Concepts in Evaluation

To address concerns about police response, the department elected to develop 

and deliver specialized training to dispatchers and line-level officers on how to 

respond appropriately to domestic violence calls. While complaint records did 

not show response to be a serious problem, the chief and training staff felt the 

department would benefit from training in how to encourage victims to come 

forward and to help officers avoid overuse of dual arrest. 

Through observation, interviews, activity forms, and surveys, the process 

evaluation would address questions about the three main components of the 

program:

Public Awareness Campaign Process Evaluation. Were the brochures 

and videos developed, tested, revised, and distributed in sufficient 

numbers as promised to the partnering groups? Did the materials 

address language barriers and cultural concerns? Did the partnering 

organizations participate as promised? How many brochures were 

distributed within the time frame and to whom? How many times were 

the videos shown and to whom? What worked, what did not, and what 

could be improved? Were there unexpected obstacles or concerns and 

how were they addressed? Did the initiative identify other materials 

(posters, articles in church newsletters, public service announcements) 

that should be considered? 

Personalized Assistance Process Evaluation. Were sufficient numbers 

of volunteers recruited and trained? Was the training adequate (by what 

criteria and according to whom)? Did schedules and locations require 

adjustment? How were those decisions made? How many people were 

served during the evaluation period? Where did participants learn about 

the program? Did they receive a brochure or view the video, or both? If 

not, how did they find out about the effort? What seemed to work well 

and what did not? 

Police Training Evaluation. Was an appropriate training curriculum 

developed and delivered? How many dispatchers and police officers 

were trained? Were participants asked to evaluate the training and, if 

so, what were the results? How often did dispatchers and officers have 

the opportunity to apply what they learned in the target area? What did 

police officers do differently during calls for service? 

Additional questions to be answered for all program components are: “Were all 

of the activities accomplished in accordance with the proposed timetable and 

budget?” “If not, why not?” “Also, have issues or concerns surfaced that must be 

addressed before the program is expanded or replicated?” 

The process evaluation should gather all relevant products or written records—

copies of the brochures and videos, curriculum and support materials used to 

train volunteers and the police, meeting minutes, calls for service records, and 

officer activity forms—to document how activities are implemented and help 

you facilitate program revisions and replication.

1.

2.

.
Measuring Excellence 23

Outcome Evaluation

This type of evaluation addresses the questions, “What is my program 

accomplishing in the short term?” and “Am I meeting my objectives?” Data 

to measure where you are in relation to program goals and objectives can be 

collected in many ways, but common methods include testing, surveying, 

interviewing, and observing participants. Questions in an outcome evaluation 

might include those listed in Exhibit II-3.

Exhibit II-3

Examples of Questions Addressed in an 

Outcome Evaluation

What effect is the project having on its stakeholders or participants 

(e.g., change in knowledge, attitudes, or behavior)?

What unexpected outcomes, if any, have resulted from the program?

What can be modified to make the program more effective?

Is there any evidence showing that funders should continue to 

support this program?

Source: National Center for the Advancement of Prevention. Achieving outcomes: A practitioner’s guide to 

effective prevention. Washington, D.C.: Department of Health and Human Services, Substance Abuse and 

Mental Health Services Administration, and the Center for Substance Abuse Prevention, 2002. 

W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998.

°

°

°

°

Domestic Violence Outcome Evaluation

For our domestic violence example, three major outcomes are to be evaluated: (1) 

Public Awareness – Did the brochure and video outreach succeed in raising public 

awareness about PPOs and how to apply for one? (2) PPO Assistance – Did this 

initiative increase the number of domestic violence victims seeking a PPO? 3) Police 

Training – Did the training improve police response to domestic violence calls? 

Public Awareness and PPO Assistance Outcome Evaluation. The 

success of the public awareness and PPO assistance hinges on how many 

more people applied for PPOs as a result. We already know the number 

of people who received brochures and saw the video, but the marker for 

success is whether that prompted the desired action—domestic violence 

victims securing a PPO. 

°
24 Key Concepts in Evaluation

Determining program outcomes in this case requires establishing a baseline 

for the number of PPOs that people applied for prior to the intervention and 

comparing that figure to the number requested during an equivalent time 

period after public awareness and PPO assistance activities were launched. Also 

important is identifying other external factors that could influence the outcome 

and consider these when interpreting your findings. For example, since economic 

stress can trigger domestic violence, if there were massive layoffs, plant closings, 

or if the local economy experienced a boom during your study’s time frame, 

you would want to note this when interpreting your findings. You would not be 

expected to introduce these factors into your measurement model, but simply 

to think about the extent to which they may have influenced the outcomes you 

are seeing. Also find out whether there might be new prevention activities or 

programs that might affect overall rates of domestic violence in the community. 

Perhaps the number of PPO applications increased 35 percent during the first 6 

months of the new initiative, compared to the same period the previous year, and 

there were no other significant external events that appeared to account for the 

increase. An additional challenge is to determine whether the increase is a modest 

or great success and, as we will see when we examine cost effectiveness, whether 

the benefits justify the cost.

Police Training Outcome Evaluation. Evaluating the outcome of the 

police training is more challenging. The training was developed to address 

concerns raised in the victim focus groups; however, the extent of the 

problem was never quantified, giving no statistical baseline for easy 

comparison. In fact, the decision to train police was not made because 

problems with dual arrest were verified, but because of a notion that a 

new training component would make sense. To establish a baseline, the 

department might survey domestic violence victims with PPOs who 

placed a call for service before the training took place and compare the 

results to those from a similar group who placed calls after the training. 

If this approach is cost-prohibitive, the evaluation simply might survey 

victims after the training has taken place to determine if perceptions 

about police response improve over time. 

Ideally, the survey would be administered face-to-face by trained interviewers, 

including those who speak the language of the person being interviewed, and 

repeat visits would be scheduled to increase the response rate by developing 

rapport and trust with the interview subject. However, the issues of cost, time, 

and talent must be considered and there often is a tradeoff between conducting 

an ideal evaluation and a more modest one that fits your budget. You must decide 

whether available money might be invested more wisely in delivering a Cadillac 

program with a serviceable, but more modest, Chevy evaluation so that the cost 

of the evaluation does not detract from the program itself. 

°
Measuring Excellence 25

Impact Evaluation

This type of evaluation answers the questions, “Is my program producing 

long-term, global changes?” and “Am I meeting my long-term goals?” Data 

are collected about the long- or wide-reaching impact of the program. Many 

people confuse outcome and impact evaluation or are not clear about the 

differences between the two. While there are differences in such issues as the 

unit of analysis, the key difference is that an outcome evaluation documents 

short-term or immediate outcomes, while an impact evaluation is focused on 

long-term, more global changes. Using our domestic violence example again, the 

outcome evaluation is focused on increasing the numbers of PPOs applied for by 

individuals. On the other hand, the impact evaluation focuses on reducing overall 

domestic violence in the county, which will take much longer to achieve than 

increasing the number of PPOs. 

Domestic Violence Impact Evaluation

Assuming the three-component program of our example will encourage victims to 

secure a PPO and improve police emergency response to domestic violence calls, 

an impact evaluation will assess whether the new initiative had a positive impact 

on reducing domestic violence in the community. Sample questions that can be 

addressed in an impact evaluation are presented in Exhibit II-4. 

The simplest indicator of intervention success is a reduction in the total number 

of domestic violence incidents. (Success criteria also could include extending 

the time between repeat incidents or a reduction in the severity of individual 

incidents, but those outcomes will not be addressed in this example.) 

While the evaluation strategy appears straightforward and relatively easy 

to accomplish, you cannot rely on a simple comparison of the number of 

domestic violence incidents reported to police before and after the program is 

implemented. Issuing more PPOs and improving police emergency response by 

themselves might not be enough to make a dent in this chronic problem. Success 

also depends on others, such as prosecutors and courts, having an impact, too.

Also, although only a fraction of domestic violence incidents are reported 

to police, the public awareness component of the new initiative might bring 

previously hidden incidents to the attention of the police, as victims become 

empowered to seek help and as concerned neighbors recognize the importance 

of reporting suspected crimes. In fact, the number of reported domestic violence 

incidents actually could rise if the program succeeds in raising public awareness.

The challenge is in how to pinpoint any impact that this focused but relatively 

limited police initiative might have on a complex, chronic, and entrenched 

problem like domestic violence. While the evaluation strategy still would include 

gathering data on the number of incidents reported to police before and after 

intervention, more must be done to analyze what those numbers really mean. 
26 Key Concepts in Evaluation

Incident data could be supplemented with surveys or interviews of professionals 

who work with victims (e.g., domestic violence shelter workers, emergency room 

nurses) to learn whether they witness fewer suspicious incidents. Combining 

those findings with feedback from victims through follow-up interviews or 

periodic repetition of victim focus groups, will supplement statistical data with 

relevant qualitative case studies. 

A combination of skill, creativity, and experience is required to craft powerful 

impact evaluation strategies. If necessary, leave the more difficult strategies to 

professional researchers or at least include them in the evaluation team. 

Cost-Effectiveness Evaluation and Cost-Benefit Analysis

This type of evaluation, which can be part of one of the evaluation types discussed 

above, answers the questions, “What does the program cost?” and/or “What do 

the changes caused by the program cost in relation to the changes that occur 

as a result of program implementation?” To answer the first question, data are 

collected about program activities and costs (e.g., salaries, materials, space). To 

answer the second question, data are collected about program outcomes or impact 

and costs. Cost data commonly are collected from accounting or audit records, 

when available. If these data are not available, you can itemize all the resources 

used by the program and estimate costs for each. These and other questions are 

listed in Exhibit II-5.

While it is helpful to compare the costs of program services with costs of services 

otherwise received, you also can compare costs of program outcomes with costs of 

outcomes anticipated if there were no program. 

Exhibit II-4

Examples of Questions Addressed in an Impact Evaluation

What effect is the project having on our long-term goals 

(e.g., change in the number of reported incidents or in rates 

associated with the problem)?

What effect did the intervention activity have on components of the 

system in which the activity was targeted?

Were there any negative outcomes? Are they the result of implementation 

failure or some aspect of the intervention itself?

What degree of confidence is there that the outcomes can be attributed to 

the intervention itself? 

°

°

°

°
Measuring Excellence 27

While cost-benefit analysis has been around for a long time, especially in the 

environmental, technology, and physical sciences, it has been applied only 

very recently to social programs like the ones you are implementing. It also is 

a very complex analytic technique that can be difficult to generalize for small, 

or even large, social programs. As a result, you may find it difficult to develop 

a cost-benefit component for your program evaluation without soliciting input 

from a consultant or other outside expert who is well-schooled in cost-benefit 

procedures and techniques. You might be able to do some simple cost estimates 

and then determine whether those costs are offset by the program’s benefits 

(e.g., outcomes). To conduct a full cost-benefit study of your program, however, 

particularly a program that is large or has multiple components and target 

audiences, your best bet may be to identify an expert to assist in its development 

and implementation. 

Domestic Violence Cost Evaluation 

This evaluation examines whether the program delivered enough “bang for the 

buck.” The first step involves calculating costs and comparing them to outcomes 

and impact. Attempting to place a value on human life and human suffering is 

controversial; however, we must remember that relatively modest investments in 

effective programs can have a profound economic as well as humanitarian benefit 

to the community. 

Research shows that real costs are associated with domestic violence, including 

hospital and treatment costs and lost wages for the victim, lost productivity 

experienced by employers, and costs to taxpayers for dealing with batterers. 

Exhibit II-5

Examples of Questions Addressed in a 

Cost-Benefit Evaluation

What does the program cost?

What are the cost savings associated with program implementation 

(e.g., the return on investment)?

Do the costs outweigh the benefits?

Do the benefits outweigh the costs?

Will the cost savings increase with ongoing implementation or will 

they level off? 

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 

W.K. Kellogg Foundation, 1998.

°

°

°

°

°
28 Key Concepts in Evaluation

In 1996, the National Institute of Justice attempted to quantify the costs of crime 

victimization. It calculated, for example, that just one murder cost society $1.03 

million in tangible costs and $1.91 million in intangible costs.

Cost-benefit analysis is more common in business where the focus is on profits 

but when the goal is achieving justice, such tools also can help decision-making 

about investing scarce resources.

Discussion 

These brief examples underscore how crafting a comprehensive evaluation strategy 

for any police program or intervention requires careful thought and planning. 

Each type of evaluation can inform any others you might conduct—baselines 

identified in the needs assessment can provide the basis for comparison in process 

and outcome evaluations, and the findings in outcome evaluations can enrich 

the analysis of impact evaluations. Moreover, all evaluations can illuminate 

opportunities and obstacles and identify potential new partners. A summary of 

the evaluation types discussed above is in Exhibit II-6.
Measuring Excellence 29

Evaluation Type Process Outcome Impact Cost

Questions 

Answered

What is the program 

doing?

Is the program 

achieving its goals 

and objectives?

Is the program 

effecting change?

What does the 

program cost? 

What do the 

changes caused by 

the program cost?

Sample Variables Number of 

brochures 

distributed to ERs 

and schools

Number of 

volunteers 

recruited and 

trained; number 

of police officers 

trained

Number and 

content of 

training session 

topics

Training style

°

°

°

°

Increase in 

the number of 

applications for 

PPOs

Increase in public 

awareness about 

how to apply for 

a PPO

Improved 

police response 

to domestic 

violence calls

°

°

°

Decrease in 

the number 

of domestic 

violence 

incidents

Increase in time 

elapsed between 

repeat incidents 

°

°

Cost per response

Cost per PPO filed

°

°

Evaluation Designs

The type of evaluation you select, based on the questions you want answered 

about your program, will determine what evaluation design you will use—what

you will measure and how often or when you will measure it. The evaluation 

design is all-important because it affects what you are able to know about your 

program and its impact. 

As we introduce some of the most common evaluation designs, remember that 

some will be more feasible than others in the policing context. In fact, program 

evaluation has its roots in the fields of medicine (curing disease) and psychology 

(changing behavior), in which “treatments” are applied to groups of individuals, 

allowing researchers to identify statistically valid differences in outcomes. 

Evaluating police treatments (interventions) can mean adapting evaluation 

techniques to identify whether a program has had an impact on a geographic 

area (reducing drug dealing in a target neighborhood) or a problem (reducing 

physical disorder by organizing a cleanup) rather than on individuals within a 

group. Consequently, your internal evaluations are most likely to use post-testonly designs and least likely to use quasi-experimental or experimental designs. 

Nevertheless, in this section we have included information on all designs to offer 

Exhibit II-6

Overview of Evaluation Types
30 Key Concepts in Evaluation

you a broader understanding of the benefits and limitations of each. 

This information will help you appreciate what you can and cannot learn about 

your programs and activities, depending on which design strategy you follow.

Five common evaluation designs—post-test only, pre- and post-test, pre- postpost-test, quasi-experimental, and experimental—are described below and 

presented in order of complexity and expertise required, from the simplest to 

the most rigorous and elaborate (Shadish, et al., 2002). Classic evaluation design 

tables are read from left to right, first noting how many groups are involved in the 

research and continuing by noting the order in which measurements are taken 

and treatments applied. Keep in mind that in program evaluation, a pre-test or 

a post-test refers to a measurement (either through an interview, survey, or other 

data-collection method) used to document change in knowledge, attitude, or 

behavior. 

For the post-test-only evaluation design, data are collected only once from the 

program or from people who live in the target area. Using this design within 

a program delivered to the target group, participants are measured after they 

have gone through the program. Post-test-only measures are the least helpful 

for evaluating the impact of a program because there often is no valid baseline

to which the post-test results can be compared. To assess change, you must be 

able to compare post-program data with a baseline measure. Although this is 

the weakest form of evaluation design, it can be used when it is important (or 

feasible) to know if participants have reached an identified outcome, rather 

than to measure degree of change. Also, if you have limited resources (included 

limited access to program participants), this could be the best or only option. 

For example, consider a grant that funds a multi-agency federal, state, and local 

task force to collaborate in a yearlong effort to take illegal guns off the street. At 

the end of that year, the task force might conduct a post-test-only evaluation that 

shows total number of guns seized, with breakdowns about kinds of weapons and 

the places where they were seized. Such an evaluation also would include data 

on the number of cases investigated, number of arrests made, and a profile of a 

typical offender.

Group Pre-measure Treatment Post-measure

Treatment Group X O

X = program intervention is applied

O = measurement is taken

Post-Test Only Design 
Measuring Excellence 31

Group Pre-measure Treatment Post-measure

Treatment Group O X O

X = program intervention is applied

O = measurement is taken

Pre- and Post- Test Only Design 

The pre- and post-test design allows you to measure an individual or groups of 

individuals at two different times—before and after the intervention—enabling 

you to assess change by comparing the two measurements. This evaluation 

design has the benefit of being relatively easy to implement because you simply 

administer the same measure twice—before and after the intervention. You must 

carefully determine when to take the post-program measure, making sure to allow 

enough time for your program to have an effect.

While the pre- and post-test design is an improvement over the post-test-only 

design, it is not perfect. Even if your evaluation reveals a change in what was 

measured, you cannot be absolutely sure that your program is responsible for 

that change (i.e., you cannot determine causality). Many other factors that were 

not accounted for in your design could be at play, and it might be one of those 

factors, not your program, that had an impact. Without a well-defined program 

and sample, this type of evaluation design cannot be accomplished. 

To illustrate, consider the issues involved in evaluating a new police effort to 

reduce prostitution in a specific neighborhood where residents have complained 

that the problem is escalating out of control. A pre- and post-test design requires 

establishing baselines that will be measured after the intervention so, in this case, 

indicators likely would include the number of citizen complaints and number of 

arrests made, with the expectation that success would be reflected in a decline in 

complaints and a rise in arrests. 

As mentioned above, the challenge is making sure there are not other factors 

operating in the community that influence results. In other words, how do you 

know that it was your efforts, and your efforts alone, that caused any changes 

that are measured at post-test? Maybe there was a dramatic decline in complaints 

because the people who run prostitution in the neighborhood began intimidating 

residents who then feared talking to police. Or perhaps local public health 

officials received a grant to launch a comprehensive effort to reduce prostitution 

through an outreach that offers substance abuse treatment and job training. These 

circumstances can be addressed only through experimental and, to a lesser degree, 

quasi-experimental designs (described below).
32 Key Concepts in Evaluation

Among the benefits of a community policing approach to chronic problems like 

prostitution is not only that police will be more likely to find out about other 

community-based efforts, but also that collaborating to address the problem 

offers the best chance of success. By partnering with other agencies such as public 

health, the police also might enjoy benefits from being able to tap into their 

evaluation capabilities, and this may result in a stronger evaluation design and a 

greater understanding of whether police efforts were responsible for any changes 

measured at post-test.

 Pre- Post- Post-Test Design

The pre- post- post-test design provides measures at three distinct points in time. 

A pre-test measure is taken before any exposure to the program. The first post-test 

measure is taken immediately at the end of treatment, and the second (or followup) post-test measure is taken some time (e.g., 3, 6, or 12 months) after program 

completion. The advantage of this design is that it allows you to understand if the 

program treatment had any sustained impact or to know if program participants 

retained knowledge after a period of time. 

This design has the same weaknesses as the pre-post design in its ability to 

determine causality (i.e., to show conclusively that the program treatment is 

responsible for any changes measured). In addition, finding people to participate 

in the second post-test can be difficult and costly. Furthermore, people who 

are measured a third time could become bored or annoyed with the same 

measurement instrument, thereby affecting their responses. However, if we 

think of the prostitution example again, it might be valuable to examine citizen 

complaint and prostitution arrest data again 6 months after the intervention ends 

to see whether the department should renew its effort before the problem escalates 

again. This approach represents a practical way for police departments to achieve 

more longitudinal data collection and evaluation. 

Group Pre-measure Treatment Post-measure

Treatment Group O X O O

X = program intervention is applied

O = measurement is taken
Measuring Excellence 33

Quasi-Experimental Design

Group Pre-measure Treatment Post-measure

Treatment Group O X O

Comparison 

Group O O

X = program intervention is applied

O = measurement is taken

The quasi-experimental design comes one step closer to achieving the 

confidence level whereby you know that it was your program that caused change. 

In this design, you compare your target (or treatment) group (the group, 

neighborhood, or community that is the target of the policing effort or program) 

with another similar group that did not receive the program (i.e., a comparison 

group). Using quasi-experimental design, you would pre-test both groups before 

the program begins then deliver the program to only one group. You would 

administer a post-test to both groups, analyze the changes, and compare the 

results from both groups.

The challenge of the quasi-experimental design is to find two or more groups that 

are sufficiently similar in demographics (i.e., gender, ethnicity, socioeconomic 

status, education) and situation (i.e., both groups are adolescent boys at risk 

for juvenile delinquency) to be truly comparable. Your ability to say for certain 

that those who went through the program were the same as those who did not is 

limited, affecting your ability to say your program is responsible for differences 

that might exist between the groups after program intervention. The more alike 

the two groups are, the more confidence you can have that your program was 

responsible for positive outcomes.

The quasi-experimental design is frequently the best option for conducting 

outcome evaluations in the social services context, as it is often the only practical 

and ethical way to study two or more groups who vary in the programming they 

receive, because it does not rely on random assignment (see the experimental 

design example that follows). Additionally, to maintain and study a control 

group can be costly, which is even more significant because that group does not 

benefit from program treatment and already limited resources are diverted from 

programming. The quasi-experimental design averts these problems by using 

groups that already exist or that are not assigned randomly. An example of a 

comparison group is in a school setting where one class that participates in a 

program is compared to another class that does not participate. 
34 Key Concepts in Evaluation

This kind of research design is used more commonly by criminal justice 

researchers, often in concert with police, rather than by police alone. 

A jurisdiction moving toward a preferred or mandatory arrest (pro-arrest) policy 

in domestic violence situations could conceivably apply the policy in one area 

or sector and not another and then compare results. This would be an example 

of a quasi-experiment. Obviously, there are challenges involved in training and 

supervision to ensure that officers are applying the policies appropriately. Also 

of concern is that such policies require the support of the county prosecutor or 

district attorney’s office, since the point of pro-arrest polices is to send the message 

that domestic violence will be prosecuted with or without the full participation 

of the victim. It also goes without saying that, if pro-arrest policies do indeed save 

lives and reduce injuries, there are ethical concerns about whether it is or is not 

appropriate to withhold the intervention to the victims who happen to live in the 

area where the new policies do not apply.

This example qualifies as a quasi-experimental design because there could be 

differences between the geographic areas being compared that could affect the 

results of the evaluation. For example, arrest might carry more of a stigma in 

one area than another, so the impact of a pro-arrest strategy might show greater 

promise in an area where arrest induces shame than in an area where arrest is a 

more common occurrence. 

Experimental Design

Group Pre-measure Treatment Post-measure

Randomly 

Selected

Treatment 

Group 

O X O

Randomly 

Selected

Control

Group

O O

X = program intervention is applied

O = measurement is taken
Measuring Excellence 35

In experimental design, people from the same target population are assigned 

randomly either to a treatment group or a control group. Random assignment 

means that each person in the target population has an equal chance of being 

selected for either group. This randomness is crucial, because it is the only way 

to ensure that two groups are similar at the beginning of the intervention period. 

With this design, as with quasi-experimental design, both groups are pre-tested, 

only one receives the program intervention, and then both groups are post-tested.

Experimental design offers the best opportunity to know (and claim to others) 

that your program is responsible for changes in outcomes. It is used most 

often by researchers in the physical sciences because they can control their lab 

environments, repeat experiments, and determine causality. For obvious reasons, 

it is much more difficult to implement valid experimental designs in the policing 

context. Ethical concerns regarding experimental designs with humans often 

render experimental evaluation of human services unfeasible. Additionally, it 

might cost more to implement an experimental design and, despite the extra 

expense, several threats to the validity of the study would still exist. Threats to 

validity are presented in Exhibit II-8. Such threats are nearly impossible to control 

in real-life settings.

If we think of the pro-arrest example above, the logistics involved in randomly 

assigning officers to use a pro-arrest strategy on one call and then not on the next 

pose serious logistical and ethical questions. We also know, however, that proarrest policies can result in problems with dual arrest, where frustrated officers 

find it easier to arrest both individuals involved in the altercation than to try to 

differentiate between victim and offender. Training half the force in strategies that 

are designed to reduce the problem of dual arrest might allow for the randomness 

required by the experimental design. Then, if the results prove positive, it would 

justify investing in training the rest of the force.

To select the design that is right for your department, consider your needs and 

resources and pick the most rigorous design possible, given those restraints. Each 

type of design has its advantages and disadvantages, so be realistic about what 

kind of design you can implement. A post-test-only design might not produce the 

information you require, but quasi-experimental and experimental design studies 

could be too complex to perform without outside help. To manage evaluation 

effectively in your department, do not waste time and resources committing 

to a design that is beyond your capabilities or will not provide the information 

that you need in the end. In most cases, you should be able to find a happy 

medium. The different types of evaluations and their strengths and weaknesses are 

summarized in Exhibit II-7.
36 Key Concepts in Evaluation

Design Characteristics Advantages Disadvantages Required 

Expertises

Post-test only Measures one 

group at one time 

only—after the 

treatment.

Can document 

the reduction of a 

problem; requires 

access to only one 

group at only one 

time.

There is often no 

valid baseline 

measure for 

comparison. Does 

not allow one to 

assess change from 

baseline.

Low

Pre- and Post-test Measures 

one group at 

two different 

times—before 

and after program 

completion

Provides a baseline 

measure; requires 

access to only one 

group.

Cannot know for 

certain that your 

program produced 

the change.

Moderate

Pre- Post- Post-test Measures a group 

at three different 

times—once before 

the program and 

twice after program 

completion.

Enables you to 

determine if there 

are sustained effects 

from your program.

Does not show 

conclusively that your 

program produced 

the change; may be 

difficult to follow up 

with everyone.

Moderate

Quasi-experimental Measures two 

groups at two 

different times—

before and after 

the program; 

one group does 

not receive the 

treatment program.

Offers a decent 

level of confidence 

that your program 

caused the observed 

change.

May be difficult to 

find two groups that 

are similar enough 

to be compared; is 

more costly to have 

comparison groups.

Moderate to High

Experimental groups whose 

members are 

randomly assigned 

at two different 

times—before and 

after the program; 

one group does 

not receive the 

treatment program.

Provides the highest 

level of confidence 

that your program 

caused the observed 

change.

Difficult to 

implement in a 

real-life setting; there 

are ethical issues 

involved with the 

random withholding 

of a beneficial 

program from one 

group.

High

Exhibit II-7

Strengths and Weaknesses of Different Types of Evaluation
Measuring Excellence 37

Different types of data

Once an evaluation design has been chosen, focus on what types of data you will 

collect to answer your research questions and the data-gathering methods you 

will use. Quantitative data measure something that can be counted and assigned 

a number, and can be collected using several methods, including simple counts, 

surveys, and tests. Police departments often rely on quantitative data such as calls 

for service, arrests made, and citations issued. You might also take a simple count 

of how many people attended a neighborhood community meeting or calculate 

the percentage of survey respondents who are satisfied with police services. 

Quantitative measurement often assesses the extent and direction of change that 

has taken place as a result of the implementation of a strategy or program, and 

statistics often are used in quantitative data analysis. Quantitative measurement 

has several advantages. First, it yields data that can be standardized across a large 

number of people, can be replicated easily, is conducive to making comparisons, 

and can be less expensive and time-consuming to collect than qualitative data. 

However, quantitative data sometimes give a simplistic representation of reality 

and have been criticized for lack of attention to participant input. In addition, 

survey questions must be developed carefully to ensure they will capture the 

information you intend to measure.

One common problem that police face is that easily quantifiable data may not 

be kept in a form that suits their needs. For example, imagine that a department 

is planning a new effort to reduce the number of commercial burglaries. You 

find, however, that your call for service data does not make a distinction between 

commercial and residential burglaries and that there is no easy way to disaggregate 

the data. We know instinctively that there are many more home burglaries than 

commercial burglaries, so even if the new program is very successful, the data 

might not show enough of a reduction in the total number of burglaries to verify 

success. Do you delay the intervention until you can persuade dispatch to begin 

capturing burglary data in the format you need? Or do you pay someone to check 

the address for each burglary call to see whether it is a commercial or residential 

address? 

Qualitative data are collected through activities such as observations, interviews, 

focus groups, and even conversations. They describe the attributes of an object 

without referring to quantity and are expressed in words, not numbers. Instead 

of measuring the extent and direction of change or program impact, qualitative 

data are better suited to describing the nature of the change or impact that has 

taken place. Qualitative data are used frequently in process evaluation to describe 

how a program functions. In outcome evaluations, qualitative data are useful 

for providing context to and humanizing the numbers. Qualitative methods 

capture nuances and insights that statistical analysis of survey instruments 

cannot measure. Qualitative data provide context and can be more interesting 

and informative for a lay audience than a technical report filled with numbers 

and statistics. However, qualitative data can be overwhelming and cumbersome 
38 Key Concepts in Evaluation

to collect and analyze, and because data collection generally is more timeintensive, it is more expensive. Moreover, qualitative data lend themselves less to 

comparison and generalization than quantitative data.

In 2003, the Michigan Bureau of Juvenile Justice launched its yearlong Building 

Restorative Communities (BRC) initiative in five counties, designed to help 

build the capacity to identify and implement “evidence-based, data-driven, and 

outcome-focused” interventions to reduce juvenile crime. The core group of 

police, prosecutors, courts, and juvenile justice officials who work with at-risk 

youth helped mobilize a cross section of community groups to partner with them 

on developing a comprehensive 5-year plan. A grant allowed each county to hire a 

coordinator and a part-time evaluator for the project. 

One of the six work groups designated for the project was tasked with hosting 

separate focus groups of juvenile offenders in detention and victims of juvenile 

crime. Focus group facilitators were given a script with questions for each group, 

and a list of themes they could use to sort participant comments into useful 

categories. One question for the juveniles was, “What could have helped you stay 

out of trouble?” The findings would assist the steering committee work group 

to prioritize its final recommendations. This project confirmed that qualitative 

research produces significant amounts of important information but that it is 

not as easy to convey (or convey quickly). In Monroe County, with only 15 

participants in the juvenile offender focus group and two participants in the 

victim focus group, the final reports totaled more than 15 pages.

Both types of data have advantages and disadvantages, therefore, it is now widely 

accepted that using multiple methods and mixing the methods is the best 

strategy for gaining the most complete understanding of a program or strategy 

and its effects. In fact, the use of mixed methods increasingly is considered a 

prerequisite for quality data.

A discussion of the different methods that can be used to collect quantitative and 

qualitative data is presented in Chapter III: Conducting an Internal Evaluation. 

Whichever data instruments you choose, you must consider the ability or 

suitability of the instrument to capture the most valid and reliable data possible. 

The validity of data collected hinges on whether the data collection instrument 

measures what it intends or claims to measure. For example, crime statistics are 

not a valid measure of citizen satisfaction with police services. The reliability 

of data collected is a term used to describe an instrument’s ability to record 

consistently what is being studied. For an instrument to be reliable, it must yield 

the same (or similar) results even when administered in different places or at 

different times. Instruments must be both valid and reliable for you to claim that 

you have collected quality data. Exhibit II-8 summarizes several common threats 

to the validity of evaluation results.
Measuring Excellence 39

Exhibit II-8

Common Threats to the Validity of Evaluation Results

History: Any event that happens between the time of your first 

measurement (pre-test) and your second (post-test) that can affect the 

measurement. 

Maturation: The normal processes of development that occur over time 

could affect your outcomes, independent of your intervention. For 

example, as children grow and mature, they are likely to develop longer 

attention spans, according to available research on child development. 

This naturally occurring phase of development may occur while the child 

is enrolled in your program, making it difficult to separate program effects 

from normal developmental growth.

Testing: These are the effects that test taking have on the study results. 

Taking a pre-test may influence the behavior of your subjects as much or 

more than your intervention does. You cannot be sure that the effects you 

see are caused by your intervention and not by the administration of a 

pre-test.

Instrumentation: Any flaws in your measurement device that can skew 

your results. For example, if you collect information from an observer, he 

or she may be less attentive to collecting data on one occasion than on 

another.

Statistical Regression: A general tendency for extreme scores on any 

measure to move toward a less extreme score at a different time. If you 

measure something at the lowest possible amount, there is no way for it 

to change except to increase. Statistical regression occurs when you select 

study participants based on their extreme scores (e.g., if you decide to 

study only very depressed adolescents). When you remeasure, there is a 

tendency for these extreme scores to move back toward the overall group 

average regardless of the effects of any intervention provided.

Selection Bias: Any determination, except random placement, by which 

you place some subjects in a treatment group and other subjects in a 

control group is a type of selection bias. Subjects assigned to groups in any 

way other than random placement opens the possibility that the groups 

differ in important ways that can affect the results of your study. For 

example, students who volunteer to attend a tutoring session may be more 

academically motivated than students who do not volunteer to attend this 

group. Different levels of motivation may affect these two groups as much 

as or more than the tutoring groups did. Random assignment to groups is 

the best way to protect against the effects of selection bias.

Source: Mertinko, E., L.C. Novotney, T. Baker, T., and J. Lange. Evaluating your program: A beginner’s 

self-evaluation workbook for mentoring programs. Washington, D.C.: Office of Juvenile Justice and 

Delinquency Prevention, Office of Justice Programs, and the United States Department of Justice, 2002. 
40 Key Concepts in Evaluation

The Continuous Feedback Loop of Program 

Planning and Evaluation and the Key Steps in the 

Evaluation process

In managing or conducting an evaluation, it is critical to know where evaluation 

fits into the larger scheme of implementing police programs and strategies. 

Ideally, programs, projects, initiatives, and strategies are planned and carried out 

with a continuous feedback loop, of which evaluation is an integral part. Exhibit 

II-9 illustrates how evaluation is part of a continuous cycle of program planning, 

implementation, and refinement, and emphasizes the importance of connecting 

evaluation with program improvement for maximum results.

Exhibit II-9

The Program Planning and Implementation Cycle

The specific steps in evaluation also can be detailed, as shown in Exhibit II-10. 

Evaluate 

the program/initiative

Compare

 and analyze results against 

baseline and/or objectives

Specify (Plan)

what is to be collected

 and how

Collect 

information with 

appropriate tools
Measuring Excellence 41

The evaluation cycle itself is composed of several steps, which can be followed 

without much evaluation expertise; however, there may be times when you need 

outside assistance in developing or implementing an evaluation step, depending 

on the experience of your team and the complexity of the evaluation. Whether 

you are conducting an internal evaluation or managing an outside one, you 

will need some specific information about all the evaluation cycle components: 

evaluation planning, data collection, data analysis, and using and disseminating 

evaluation findings. Chapters III and IV in this guide provide more details about 

these components.

Exhibit II-11 highlights the integration of the planning and evaluation cycles.

Exhibit II-10

The Evaluation Cycle

Evaluate 

the program/initiative

Refine

the program based 

on evaluation results

Implement

the program/initiative

Plan 

the program/initiative
42 Key Concepts in Evaluation

Exhibit II-11:

Integration of the Planning and Evaluation Cycles

Collect 

information with 

appropriate tools

Compare

analyze results against 

baseline and/or objectives

Specify

what is to be collected and how

Evaluate 

the program/initiative

Refine

the program based 

on evaluation results

Implement

the program/initiative

Plan

the program/initiative
Measuring Excellence 43

Chapter Summary

Chapter II presented an overview of evaluation types and designs and described 

evaluation’s place in the cycle of program implementation and improvement. 

This chapter also covered the distinction between process, outcome, and impact 

evaluation and the pros and cons of several evaluation designs. Understanding 

these key evaluation concepts and the steps in the evaluation process will 

prepare you to conduct and manage valuable evaluation research in your police 

department. 

Evaluation is a powerful tool and the process is logical and relatively easy to 

grasp, particularly when you follow the logic model approach and other useful 

tips outlined in the next chapter. The rest of this guide concentrates on particular 

steps in the evaluation cycle, including specifics on how to conduct internal 

evaluations and manage them successfully so that results are disseminated and 

used effectively for program refinement and improvement. Chapter III provides 

details and guidelines, including examples and worksheets, for conducting an 

internal evaluation, while Chapter IV offers practical tools and instructions to 

help you manage evaluative research in your police agency.

Chapter III:

Conducting an Internal Evaluation

Measuring Excellence 47

Conducting an Internal Evaluation

An internal evaluation is one in which the police agency relies primarily on 

people within the department to conduct an evaluation of policing efforts and 

to disseminate results. Internal evaluations are designed to help police agencies 

learn more about the operation, performance, and perceptions of all or part of 

their departments. Evaluations can vary from a process evaluation to determine 

whether a new call management system has been implemented as directed, to an 

outcome evaluation of a narcotics task force, to a community satisfaction survey 

as one indicator of the long-term impact of community policing on attitudes 

about police.

The following are some benefits of an internal evaluation:

The evaluation will reflect staff knowledge about the program.

The evaluation may be more successful in getting support from police 

administration and other staff.

The evaluation may be less expensive than other evaluation options.

While an internal evaluation might meet the needs of the program staff and 

administrators better, the following are some potential criticisms of internal 

evaluations:

The evaluation can be perceived as less objective and less credible.

The evaluation might not be adequate for answering the evaluation 

questions.

Perhaps the biggest misunderstanding about internal evaluation is that it must 

follow an academic research model, which focuses on testing hypotheses. Most 

law enforcement agencies should focus instead on answering practical program 

and policy questions. While academic research requires a more controlled 

approach with comparison and control groups, this is seldom realistic for a law 

enforcement agency operating in a real community with many uncontrolled 

variables and where using a control group is not feasible or is unwise. For an 

internal evaluation designed to improve programs and practices, you might not 

need such control or sophisticated statistical analyses. The methodology for 

practical research should be based on identifying program or effort that is being 

evaluated, the type of data that can be collected, and the availability of resources 

for data collection and analysis. 

Some of the primary differences in purpose, method, and statistics between 

academic research and internal evaluation are presented in Exhibit III-1

°

°

°

°

°
48 Conducting an Internal Evaluation

If evaluation results are intended mainly for internal use, the purpose, method, 

and statistics of academic research often are too costly for most law enforcement 

agencies or are beyond their scope. Nevertheless, employing evaluation procedures 

that are explicit, formal, and justifiable is important so that police departments can 

have confidence in the validity of their findings. Law enforcement agencies, therefore, 

should do the best evaluation they can with available resources, carefully noting the 

limitations of findings. Ultimately, the standard by which your evaluation should be 

judged is whether the collected information was used to make decisions or 

improve the program (Loe, 2001). 

If you have decided, based on your evaluation needs and resources, 

that conducting an internal evaluation is your next step, the rest of this 

chapter provides an overview of how to conduct an evaluation, including 

developing an evaluation plan, determining data-collection strategies, 

and conducting data analysis. Several tips and practical worksheets are 

included throughout the chapter and in the appendices. The resource list 

in Appendix L directs you to more evaluation references. 

Academic Research Internal Evaluation

Purpose Test hypotheses; scientifically 

test a program model; advance 

the overall literature and field of 

law enforcement

Improve program and practice; 

address the needs of a single 

law enforcement agency or a 

group of agencies

Method Controlled environment Context-sensitive

Statistics Sophisticated Simple

Exhibit III-1

Difference Between Academic Research and 

Internal Evaluation

Source: The University of Texas-Houston Health Science Center School of Public Health, and The Texas Department 

of Health. Practical evaluation of public health programs. Atlanta, Georgia: The Centers for Disease Control and 

Prevention and The Association of Schools of Public Health Cooperative Agreement, 1998.

How We Did It: 

Colorado Springs Police Department

The Colorado Springs Police Department’s 

(CSPD) Police Accountability and Service 

Standards (PASS) model demonstrates how 

internal evaluations can produce valuable 

information to both the law enforcement 

agency and the community. CSPD uses 

internal data collection and analysis of 

its operations and programs to improve 

policing efforts and ensure that they are 

meeting community expectations.
Measuring Excellence 49

Steps in the Evaluation Process

The four main steps in the evaluation process were introduced in Chapter II: 

SPECIFY—evaluation planning, COLLECT—data collection, COMPARE—

data analysis, and COMMUNICATING and using the results. The first three 

steps are the focus of this chapter. The final step is a management task and is 

discussed in greater depth in Chapter IV.

Step 1: SPECIFY—Evaluation Planning. An evaluation plan provides a 

framework that will help you specify what to evaluate and how. A clear 

evaluation plan facilitates not only evaluation implementation but also 

its management. Evaluation planning steps that will be discussed in 

this chapter include the following: assessing whether your program is 

ready to be evaluated, assessing stakeholders’ commitment to evaluation 

and planning for their involvement, understanding and defining your 

program goals and objectives, developing a logic model, determining your 

evaluation questions, and selecting an evaluation design based on the 

careful planning you have done.

Step 2: COLLECT—Data Collection. Your evaluation data will be 

collected through surveys, interviews, focus groups, document review, or 

other methods, depending on what has been specified in your evaluation 

plan. After determining what data already exist and will be useful, you 

might still need to develop your own data- collection instruments. Tips 

for creating and implementing surveys, interviews, and focus groups are 

included in this chapter.

°

°

Evaluate 

the program/initiative

Compare

 and analyze results against 

baseline and/or objectives

Specify (Plan)

what is to be collected

 and how

Collect 

information with 

appropriate tools
50 Conducting an Internal Evaluation

Step 3: COMPARE—Data Analysis. The data you collect will be analyzed 

according to your evaluation plan and the data-collection instruments 

that have been used. The purpose of analysis is to reduce, categorize, and 

summarize your data in a way that helps you and others understand what 

is happening with your policing programs and activities and/or what is 

changing as a result of those efforts. Evaluation data often are analyzed 

using simple counts, statistics, and content analysis. This chapter gives a 

brief overview of each.

Step 1: SPECIFY—Evaluation Planning

Preparing for any evaluation requires an investment of time and thought. 

Attention to planning each evaluation step will increase the likelihood of 

producing useful results. Evaluation planning helps you carry out activities 

efficiently and effectively and allows someone outside the organization to 

understand what and how you are evaluating. 

Many police agencies make the mistake of delaying their thinking about evaluation 

as long as possible, often until after the program is underway. Treating evaluation 

as an afterthought not only jeopardizes its quality, but also robs the department of 

the benefits that evaluation can deliver to planning and implementation. Thinking 

through the evaluation plan while the program or intervention is being developed 

helps clarify what should be done and how. Identifying the various measurements 

that will be used also focuses participants on what it will take to make the initiative 

a success. Delaying evaluation planning also might make it more difficult to gather 

the data you need or to put it in the format that best suits your purposes than if you 

had explored various options earlier. 

°

Evaluate 

the program/initiative

Compare

 and analyze results against 

baseline and/or objectives

Specify (Plan)

what is to be collected

 and how

Collect 

information with 

appropriate tools
Measuring Excellence 51

Perhaps the biggest advantage of early evaluation planning is that it can allow 

police to make course corrections along the way, increasing the odds of success. 

Police executives who worry that evaluation could trumpet bad news should 

understand that it is in their interest to engage in evaluation planning in advance, 

incorporating opportunities to monitor progress so that appropriate changes and 

improvements can be made quickly. 

Evaluability Assessment

Not all programs can or should be evaluated because some are too small, too 

short, or are ill-conceived. For others, it might be too early in their development 

to conduct an outcome or impact evaluation. As part of planning, it can be 

helpful as a first step to identify whether your program is ready to be evaluated. 

To do so, you can conduct an informal evaluability assessment, which lets you 

verify whether the program has clearly specified activities and well-articulated and 

measurable goals and objectives that lend themselves to identifying data that can 

be collected. This process includes reviewing the program history, design, and 

operation, perhaps by observing the program in action; determining the program’s 

capacity for data collection; and assessing the likelihood the program will meet 

its goals and objectives based on implemented activities. While the concept of an 

evaluability assessment might be unfamiliar, this type of preassessment increases 

the likelihood that a full-blown evaluation will provide timely, relevant, and 

responsive findings (Juvenile Justice Evaluation Center, 2003). Poorly planned 

or implemented programs should not be evaluated because findings will be 

inconclusive at best and misleading at worst. Therefore, it is often in a program’s 

best interest to invest the resources necessary for either a formal or informal 

evaluability assessment (Juvenile Justice Evaluation Center, 2003). If your 

evaluability assessment results show a program is not ready for an evaluation, an 

outside evaluator might be able to work with the program manager and staff to 

bring the program to the appropriate stage. 

In addition to helping determine whether to conduct an evaluation, data collected 

from an evaluability assessment can describe the sequence of critical elements 

that influence program design, implementation, and effectiveness, and they can 

direct you toward developing suitable measures and research designs. This often is 

graphically represented in a logic model, which is described in greater detail later 

in this chapter.

Planning Stakeholder Involvement and Assessing 

Stakeholder Commitment

Another step in evaluation planning is to identify key stakeholders and assess how 

committed they are to an evaluation. In addition to your police department, an 

evaluation stakeholder could be the agency or organization that funds the grant 

under which your program operates. For example, if you are receiving funds 
52 Conducting an Internal Evaluation

from the Department of Justice to implement a gang-prevention program, you may want to 

talk with your federal program manager to see if he or she would supportive an evaluation. 

Other important stakeholders might include local government, members of the business 

community, schools, and nonprofit agencies, as well as community residents who have been 

involved in identifying problems and finding solutions. The following questions will guide 

you in thinking about the level of stakeholders’ commitment and the ways in which they 

might facilitate your evaluation:

What might be the concerns stakeholders have about conducting evaluation at this 

time? How might obstacles be overcome?

What are the reasons that stakeholders might support an evaluation at this time? 

What questions do they have about the implementation or effectiveness of policing 

efforts?

What information do the stakeholders need about your program? How will key 

stakeholders use information from this evaluation?

What information or expertise can stakeholders contribute to your evaluation? What 

are stakeholders willing to contribute to an evaluation at this time?

How can evaluation help make a persuasive argument for community support?

Understanding stakeholders’ concerns—and what they might be most enthusiastic about—

can help in planning a more effective evaluation strategy. At the same time, addressing their 

concerns and interests generates buy-in. Stakeholder commitment often can be increased by 

establishing an advisory group or evaluation team. A well-constituted advisory group, made 

up of the evaluator, representatives from the police agency, program participants, and/or 

interested citizens, can be an asset for formulating evaluation questions, providing additional 

evaluation resources, and being advocates and spokespeople for the dissemination of results 

(Center for Substance Abuse Prevention, 2003). 

The Evaluation Plan

An evaluation plan documents a system for collecting evidence of 

program effectiveness and will help ensure that you are collecting the 

information you need in the most meaningful way. Developing your 

plan means defining carefully what to evaluate and how to measure 

success, formulating evaluation questions, selecting the appropriate 

evaluation design, and developing data-collection methods. During 

evaluation planning, you also should think through a project timeline 

and budget and a strategy for using and disseminating your findings. 

These management issues are discussed in the chapter “Managing 

Evaluations.” 

°

°

°

°

°

Key Steps in Evaluation 

Planning

Determine whether the program is 

ready for evaluation. 

Assess stakeholder commitment to 

evaluation.

Define program goals and objectives.

Develop a program logic model.

Determine evaluation questions.

Choose an evaluation design.

Plan for data collection and analysis.

°

°

°

°

°

°

°
Measuring Excellence 53

Determine What Is Going To Be Evaluated

Although it might seem obvious, determining what is going to be evaluated could 

be difficult if your program was not well defined at the outset or if activities 

were not designed and implemented so they could achieve the stated goals or 

produce meaningful benefits. Before conducting an evaluation, be clear about 

the program’s purpose, its goals and objectives, and the activities that have been 

put in place to meet them. Understanding the relationship among them also 

might help determine whether you want to examine the whole program or just a 

particular component. For instance, one program component might be selected 

if you think it has the strongest effect on participants, or if it requires the greatest 

proportion of your resources and, therefore, you are particularly concerned about 

its effectiveness. 

Define Program Goals and Objectives

Understanding your program’s mission, goals, and objectives will give you ideas 

about how success can be measured in your evaluation. Before evaluation begins, 

therefore, become familiar with your program’s goals and objectives and know 

whether they are measurable. If goals and objectives have not been articulated 

or written, you must work with your evaluation team and stakeholders to do 

so to ensure you do not undermine your results. See Exhibit III-2 for tips for 

developing goals and objectives.

A goal is a measurable statement of the desired longer term, or global, impact of 

the program. Goals typically address change. For example, recalling our domestic 

violence example from Chapter II, a goal might have been, “By 2006, our 

program will reduce domestic violence in our county by 10 percent.” 

An objective is a specific, measurable statement of the desired immediate or direct 

outcomes of the program that support the accomplishment of a goal. Again, using 

our domestic violence example from Chapter II, an objective could be, “Our 

program will increase the number of personal protection orders (PPOs requested 

in our county by 50 percent.” 

Note that objectives state the desired immediate or direct outcomes of 

programmatic efforts; therefore, it is useful to distinguish between outcomes and 

outputs. Outcomes refer to the changes that result from a program or its activities 

(e.g., increase the number of PPOs requested). Outputs are units of service 

that are provided by the program (e.g., the number of police officers who went 

through domestic violence/PPO training).

Program goals and objectives should be tangible and measurable. Use the exercises 

in Appendix B to guide you in developing program goals and objectives or in 

adopting those you feel accurately reflect your policing mission. 
54 Conducting an Internal Evaluation

Develop a Program Logic Model

Determining exactly what to evaluate can be facilitated by using a logic model, 

which is a graphic representation of the logical relationships among program 

conditions, activities, outcomes (objectives), and impacts (goals). A program logic 

model for an evaluation plan provides explicit insight into how the program is 

supposed to work to achieve the desired benefits. A logic model incorporates a 

series of “if-then” changes that the program aims to achieve through its activities, 

outputs, and outcomes, and which can help you evaluate whether the program 

has met its goals and objectives. A logic model does the following:

Requires program staff to think through goals, objectives, and strategies 

and to develop a realistic idea of what the program can accomplish

Clarifies assumptions and relationships between program efforts and 

expected results

Provides a useful framework for examining program outcomes 

and impacts

Helps specify what to measure through evaluation.

Logic models usually are diagrammed as a series of boxes representing the 

conditions (needs or problems), activities, outputs, outcomes, and impacts. 

Each box in the logic model will give you a reference point for comparing your 

program’s progress toward achieving the desired goals and objectives (note: 

outcomes and impacts are the measure of your objectives and goals, respectively). 

Obtaining stakeholder input into the development of the program logic model 

can be a strong foundation for planning and prioritizing evaluation activities. 

°

°

°

°

Exhibit III-2

Tips for Developing Goals and Objectives

Be careful about defining goals too narrowly so that they appear 

to be outcomes.

Beware of stating activities as goals or objectives.

Be sure not to write compound goals and objectives.

Be realistic.

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 

W.K. Kellogg Foundation, 1998.

°

°

°

°
Measuring Excellence 55

A logic model can guide all types of evaluations and help you specify what needs 

to be collected. For process evaluations, the logic model identifies expectations for 

how the program should work so you can see whether your program has derailed 

or is on track. For outcome and impact evaluations, the logic model displays how 

and for whom certain services or activities are expected to create change.

A logic model with law enforcement examples is provided in Exhibit III-3. The 

corresponding worksheet and template in Appendix C will walk you through the 

steps of developing a logic model pertinent to your evaluation needs. 
56 Conducting an Internal Evaluation Exhibit III-

3

Sample Logic Model

ActivitiesWhat are the strategies, techniques, or types of services that comprise a program’s service? (e.g., police officer training; block watch programs)

Conditions(problems or needs)What problem or condition is your program designed to change? (e.g., need to reduce domestic violence; need to increase citizen participation)

OutputsWhat are the direct products of program activities? (e.g., number of police officers trained; number of people who attended block watch programs)

Outcomes(short-term changes tied to program objectives) What are the immediate benefits of the program activities? (e.g., increase in number of PPOs requested; increased citizen participation)

Impacts(long-term changes tied to program goals)What changes have been achieved in the original conditions? (e.g., reduced domestic violence; improved relations with community) 
Measuring Excellence 57

Develop Your Evaluation Questions

The next step in evaluation planning is determining what you and other key 

stakeholders want to know about the program through your evaluation inquiry. 

As noted in Chapter II, there are four major types of evaluation: needs assessment; 

process evaluation that documents whether a program is being implemented as 

planned; outcome evaluation that documents short-term outcomes and often is 

used for ongoing program improvement; and impact evaluation, which tells you 

whether the program achieves longer term goals. 

The evaluation questions you choose will be tied to your program goals and 

objectives as well as the type of evaluation you plan to implement. Evaluation 

questions should be developed according to what you want to learn about your 

program. The following are some typical evaluation questions:

What is the nature of the problem (e.g., delinquency, graffiti)? 

Where is the problem? How widespread is it? Whom does it affect?

What is the program or intervention? 

Is the program being implemented as intended? 

Is the program reaching its target population?

What are the barriers to implementing the program?

What approaches to overcoming implementation obstacles have been 

effective?

How much does the program intervention cost?

Is the program achieving its objectives? 

What are the actual outcomes?

Is the program cost-effective?

Choose an Evaluation Design

Understanding your program goals and objectives, creating your logic model, 

and delineating your evaluation research questions will help point you to the 

appropriate evaluation design. Five of the most common evaluation designs, 

post-test only, pre- and post-test, pre- post- post-test, quasi-experimental, 

and experimental, were discussed at length with specific examples from the 

law enforcement context in the chapter “Key Concepts in Evaluation.” Please 

reference the chapter to refresh your memory and to help select the design that 

is most appropriate for your needs and resources. The evaluation design that you 

choose can help direct you to the appropriate data-collection sources, which in 

turn will influence the analytic procedures you use to make sense of your data and 

answer your evaluation questions. Data collection and analyses are the focus of 

the remaining section in this chapter.

°

°

°

°

°

°

°

°

°

°

°
58 Conducting an Internal Evaluation

 Step 2: COLLECT—Data Collection

Once you have determined your evaluation purpose and design, you can begin 

preparing for data collection. The data-collection methods you choose will 

reflect the type of evaluation you decide to conduct, the evaluation design that 

you have chosen, the evaluation research questions on which you plan to focus 

your activities, and the specific data needed to answer those questions. Since 

data collection can be resource-intensive (both expensive and time-consuming), 

plan your data collection carefully and limit it to only the data that answer the 

evaluation questions you have specified. Your data-collection strategies can be 

relatively simple and straightforward if you plan wisely. In fact, data collection 

can be incorporated easily into routine program activities and often is something 

police officers already do, perhaps without realizing it. 

Key Steps in Data 

Collection

Review existing data and data sources. 

Develop any necessary new datacollection protocols.

Obtain the necessary clearance and 

permission.

Collect and track data.

°

°

°

°

Evaluate 

the program/initiative

Compare

 and analyze results against 

baseline and/or objectives

Specify (Plan)

what is to be collected

 and how

Collect 

information with 

appropriate tools
Measuring Excellence 59

Review Existing Data and Data Sources

Prior to selecting new data-collection strategies, you should first consider and 

review existing data and data sources. For example, traditional measures of police 

performance and crime, such as Uniform Crime Report (UCR) data, provide 

a nationwide view of crime based on statistics from law enforcement agencies 

throughout the country. The UCR program routinely collects data on known 

offenses and persons arrested by police departments for murder and non-negligent 

manslaughter, forcible rape, robbery, aggravated assault, burglary, larceny-theft, 

motor vehicle theft, and arson. These data may be appropriate to use if you are 

conducting a needs assessment (to determine the nature and extent of criminal 

activity in your jurisdiction) or an impact evaluation (to determine the extent 

to which your programs, strategies, and initiatives are effecting change in longer 

term criminal activity in your jurisdiction). However, they might not help you 

understand much about any single program or specific outcomes related to your 

targeted activities. For that, you would need to collect outcome data, which are 

tied to your shorter term objectives, and which typically are gathered directly 

from program participants or stakeholders (e.g., citizens, businesses, or police 

officers who go through special training) before and after implementation of your 

program or strategies. These outcome data will help you assess the extent to which 

your efforts are meeting defined objectives; impact data will show you whether 

you have met your longer term goals. 

Data collection can be one of the most resource-intensive steps in the evaluation 

cycle. To find and use existing data for your evaluation, we encourage you 

to explore other less traditional data sources that might inform an outcome 

evaluation, including calls for service, medical admissions (e.g., gunshots, 

overdoses), vandalism and graffiti costs from parks and recreation, and insurance 

information on stolen cars and burglaries.

Develop Necessary Data-Collection Protocols

Reviewing your existing data and data sources will allow you to decide what data 

you need to collect that you are not capturing now. Once you have done this, 

make a list of topics about which you need to know more. Prioritize among the 

items and finalize the list based on the relative importance and usefulness of the 

information as well as the cost and ease of collecting those specific data. Now 

you are ready to develop data-collection protocols you might need to help you 

obtain the information on that list. Quantitative and qualitative approaches to 

data collection are reviewed briefly in this section (also see Chapter II), along 

with an overview of specific data-collection methods and tips for creating surveys, 

planning and conducting focus groups, and developing interview guides.
60 Conducting an Internal Evaluation

Quantitative Data

Quantitative methods capture data that can be counted, measured, compared, 

or expressed in numerical terms. Surveys, questionnaires, and archival data are 

commonly used quantitative data sources. Archival data already have been 

collected by your department or by another organization, such as a government or 

nonprofit agency. These documents might include program records or materials 

such as proposals, annual or monthly progress reports, budgets, organization 

charts, memoranda, policies and procedures, operations handbooks, and 

training materials. Archival data are analyzed through document review. These 

data can be the simplest and cheapest kind to gather, because someone else has 

collected them. Additional benefits to using archival data are that the review of 

program documents can provide an idea of how the program functions without 

interrupting program staff or activities and document review requires relatively 

little expertise to conduct.

Surveys and questionnaires involve collecting data directly from individuals. 

Through self-administered or face-to-face surveys, telephone or mail surveys, 

questionnaires, and checklists, you can ask program stakeholders directly and in 

a standardized way whether and how your efforts are making a difference (Salant 

and Dillman, 1994). Surveys and questionnaires can be especially useful in 

evaluation if the same set of questions is asked at the beginning of a program (i.e., 

a pre-test used for baseline information) and again when the program is over (i.e., 

a post-test to document any change over time). 

In law enforcement, surveys can help you collect information about criminal 

victimization, community views about crime, and residents’ willingness to report 

crime to the police. Since such data provide a measure of police performance, 

they can be used to analyze the way police deliver services and possibly change 

resource allocation, if needed. Some community surveys provide detailed 

information about specific problems affecting parts of a city, which help to focus 

police activities (Weisel, 1999). 

Law enforcement agencies that implement community policing strategies often 

conduct citizen surveys to gauge community satisfaction and identify citizen 

concerns. Such surveys can help identify target areas and document progress 

and change from year to year. An example of a community satisfaction survey is 

found in Appendix D. You might find it beneficial and cost-effective to adopt or 

modify existing surveys that have been used in the policing context or in other 

relevant research. Data-collection instruments often are free, although some are 

copyrighted and must be purchased. For example, the Bureau of Justice Statistics 

and the COPS Office developed a software program for localities to conduct 

their own telephone surveys of residents to collect data on crime victimization, 

attitudes toward policing, and other community related issues. This survey can 

produce information similar to that published in Criminal Victimization and 

Perception of Community Safety, Twelve Cities. This software has recently been 

developed into an on-line application called Justice Survey Software and can be 

found at www.bjsjss.org
Measuring Excellence 61

Surveys and questionnaires have limitations, in general terms and when used 

in the policing context. In general, written surveys can preclude participation 

by people who speak other languages, who are illiterate, or who have learning 

disabilities, while telephone surveys can omit people who do not have telephone 

service. When using surveys to evaluate policing efforts, there are special concerns 

and barriers inherent in surveying victims and offenders. Many crimes occur in 

high-crime areas where people often are transient, so a disproportionately high 

number of potential survey subjects are likely to be harder to reach by telephone 

or written survey. Residents in high-crime neighborhoods also are more likely 

to be undereducated and under- or unemployed, which makes them harder to 

reach and potentially less likely to have the literacy skills required to respond to 

a written survey. Traumatized victims could be reluctant to share information 

over the telephone to strangers they cannot see, for fear of retaliation and 

because of the personal nature of information they might be asked to share. 

Honesty and candor are obvious concerns in surveying offenders and there are 

often insurmountable hurdles in identifying potential offenders who have not 

yet committed a crime. Ensuring that surveys are the appropriate means of 

data collection for your evaluation requires thinking through these potential 

barriers. Should you decide to use surveys in your research, some tips for survey 

development are presented in Exhibit III-4.
62 Conducting an Internal Evaluation

Exhibit III-4:

Tips for Developing Surveys and Survey Questions

Keep your survey short, ideally no more than three or four pages.

Make it easy to use—participants will be more likely to complete it.

State the purpose of the survey on the first page, and also address the 

matter of confidentiality. Include necessary directions, explanations, or 

definitions.

Indicate the approximate time it should take to complete the survey, 

where the respondent should return it, the due date, and contact 

information for answers to survey related questions.

Keep your survey instrument simple, with short questions and clear 

answer categories.

Look for other survey instruments with similar content that have been 

used in the field. Often, data-collection instruments are free, although 

others are copyrighted and have to be purchased. When relevant and 

appropriate, use existing questions from these surveys.

Avoid “double-barreled” questions (i.e., questions that ask two things 

at one time). For example, “Did you find the training useful and 

timely?” The respondent may have found the training timely but not 

useful, so by combining what are two separate ideas into one question, 

you will not get an accurate answer.

Use appropriate language for the audience. Questions should be 

devoid of vague qualifiers, abstract and biased terms, and jargon.

Group your questions thoughtfully and logically. In general, begin 

with easier questions and progress into more sensitive or complicated 

topics.

Request demographic information.

Always include a statement at the bottom of your instrument thanking 

respondents for their time.

Conduct a pilot test of your instrument.

For written or e-mail surveys, send reminders to respondents to 

increase your response rate.

°

°

°

°

°

°

°

°

°

°

°

°

°
Measuring Excellence 63

Qualitative Data

Qualitative methods allow you to capture information that is difficult to measure, 

count, or express in numerical terms. Four qualitative data-collection methods 

and several tips for conducting interviews and focus groups are described on the 

following pages.

Interviews

Qualitative interviews are used to gather detailed information on a topic. The 

researcher asks open-ended questions to allow respondents to express in their 

own words their attitudes about, and experiences with, the program. Interviews 

typically ask people to describe or explain particular program practices or issues in 

relation to their personal experiences with them. Interviews can be a primary data 

source, a first step in developing other data-collection strategies (e.g., survey or 

focus group questions), or a complement to quantitative data collection. 

Interviews can be conducted by telephone or in person. Among the strengths 

of interviews is that they let you gather information you cannot learn through 

document review or observation. While interviews can provide more detail and 

depth than surveys can about stakeholders’ experiences and opinions, one of their 

disadvantages is that they are time-consuming. Interviewing tips are offered in 

Exhibit III-5.
64 Conducting an Internal Evaluation

Exhibit III-5:

Tips for Conducting Interviews

Prepare your questions.

Use interview questions that are open-ended, clearly worded, 

and neutral. 

Organize broad questions around the general themes in which you 

are interested, then you can probe to gain more information based 

on participant responses. 

Ask how and why questions and make sure you prompt 

respondents to elaborate on yes or no answers.

Use appropriate language to which respondents will relate.

Organize questions so they flow naturally and easily. Begin by 

asking nonthreatening questions that put respondents at ease and 

end the interview by asking if there is anything important you 

forgot to ask.

Choose an appropriate setting. Interviewees should feel comfortable 

and not be distracted by other activities, lights, or noises.

Explain the purpose and format of the interview to respondents.

This explanation also should cover terms of confidentiality, 

approximate length of the interview, and securing permission to record 

or take notes during the interview.

Practice “active listening skills” so participants are fully aware that 

you are paying close attention to their responses. Use silence to allow 

for elaboration.

Probe for clarification. Answers to questions are not always direct 

and might require clarification or follow-up. Use probes to steer 

respondents toward the issue at hand.

Tape-record the interviews, with respondents’ permission.

Always thank interview participants for their time.

°

 ̇

 ̇

 ̇

 ̇

 ̇

°

°

°

°

°

°
Measuring Excellence 65

Focus Groups

A focus group is a small-group discussion guided by a trained leader, which 

explores a topic in depth to steer future action. Focus groups are a quick way to 

gain impressions and opinions from various stakeholders and are appropriate 

when action should be guided by public opinion. Some law enforcement agencies 

use focus groups when securing community input on a critical issue before 

moving forward. 

As with all data-collection strategies, successful implementation depends on 

thorough planning. To prepare for a focus group, ask yourself, “What do I want 

to learn?” and “Why am I conducting a focus group?” Advice for preparing for 

a focus group is offered in Exhibit III-6 and a focus group checklist with more 

detailed information is in Appendix E.
66 Conducting an Internal Evaluation

Exhibit III-6

Tips for Planning and Facilitating Focus Groups

Find a good focus group leader. A good leader should be 

experienced, understand the focus group topic, and relate well to 

participants. While you might be tempted to facilitate the focus group 

yourself, it would be more appropriate to ask someone else within 

your agency to facilitate or hire an outside facilitator who will be 

familiar with focus group procedures. These include asking appropriate 

questions, reviewing the goals of the meeting with participants, setting 

participation ground rules, ensuring that all opinions are heard, 

concluding the meeting on time, and providing participants with 

information about the next steps in the process.

Prepare your questions. Before the focus group starts, you should 

prepare a list of topics and questions that you want to ask. Word and 

organize them strategically. For example:

Choose easy and nonthreatening opening questions designed to 

break the ice and make participants feel comfortable.

Use transition questions to get the data you need.

Prioritize your key questions and make sure they are answered 

before you finish the focus group.

Include a final question that summarizes the discussion and 

gathers missing information.

Find a recorder. A focus group requires someone to take notes similar 

to taking minutes at a meeting. You also can tape-record focus groups 

with the group’s permission. It can be useful to have an audio record in 

case you need to enhance your written notes. 

Decide about incentives. Sometimes focus group members get paid, 

even a small amount. If you can’t afford it, other possible incentives 

include refreshments, public recognition, or something to take home. 

Recruit your members. Ideally, those invited should be a 

representative sample of those whose opinions you value. Keep the 

focus group small, usually six to eight people, to encourage active 

participation by all group members. 

°

°

 ̇

 ̇

 ̇

 ̇

°

°

°
Measuring Excellence 67

An overview of the data-collection methods described in this section, including a 

summary of the advantages and disadvantages of each, is provided in Exhibit III7. Because each data-collection method has its pros and cons, you should consider 

using a mix of both quantitative and qualitative methods in your evaluation. 

In addition to describing data-collection methods, this section offered practical 

advice to develop data-collection protocols to meet your evaluation needs. Refer 

to the Instrument Development Checklist in Appendix F for more guidance as 

you create your own evaluation tools. 

Methods Pros Cons Costs Time To 

Complete

Expertise 

Needed

Document 

Review

Objective; least 

obtrusive

Access to data 

may be tricky; data 

can be difficult 

to interpret; 

data may be 

incomplete

Inexpensive 

for an internal 

evaluation

Depending on the 

volume of data, 

may require a lot 

of time

Some expertise 

required to access 

and interpret the 

data

Interviews Gather in-depth, 

detailed info; data 

can be used to 

generate survey 

or focus group 

questions

Takes time and 

expertise to 

conduct and 

analyze

CInexpensive 

for an internal 

evaluation

About 45 minutes 

per interview; 

analysis can be 

lengthy

Good interview/

conversation skills; 

qualitative analysis 

skills

Surveys Can be conducted 

anonymously; 

standardized data 

collection

Selecting an 

appropriate 

sample may be 

difficult

Moderate Moderate, depends 

on method of 

distribution (i.e., 

mail, telephone, or 

in-person)

Some training 

needed to 

systematically 

administer and 

collect survey data

Focus Groups Can get 

information 

quickly about 

needs, attitudes, 

and opinions

Can be difficult 

to recruit 

participants, 

facilitate the 

group, and 

analyze data

Inexpensive 

for an internal 

evaluation

About 90 minutes Good group 

facilitation/ 

conversation skills; 

qualitative analysis 

skills

Exhibit III-7

Data-Collection Methods at a Glance
68 Conducting an Internal Evaluation

Obtain the Necessary Clearance

Depending on the nature of your program and the evaluation you have designed, 

you might need approval from your agency leadership before moving ahead 

with data collection. Invasion of privacy and confidentiality issues are a serious 

concern, which is why academic researchers almost always send their survey 

protocols for pre-implementation review to an independent review board to 

certify that the research will not have an adverse impact on any human subject. 

Evaluation data often are very sensitive, and in some cases you might need to 

protect the confidentiality of program participants and obtain their informed 

consent. Informed consent allows participants to decide voluntarily whether to 

participate in the evaluation. Securing informed consent can involve describing 

the benefits that participants reasonably might expect to encounter, explaining 

alternatives to participating in the research, and telling participants the extent to 

which their personal information will be confidential. An example of an informed 

consent statement that you can modify for your evaluation needs is in Appendix 

G. To ensure that data are handled appropriately as they are collected, limit the 

use of data that personally identifies an individual (such as name and address) and 

keep evaluation data (e.g., notes, papers, or cassette tapes) under lock and key.

Plan for Collecting and Tracking Your Data

Once you have finalized your data-collection protocols and procedures and are 

ready to begin, you will need a way to manage data collection and organize 

the data you gather. A data-collection plan should be written and included as a 

subset of your overall evaluation plan. The plan should list a schedule of required 

data-collection activities matched to the appropriate data-collection instrument 

and person responsible for each task. An estimate of the level of effort necessary 

to complete each task also might be included. Strategies for data analysis can be 

linked to each data-collection protocol or strategy, as well (see Appendix H).

In addition, you should think through processes for entering, tracking, storing, 

and securing your data. As you collect data, there are several procedures that will 

allow you to manipulate easily and understand your data once you begin analysis. 

Several tips for ongoing data collection and management areas follows:

Data tracking. It is important to track carefully the data you collect. For 

instance, you can create a chart that includes the title of the instrument, 

the source of the data (e.g., program participant, community resident, 

program document), how the instrument was administered (e.g., focus 

group, mailed survey), and the period of data collection. 

°
Measuring Excellence 69

Data entry and management. You will need to create a system for 

entering and managing your data. A database system will allow you to 

know at a glance how many people have responded, when, and if there 

are any missing participant data. Your database also will be used to carry 

out more complicated analysis. Before entering data into a computer from 

paper documents, be sure to make a copy of the form and you can make 

edits or comments on the copies. If the data are in electric form to begin 

with, make a backup copy on your hard drive or keep separate copies of 

your database on floppy disks or compact discs (CD).

Data storage and security. Think about where and how you will store 

and secure the data you collect. If you collect data in hard copy form, you 

will need to decide where to keep the forms. Store written surveys and 

program documents in a place safe from damage or loss. For electronic 

data such as databases or interview transcripts, back up hard drives or 

keep separate copies of your documents on a floppy disk or CD.

By following these tips while you collect information, you will be well organized 

to analyze your data and learn about your program. Data analysis is the focus of 

the next section in this chapter.

Step 3: COMPARE—Data Analysis 

Once your data have been collected, analyze them following procedures 

that reduce and categorize the information to make it easier to manipulate, 

understand, and report. Always begin your analysis by reviewing your evaluation 

purpose and specific research questions, then proceeding to analyze data to 

yield results that are easily understandable. To be useful to stakeholders and 

decision-makers, data analysis should lead to straightforward, readily interpretable 

information about program processes and effectiveness. 

°

°

Evaluate 

the program/initiative

Compare

 and analyze results against 

baseline and/or objectives

Specify (Plan)

what is to be collected

 and how

Collect 

information with 

appropriate tools
70 Conducting an Internal Evaluation

Techniques for analyzing data range from basic counts, to content analysis, 

to complicated statistical analyses. Databases, spreadsheets, and statistical or 

qualitative analysis software often can facilitate steps in data analysis, including 

creating a computer database file to enter data, coding and entering them, 

cleaning data, using a computer package to analyze them, and interpreting results. 

Depending on the type of data you collected, you might conduct quantitative 

data analysis, qualitative data analysis, or both. Simple procedures for analyzing 

qualitative and quantitative data are discussed in this section.

Quantitative Analysis

Using statistical analysis to examine quantitative data (i.e., data that can be 

counted or assigned a number) can be difficult. Law enforcement agencies that 

have been successful with complex statistical analysis report having sophisticated 

data-tracking systems and using standard analysis techniques and evaluation tools. 

For law enforcement agencies that do not have the expertise or resources for data 

analysis, or when sophisticated statistical analyses are needed, consider hiring a 

consultant for guidance. 

Simple frequencies, percentages, and cross tabulations often are all that is needed. 

The following a few tips for analyzing quantitative data that do not require 

complicated statistical procedures: 

Clean your data (i.e., review data for missing or incorrect responses). 

Tabulate the information by counting the frequency of its appearance. 

For example, if you want to know how many people have ever 

attended a neighborhood watch meeting, simply count the number 

of “yes” responses to the survey question, “Have you ever attended a 

neighborhood watch meeting in your area?”

If it is ratings and rankings you are interested in, you can compute the 

mean (average), median, or mode. You can find the mean, or average, by 

dividing the sum of all responses to a particular question by the number 

of responses to that question. You can generate the median score by 

arranging the responses in a list. The middle number is the median. If 

there are two middle numbers, then add the two numbers and average 

them for a median. Last, you can tabulate the mode by looking for the 

number or value that occurs most often.

During analysis, keep in mind the following questions:

What do the raw data tell me?

Are the results low, average, or high?

Are there any red flags or extreme values to be aware of? 

What do they mean?

°

°

°

°

°

°
Measuring Excellence 71

Qualitative Analysis

As with quantitative data analysis, qualitative data analysis is a process for 

reducing the amount of data that you have as well as aggregating and organizing 

individual responses into themes that allow you to identify data trends. 

Qualitative data include information from interviews, focus groups, written 

documents, observations, and open-ended survey questions. While qualitative 

data analysis can seem less straightforward than quantitative analysis, there are 

ways to analyze qualitative information systematically. Much of your analysis will 

be focused on providing a mechanism for organizing and describing the themes, 

patterns, activities, and content of the study rather than on illustrating causal 

linkages between processes and outcomes (Patton, 1990). 

While it is not within the scope of this guide to delve deeply into the strategies 

and nuances of qualitative data analysis, many handbooks and manuals exist for 

those who are interested in learning more.1 Basic steps in qualitative analysis are 

as follows:

Review all the data (e.g., interview transcripts or focus group notes; 

program documents).

Organize and label (code) responses into similar categories or themes. 

Try to identify patterns or associations among the data; look for 

similarities and differences across respondents (e.g., all law enforcement 

officers who attended training have a positive attitude toward the 

program, but officers who did not attend training think the program is a 

waste of time).

Identify the meaning that has emerged from the data (i.e., ask yourself, 

“What are respondents telling me through these data?”).

Interpret the data in relation to the issue of interest (i.e., ask yourself, 

“What does this mean in light of my evaluation questions?”).

Description is the foundation of all qualitative data reporting. Several options for 

organizing and reporting descriptive findings (Patton, 2002) are summarized in 

Exhibit III-8.



 For more information on qualitative analysis, we encourage you to consult Miles, M.B. and A. M. Huberman. 

Qualitative data analysis. Sage: Thousand Oaks, California, 1994. Another excellent reference is Patton, M.Q. 

Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002.

°

°

°

°

°
72 Conducting an Internal Evaluation

Exhibit III-8

Options for Organizing and Reporting Qualitative Data

Analytic Framework Approaches

Processes: Qualitative data can be organized to describe important 

processes. For example, if the department stations an officer in a public 

housing project to reduce youth violence, describing the process of 

how the officer begins interacting with the community, especially 

young people, might become an important theme that could be used 

to frame evaluation questions or analyze the data gathered.

Issues: An analysis can be organized to illuminate key issues, often the 

equivalent of the primary evaluation questions; for example, variations 

in how participants changed as a result of the program. With the 

public housing example above, building trust is an important issue. 

The evaluation could explore various tactics that the officer used to 

promote trust and how perceptions and behaviors changed on both 

sides. 

Questions: Responses to interviews can be organized question 

by question, especially where a standardized interviewing format 

was used. For example, the evaluator might want to ask the same 

questions of the officer and community residents, and then organize 

the responses by question, allowing readers to compare police and 

community responses.

Sensitizing Concepts: When sensitizing concepts such as “trust” 

versus “mistrust” might have played an important role in directing 

data collection, the data can then be organized and described through 

these concepts. 

Source: Patton, M.Q. Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002.

°

°

°

°

Analysis of Program Costs and Benefits

The need to link criminal justice program costs to outcome measures has become 

critical as policy makers increasingly require decision-making tools to allocate 

scarce funding. Systematic economic valuation of program costs and dollar 

benefits improves the basis for making funding decisions. 
Measuring Excellence 73

Cost Analysis

Cost analysis consists of a thorough examination of the type and amount of all 

resources used to support a law enforcement agency’s activities. Determining the 

cost figures for the agency as a whole, or in its component parts, is a basic form 

of cost analysis. Tangible program items such as salaries and equipment are given 

a dollar value. Costs typically are provided at several levels, from the total cost of 

the agency to the costs associated with each area of responsibility.

Cost-Effectiveness and Cost-Benefit Analyses

There are two levels of cost analysis evaluation: cost-effectiveness and cost-benefit. 

The former assesses whether the money for a program was well spent to reach 

the program outcomes, while the latter assesses costs of achieving the intended 

impact.

Cost-effectiveness is the relationship between agency costs and its effectiveness 

in providing community safety. Cost-effectiveness is a technique in which costs 

are measured as dollars spent, whereas effectiveness is measured in changes 

in outcomes (alternatively, outcome counts can be used). The overall costeffectiveness of an agency is improved by first determining the activities of the 

agency that contribute most to effectiveness and then determining which of 

those activities have the lowest cost. For example, the cost-effectiveness of a law 

enforcement agency might be measured as the number of offenders apprehended 

in one month versus the costs of various programs that an agency uses to 

accomplish this (e.g., community outreach, officer training, crime mapping).

Cost-benefit analysis compares costs and benefits using the same monetary 

units. Determining the dollar value of a benefit is the primary challenge in 

cost-benefit analysis. The benefits of law enforcement activities are many and 

include decreased social disorder, decreased fear of crime, and better or more 

public services to respond to crime. Another prime benefit is decreased crime, 

resulting in decreased justice costs and lower costs to potential victims and society.2

The criminal justice literature provides estimates for such of benefits. 

During cost-benefit analysis, the present value of future benefits must be 

calculated and compared to present investment costs. Economists use discounting 

when benefits will be received in the future to make future monetary benefits 

comparable to current dollar values. To accomplish this, future benefits are 

discounted by some factor—the higher the rate, the lower value of future benefits. 

The discount rate is important because offender benefits might not be realized for 

years. Generally a rate between 3 and 5 percent is used. To determine a return on 

investment, the cost-benefit analysis will present results for benefit-cost ratios and 

net present values.

2

 An example can be found in Cohen, M.A. “Measuring the Costs and Benefits of Crime and Justice” (pgs. 

263–315). In Measurement and Analysis of Crime and Justice. Washington, D.C.: U.S. Department of Justice, 2000. 
74 Conducting an Internal Evaluation

If we think of the domestic violence example used in Chapter II, both approaches 

require totaling program costs, often a combination of grant funding and regular 

operating costs. A cost-effectiveness analysis would explore whether the money 

invested was justified by the major outcomes—the number of PPOs issued and 

the number of police personnel trained. A cost-benefit analysis would attempt to 

answer whether money expended on the program was justified by a reduction in 

domestic violence cases, including deaths and injuries. As this example suggests, it 

is easier to conduct a cost-effectiveness analysis than a cost-benefit analysis, in part 

because it is difficult to quantify deaths and injuries prevented. Ascribing a dollar 

amount to those costs also is problematic.
Measuring Excellence 75

Chapter Summary

While the part-time or full-time participation of an evaluation 

consultant could enhance the strength and objectivity of evaluations of 

policing programs, many departments can and should perform more 

evaluation activities by relying on their internal resources. This chapter 

detailed the major steps in the evaluation cycle and provided practical 

information and advice to help you conduct an internal evaluation of 

your policing strategies. 

In the first section of this chapter, the myriad components of 

evaluation planning were discussed, including evaluability assessment, 

stakeholder involvement in the evaluation process, defining program 

goals and objectives, and determining the logical relationship among 

program inputs, outputs, and outcomes. Tips were offered for 

developing evaluation questions and selecting an appropriate design 

based on evaluation planning activities.

This chapter also provided guidance about ways to identify and 

implement data-collection strategies beyond traditional police 

measures. Material presented in the section on data collection included 

a brief summary of the pros and cons of different data-collection 

methods as well as tips for developing data-collection procedures and 

protocols such as surveys, interview guides, and focus group guides. 

The importance of obtaining necessary clearances and permissions to 

collect data from program participants was underscored. Last, simple 

procedures for making sense of your data were discussed in the section 

on data analysis. 

Now that you are familiar with all the steps involved in planning 

and implementing an evaluation, you should be able to create your 

own evaluation plan. A sample evaluation planning template can 

be reviewed in Appendix H and used to assist you in developing 

and documenting your general evaluation plans. The following 

chapter provides useful strategies for managing internal and external 

evaluations of policing activities. 

Chapter IV:

Managing Evaluations

Measuring Excellence 79

Managing Evaluations

There are two types of evaluations in which law enforcement agencies are likely 

to be involved—internal evaluations and external evaluations. As noted in 

Chapter III, internal evaluations are those in which police agencies rely primarily 

on employees within the department to conduct evaluations and disseminate 

results. In some instances, agencies have research or evaluation divisions (or 

staff) specifically to support the development and implementation of evaluation 

within the agency. On the other hand, external evaluations are those for which 

the agency hires an outside expert to conduct the research. Generally, external 

evaluators are hired when the agency does not possess the expertise or resources 

to conduct evaluations internally or when a particular policing program, 

intervention, or strategy requires an evaluation design that is more complex than 

the internal evaluation staff are prepared to handle. 

While internal and external evaluations are conducted for different reasons, many 

of the steps required in planning and managing them are very similar; although 

each has its own unique set of challenges as well. This chapter first outlines the 

steps required to plan and manage an internal or external evaluation; the section 

that follows outlines some of the challenges unique to managing each type of 

evaluation. Finally, this chapter provides important information about how to 

hire an external evaluator, including assessing evaluator capacities, what to look 

for in an external evaluator, and sources for finding external evaluators. 

How Do I Manage an Evaluation?

The planning and ongoing management of evaluation activities, whether internal 

or external, can be as important as the methodological aspects. While there is no 

single recipe for managing an evaluation, law enforcement administrators who 

do must possess a basic understanding of evaluation for appreciating the work to 

be conducted, assisting in its planning, and identifying weaknesses in the design 

or data-collection plan. In addition, there are management issues to consider at 

critical junctures prior to, during, and following an evaluation. These are outlined 

in more detail below. 

What do I need to consider prior to the evaluation?

If you are assigned to manage the evaluation, you might consider obtaining 

input from several colleagues, particularly those with an evaluation background 

or experience with the program or strategy the evaluation will target. If possible, 

also involve groups that are in some way connected with the program or strategy, 

but are not affiliated with the law enforcement agency. These might include 

community members (in the case of community outreach activities), stakeholders 

(those supporting or funding the policing activity or strategies), and program 

staff. Finally, involving the leadership within your agency will help ensure support 

at the highest level. 
80 Managing Evaluations

Before conducting an evaluation, determine the evaluation requirements,

develop the evaluation plan, and set up administrative and monitoring tools. 

These activities are described in the section that follows. 

Determining the Evaluation Requirements

Before anything else, think about the reasons for your evaluation. Are you 

considering it because you want to use the information to improve a particular 

program or strategy? Do you want to know if something is producing its intended 

outcomes? Is the evaluation a requirement of a grant or other contractual 

relationship? Understanding the reasons will help you determine the best 

approach. For example, if the goal is to modify or improve a program or strategy, 

you might consider an evaluation that is focused on process rather than outcome. 

Or, if the evaluation is a grant requirement, you must develop an evaluation 

plan that meets the information needs of your contract with the funding source. 

Most important, determine if an evaluation even is feasible by considering several 

issues: 

Are the data you need to answer the research questions available?

If available, can you access them? For example, school records may be 

available but school rules or consent procedures may prohibit you from 

accessing them. 

Do you have the financial and other resources to conduct an 

evaluation? If internal, this includes having staff with the time and 

expertise necessary to conduct the evaluation. If external, this includes 

having access to an experienced evaluator who can work within time and 

budget constraints. In either case, financial resources must be available to 

support evaluation costs. 

Do you have buy-in and support from your agency administrators? 

If the leaders in your agency (e.g., your chief or executive) do not 

support and approve of the evaluation, it may preclude you from fully 

implementing or funding it. 

Once you have determined that an evaluation is both necessary and feasible, you 

can proceed to the next step—developing the evaluation plan. 

Developing the Evaluation Plan 

Developing an evaluation plan, whether you are managing an internal or external 

evaluation, requires several important steps. Some of these are outlined in 

detail in Chapter III and will not be covered here, including determining what 

will be evaluated, establishing goals and objectives, developing logic models, 

developing evaluation questions, and determining the appropriate design. 

Other management tasks that are an important part of an evaluation plan will 

be explored here: determining the evaluation budget and timeline, identifying 

resources available to conduct the evaluation, and devising a plan for using and 

disseminating evaluation results.

°

°

°
Measuring Excellence 81

Budgets and Timelines

Developing budgets and timelines are very different tasks. Because your budget 

will have an impact on the duration of your evaluation, and vice versa, developing 

one without consideration of the other is not recommended. On the other 

hand, each one also has unique circumstances that require consideration during 

planning. With this in mind, we present each issue separately. The final paragraph 

ties the two together. 

Budgets 

When thinking about your evaluation budget, take into account several categories 

of expenses. Labor typically is the most costly evaluation expense, whether your 

evaluation is internal or external. Internal labor expenses include staff time and, 

for full-time staff, associated costs such as overhead (e.g., electricity and rent) 

and fringe benefits (e.g., insurance, retirement, and vacation time). If you use 

an external evaluator, you will not need to estimate overhead or fringe benefits, 

but you will likely require more financial resources than if you were using 

internal staff because external evaluators, especially academic or other highly 

qualified professional evaluators, generally are more expensive than internal 

staff. Depending on the extent of the evaluation, an external evaluator may cost 

you approximately 20 percent of your overall budget. For example, you might 

consider setting aside $20,000 if your full budget is $100,000. 

A more efficient method of estimating external costs is to solicit cost proposals 

from several evaluators. Cost proposals require that each bidder describe in detail 

the tasks to be completed, the number of hours each task will take, and the 

costs associated with each, including labor. You should receive several bids that 

are approximately the same. If you receive bids that are very different from the 

rest, you can assume they either under- or overbid the work. In all cases, your 

resources are finite so be sure you have considered all possible costs and that they 

are fully addressed in the cost proposal of the bidder you choose. Unanticipated 

costs can derail an evaluation quickly, particularly if the costs are associated with 

activities that are critical to the evaluation (e.g., data collection or analysis).

In addition to labor, evaluation budgets also can include incentives or rewards 

for participants, if appropriate, and other indirect costs (e.g., postage, telephone, 

duplication). Travel expenses (e.g., airline tickets, rental cars, per diem) also must 

be estimated; be careful, because extended travel expenses can add up quickly. 

In addition, you can never fully anticipate costs associated with cancelled and 

rescheduled flights, and stays in expensive hotels when the preferred (and less 

expensive) ones are full or cannot accommodate a government rate. If data can 

be collected locally rather than off site, your travel expenses can be reduced 

drastically. Finally, consider whether the evaluation will require any special 

equipment, varying from standardized instruments (e.g., existing surveys and 

questionnaires available for purchase) and tape recorders for data collection, to 

data analysis and management tools such as software and computer upgrades. 
82 Managing Evaluations

As you estimate evaluation expenses, be certain you have the resources to fund 

your evaluation before you take any steps toward implementation. There typically 

are several options for funding evaluation activities, including the following:

Agency budgets. Many law enforcement agencies have a line item in the 

budget to cover evaluation activities. If this is the case in your agency, be 

clear about how much money is in that budget, the types of activities it is 

designed to fund (some set-asides are earmarked for specific categories of 

expenses such as labor or data collection), and if other officers intend to 

use the money for their own evaluations. 

Grant funds. Many grants require the grantee to set aside a certain 

percentage of the grant funds (most often 10 percent) to cover evaluation 

expenses. Unfortunately, these funds often are insufficient to cover all 

related evaluation expenses. As a result, if you are using grant funds for 

your evaluation, you may need to consider additional sources of funding. 

Outside funding sources. Government, private, and some nonprofit 

agencies and foundations offer grant funds specifically for evaluation 

activities. These funds are offered most often in the context of advancing 

knowledge in a particular area or to support the implementation and 

evaluation of best or promising practices. Therefore, you may have 

to search to find an evaluation grant program focused on funding 

evaluations in your program area. 

Timelines

The requirements of your evaluation most often determine its duration. For 

evaluations that are designed to meet funding requirements, the funder and the 

duration of the grant generally determine the evaluation timeline. For example, 

if you are awarded a 3-year grant to implement a community policing strategy 

in a local neighborhood, you might need to measure and report outcomes of 

interest to the funder at the end of the 3 years. In the interim, the funder could 

require that you document certain components of the strategy, and/or measure 

short-term or intermediate outcomes and submit the findings in quarterly or 

semiannual reports. In this situation, the evaluation as funded is expected to 

terminate at the end of the grant period and will continue only if you choose to 

and have resources to support ongoing evaluative efforts. 

For evaluations that are not bound by funding requirements (e.g., those that are 

funded using agency dollars or are designed to monitor program progress), it 

could be up to you to determine the timeline. If this is the case for your program 

or strategy, there are several issues to consider. First, think about what outcomes 

the program or strategy is designed to achieve and when those outcomes 

realistically might be realized. 

°

°

°
Measuring Excellence 83

For example, if you are implementing a substance abuse awareness program in 

the local high school, you might expect knowledge about the dangers of drug use 

to be evident in the participants immediately after they complete the program. 

Lower substance use rates in the high school population, however, could take 

longer to achieve; in fact, they might not be evident for years. Determine which 

outcomes your evaluation is intended to achieve and when they will occur, and 

develop your timeline accordingly. If you conduct an impact evaluation (e.g., 

demonstrating long-term outcomes), extend the evaluation timeline outward to a 

point at which these outcomes can be realized. On the other hand, if you conduct 

an outcome evaluation (e.g., demonstrating immediate, short-term outcomes), 

your evaluation timeline will be more closely related to the duration of your 

program or strategy. For example, if you implement a program or strategy that is 

10 weeks long, you should gather valuation data right before (pre-test) and right 

after (post-test) program implementation. 

Despite their differences, budgets and timelines should be developed 

simultaneously. The money and other resources that are available for the 

evaluation will have implications for how long it can continue. Whether internal 

staff conduct your evaluation or you hire external evaluators, you must know the 

extent and duration of available resources. You could have ample resources for 

evaluation one year, but as agency priorities shift, funds might be reallocated in 

future years. In addition, you might not start new evaluation projects if you know 

the chief is retiring or the governor is running for reelection. Changes in political 

structure sometimes mean law enforcement priorities change and budgets can be 

realigned to reflect those changes. You do not want to discover midway through 

your evaluation that funds have been exhausted. Be certain from the outset what 

funds are available and for how long so you can implement your evaluation 

completely and successfully. 

Evaluation Resources

Even if your organization maintains a research and evaluation division through 

the annual budget, you might need to hire an external evaluator or outside 

consultants to assist internal staff with complicated evaluations or those that 

exceed your in-house capacity. Therefore, you could require evaluation funds in 

addition to those in your existing budget. Or, your organization could be one of 

the many law enforcement agencies that lack not only an evaluation unit but also 

an evaluation budget. Fortunately, regardless of your situation, several evaluation 

resources are available, including the following: 

Writing a grant proposal to request evaluation funds. Increasingly, 

government and foundations are willing to support thoughtful evaluations 

that have the potential to identify and foster promising practices.

°
84 Managing Evaluations

Building your agency’s internal evaluation capacity. Training and 

technical assistance (TA) often is available for staff development on 

evaluation-related topics. Providers include trained evaluators, academics, 

and individuals employed by federal agencies or professional evaluation 

associations. Because TA providers often are supported by state or 

federal grants, many of these resources are available free of charge or 

for a nominal fee. Check with agencies from which you currently 

receive program or other funding to learn if they offer evaluation TA 

opportunities to their grantees. See Appendix L for a list of technical 

assistance resources.

Reviewing existing law enforcement evaluation reports for best 

practices and lessons learned. An inexpensive way to learn about 

evaluation is by reading evaluation reports and related materials. 

Evaluation reports often are available to the public and can be a good 

source of information regarding evaluation strategies that have been 

implemented in your field and those that have been particularly 

successful. In addition, hundreds of evaluation resources are available 

on the Internet, including manuals, step-by-step guides, and technical 

reports. Most of these materials are either free of charge or can be 

obtained for a nominal fee. See Appendix L for a list of evaluation reports 

related to policing.

Other law enforcement agencies. Some law enforcement agencies 

have research or evaluation divisions staffed with competent evaluators 

who also are law enforcement agents. These individuals might be willing 

to assist you and your staff to plan and conduct evaluations, share 

information about what has worked for them, or provide technical 

assistance to build the internal capacity of your agency. 

Graduate students. Graduate students often are required to conduct a 

research study as part of their graduation requirements. Master’s degree 

students must complete a thesis, while doctoral students are required 

to prepare a dissertation, and they often will take on an evaluation or 

other research study without being paid. In addition, because the work is 

being conducted to fulfill graduation requirements, a team of academics 

generally is assigned to oversee and monitor the project from planning 

through completion, ensuring a high-quality and rigorous evaluation. 

°

°

°

°
Measuring Excellence 85

Questions to Guide Your 

Internal Monitoring Plan

How will you maintain frequent contact with your evaluation staff?

How will you require your evaluation 

team to document its progress?

How will you learn of problems 

promptly?

Do you need a project manager or 

primary staff person to oversee the

evaluation?

°

°

°

°

Utilization and Dissemination Plan

The likelihood that evaluation findings will be used increases with 

comprehensive planning, preparation, and follow-up. An evaluation 

might meet standards for rigorous design, careful data collection and 

analysis, and a well-written report, but if it does not have practical 

implications and is not used, it has failed. When an evaluation proves 

to be useful, it usually succeeds in answering a specific, clearly defined 

question that someone wanted answered. As evaluation manager, you 

must make sure the appropriate evaluation questions are asked. Soliciting 

stakeholder input about what they need from the evaluation and what 

would be most important for them to know about the program or 

strategy ensures that evaluation results will not simply sit on a shelf 

somewhere. 

As evaluation manager, you also should have an information 

dissemination strategy to fit stakeholder needs, using a variety of 

communication methods, which might include formal reports and 

briefings as well as informal strategies, if appropriate. The dissemination 

strategy could identify the purposes and stakeholder audiences, which 

might include program staff and supports, community members, 

funders, policymakers, and the media. 

Developing Monitoring Plans

During planning, establish procedures to ensure that you can monitor 

both the evaluation work and budget effectively, regardless of whether 

the evaluation is conducted internally or externally. For example, how 

and when will your external evaluator submit invoices for payment? 

What means will you use to monitor the work 

of your internal evaluation staff? These issues are 

discussed in detail in the section that follows. 

Monitoring Procedures

Monitoring your evaluation effectively requires 

you to supervise your staff, their work, and the 

budget. The procedures you use, however, depend 

on whether you are managing an internal or 

external evaluation. 

Internal Evaluations

Having frequent and regularly scheduled 

meetings with internal staff is a good way 

to monitor the extent to which they are 

implementing the work plan as intended and 
86 Managing Evaluations

are staying on track with the evaluation timeline. Staff meetings also provide 

opportunities for identifying potential problems and solving them before they 

grow. Frequent contact with your evaluation staff also allows you to monitor 

projected labor hours. If your staff are spending more hours than you have 

allocated for them (or fewer), you may need to modify your work plan and/or 

find additional resources (e.g., money or staff) to assist with implementation. 

You also can monitor internal staff through e-mails and telephone conversations. 

These methods can be used in between staff meetings to keep abreast of progress 

and to solve problems routinely. 

Also keep in mind how your internal team will document progress, including 

challenges to implementation that might disrupt your timeline. You might set 

aside time during regularly scheduled staff meetings for oral progress reports 

that are structured in content, refer to a specific period (the 2 weeks prior to the 

meeting, for example), and might cover major activities, problems or challenges, 

action steps taken, and future activities. If you plan to have staff report orally, be 

sure to have someone document what is said, perhaps the staff person responsible 

for taking meeting minutes. A written record of what is happening in your 

evaluation is important as a reference if your funder or chief has questions about 

the evaluation or if problems arise. 

In developing a plan for monitoring internal staff, decide if you want to designate 

someone other than yourself as the evaluation manager or primary staff person 

(e.g., this might be the most senior person on the evaluation team or the one with 

the most evaluation experience). Designating someone as the project manager not 

only provides a point of contact your staff can turn to when you are not available, 

but also identifies one person whom you (or your chief or funder) can call when 

questions or problems arise. You do not want to waste time figuring out which 

staff member to call when your chief or a federal project officer is on the line 

with a question about the evaluation. In addition, a project manager is someone 

to whom you can delegate certain responsibilities to assist you to do your job 

more effectively. For example, you can ask your project manager to facilitate staff 

meetings and report to you about progress and problems.

External Evaluations

Compared to an internal evaluation, managing an external evaluation can 

demand more effort in advance to select and hire a qualified evaluator, to work 

collaboratively with the evaluator throughout the evaluation, and afterwards to 

review the report and disseminate findings. Many of the techniques for managing 

internal evaluations, however, also apply to external evaluations. 

In-person meetings and frequent telephone and e-mail contact also are good 

methods for monitoring external evaluators. Monitoring an external evaluator, 

however, also poses a unique set of challenges. 
Measuring Excellence 87

First, external evaluators are just that—external. You do not have the same 

opportunities to meet with and monitor their progress as you would with internal 

staff. Nonetheless, you can offset this problem by making regularly scheduled 

meetings a part of the contract with the evaluator. You might stipulate that 

monthly (or weekly or biweekly) evaluation meetings will take place either inperson or by telephone, and that these meetings will be used to discuss progress 

against the work plan, problems with or challenges to implementation, and 

budgetary matters. In the contract, also outline the consequences if certain 

requirements are not met. For example, you can reserve the right either to cancel 

the contract or withhold payment until requirements meet with your satisfaction. 

Should the evaluator fail to appear for meetings or be otherwise unavailable to 

you throughout the evaluation, you can initiate the agreed-on consequences. We 

also recommend that you require some type of progress report from your external 

team. Written progress reports are a standard monitoring method and, as with 

invoicing, you will want to determine how, and how frequently, you will require 

that your evaluator submit these. In the contract with your external team, you 

also should state clearly the purpose, content, and timeline for these reports. 

Determine how you will monitor the evaluation budget of an external evaluator. 

Will you require monthly or quarterly invoicing? More frequent invoicing might 

allow you to monitor your budget more closely but, if your budget is small, less 

frequent invoicing may be acceptable. Also consider 

what details you want each invoice to contain. Some 

contracts require little detail about how the money was 

spent or expenses were incurred, while others specify 

all items must be clearly identified and documented. In 

addition, you should know if your agency has standard 

invoicing procedures for an outside contract so that 

you can adhere to them. 

In managing an external evaluation, your contract 

is your leverage; therefore, it must state clearly each 

requirement and how it will be monitored. Be sure 

your evaluator or team understands the contract 

requirements before they sign it. If, like many large 

law enforcement agencies, your agency has a contracts 

department, consider involving the department in 

contract development and negotiations. Contracts are 

legally binding documents and can be challenged in a 

court of law; therefore, make sure your contract meets 

the standards for contracts of its kind. These steps will 

contribute to the leverage and documentation you 

need to monitor your external evaluator effectively. 

Questions to Guide Your 

External Monitoring Plan 

Have you contacted the evaluator’s 

references? 

Do you have a context for his or her 

previous work?

Does the contract stipulate: How 

often and by what means meetings 

will take place? What consequences 

will be implemented if the contract is 

violated? 

Do you have a plan for monitoring the 

budget? Does it allow you to identify 

problems immediately? 

How will you learn of problems 

promptly so that they can be resolved 

efficiently and immediately? 

Have you identified a primary contact 

person with whom you can work in 

the principal investigator’s absence? 

°

°

°

°

°

°
88 Managing Evaluations

During the Evaluation...

Conduct frequent meetings with your evaluation team.

Review project expenditures.

Follow up immediately on problems or other 

concerns. 

Provide feedback about the evaluation to 

the team and your other stakeholders.

Be prepared to make program modifications 

if early findings are not promising.

°

°

°

°

°

You can use administrative procedures such as those suggested above to keep 

your external evaluator in check. However, nothing beats having a positive 

working relationship, which facilitates communication between you and your 

evaluator and, in turn, ensures that critical information is available to you in a 

timely manner for smoother process monitoring. You can increase the likelihood 

of a positive relationship with your evaluator by hiring an individual or agency 

you know, that you (or someone you know) have worked with in the past, or 

is referred to you by someone you know and trust. If this is not possible, do 

not hesitate to ask potential hires for past evaluation reports and references. 

Past reports let you see the kind of work done previously and identify agencies 

or individuals who have contracted with the person for similar work. If these 

agencies or individuals are not given as references, do not hesitate to contact 

them yourself. Also, be sure to contact the references that are provided, not only 

asking questions about the candidate’s work, but also about work style. You also 

might ask for additional references to others for whom the candidate has worked. 

Contacting these individuals or agencies will give you additional information 

about the credibility and integrity of the potential evaluator. 

What Do I Need to Do During the Evaluation?

For the most part, what you do during the evaluation is to implement the 

monitoring techniques you developed during the planning phase. For internal 

evaluations, this includes conducting regular staff meetings, conducting 

e-mail and telephone communication with your evaluation staff, and reviewing 

project expenditures. If you have an internal project manager, that person can 

handle many of these activities and report to you. For external evaluations, hold 

regular meetings with your evaluator either in person or by 

telephone; review progress reports and invoices, and follow 

up on any issues that emerge. During the evaluation, be sure 

you are meeting other demands as well. For example, if you 

are required to give periodic progress updates to your chief or 

funding agency, you must plan for these activities, especially if 

they must be in writing. 

During the evaluation, find ways to provide feedback to the 

team (and those individuals and agencies involved with the 

program or strategy under study) about how things are going 

and what initial findings are emerging. As evaluation manager, 

you will have the most comprehensive information about the 

study and the outcomes being achieved. Keeping your team 

informed is important because, in completing day-to-day 

evaluation activities, staff can lose sight of the “big picture.” 

Providing staff, whether internal or external, with frequent 

updates about the evaluation not only provides you with an 

opportunity to acknowledge their hard work, but also helps 

them remain focused on overall project goals. 
Measuring Excellence 89

Despite your best intentions, a variety of things can go wrong during an 

evaluation, whether it is internal or external. Staff turnover at the program 

site (if your data are being collected by program staff or those responsible for 

implementing your strategy directly) and on the evaluation team, low program 

participation rates and poor program implementation can derail even the best 

evaluation plans. If you can anticipate these events, you can respond to them 

appropriately and quickly. However, some issues, such as staff turnover, are 

difficult to anticipate. Generally, staff do not make public their intentions to leave 

a position, at least not with enough time to plan for their departure. Therefore, 

in addition to building a relationship with your evaluator, you should maintain 

contact with those close to the program or strategy being evaluated (e.g., a project 

director or program staff person). These relationships help you learn important 

information about events you might otherwise not obtain (e.g., a staff person is 

leaving the agency). 

What Do I Need to Do After the Evalaution? 

Your evaluation responsibilities do not end with submission of the final report. 

You will need to implement several important activities after the evaluation ends 

to ensure that your evaluation has resulted in the type of information you need to 

understand. This means not only how effective your program or project was but 

also what improvements need to be made to increase the likelihood of achieving 

positive results in the future. These activities are outlined in detail in the sections 

that follow. 

Review the Evaluation Report

The evaluation manager has primary responsibility for responding to and using 

evaluation results. This responsibility encompasses reviewing the report to 

determine that it is a clear accounting of the evaluation process and results and 

that it meets the reporting requirements set out in the contract. Specifically, the 

evaluation manager should review the evaluation’s key findings, conclusions, and 

recommendations. Most well-written evaluation reports include six sections: 

Executive summary. An executive summary should contain a very brief 

description of the evaluation’s purpose, questions, procedures, most 

important findings, implications, and recommendations. It should not be 

more than a few pages in length.

Introduction. The introduction should contain the purpose of the 

evaluation and the overview of the report contents. The introduction 

sets the stage for the rest of the report by outlining basic evaluation 

purposes and introducing the report contents that follow. It often 

states the intended audiences for the report and cautions the reader 

about limitations that might have affected the collection, analysis, or 

interpretation of information.

1.

2.
90 Managing Evaluations

Description of the evaluation focus. The third section, the focus of the 

evaluation, should contain a description of the program being evaluated 

and the evaluation questions. This section typically discusses the rationale 

for the evaluation, who it is intended to benefit or serve, activities and 

strategies followed in implementing the evaluation, settings, and resources 

devoted to the program. Also in this section are the specific questions that 

shape the evaluation. You might want to include a copy of your program 

logic model here.

Evaluation plan and procedures. The evaluation plan and procedures 

section contains a description of the evaluation design, data-collection 

strategies and instruments, and an overview of data analysis. This is 

the section in which the evaluator presents the methodological and 

technical information to establish evaluation adequacy and credibility. It 

is helpful to outline the information being collected to answer each of the 

evaluation questions and describe the procedures for data collection and 

analyses and include any instruments as appendices.

Presentation of evaluation results. The presentation of evaluation 

results section should include a summary of evaluation findings and an 

interpretation of them. More extensive details related to the findings can 

be included in supporting appendices. The specific criteria being used to 

judge the results should be stated explicitly so they can be viewed in light 

of these standards. For example, if the standard you are using to judge 

effectiveness is citizen satisfaction for your area, data should indicate 

whether citizen satisfaction has increased, decreased, or stayed the same. 

Interpretation of the findings should reflect a careful process of synthesis 

and analysis that supports judgments and subsequent recommendations. 

It is helpful to relate the findings clearly to the specific evaluation 

questions.

Conclusions and recommendations. The conclusions and 

recommendations section should outline the implications of the 

evaluation findings for your program and specific recommendations 

for program modifications or improvements. Organize conclusions 

by strengths and weaknesses that focus attention on both positive and 

negative judgments. Descriptions of strengths and weaknesses must 

contain sufficient detail to give stakeholders an understanding of the 

rationale and to serve as the basis for recommendations.

Disseminate Evaluation Findings

After evaluation data have been collected and analyzed, communicating the 

results is critical if they are to be used effectively. When overseeing an evaluation, 

the evaluation manager should seek opportunities to share results with 

stakeholders and program managers throughout the process, but the final results 

are often what stakeholders want.

.

.

.

.
Measuring Excellence 91

Specific strategies for developing appropriate evaluation reports for many 

stakeholder groups are presented below.

Program staff and supporters

Usually, the people who are involved in implementing your program believe that 

it works well and brings about positive results. They will be most interested in 

what the evaluation can tell them about ways to improve the program’s function 

and about how participants feel about the program. Reports that include detailed 

information on these topics will be most appropriate for this audience.

Funders. Funders want to be sure a program is making good use of 

the resources they provide and the program is accomplishing its goals. 

Therefore, when reporting evaluation findings to funders, present 

information regarding the implementation and specific outcomes of your 

program. 

Policymakers. These people make strategic decisions about the program 

itself and tend to be interested in the “big picture” rather than being 

concerned with details. A brief document highlighting the findings that 

address program effectiveness is appropriate for them.

Media. One of your main objectives in delivering findings to the media 

might be to gain visibility for your program. Media outlets tend to run very 

brief, straightforward stories and probably will not be interested in detailed, 

extensive reports. Your best bet in working with the media is to provide a 

short, concise document that includes an overview of your program, the 

particular findings you want to highlight, and the implications of those 

findings for your community and future policing efforts.

How you use your findings depends greatly on your audience. While written 

reports and briefings are common strategies for communicating evaluation results, 

you should also consider using other techniques. The most useful evaluation 

products are often presentations to decision makers or community groups, but 

a brief summary report can be distributed to a wider audience, press releases 

can provide more exposure to evaluation findings, and web sites can include 

visual supplements such as photos and color charts. Web sites, e-mail messages, 

question-and-answer statements, press releases, op-ed items in newspapers, 

speeches, written testimony, newsletters, articles in professional journals, brownbag lunches, videotapes, or computerized evaluation presentations are increasingly 

being used as communication tools and should be considered when deciding how 

to disseminate evaluation findings (USAID Center for Development Information 

and Evaluation, 1997). 

°

°

°
92 Managing Evaluations

Apply the Evaluation Results

Evaluation findings can be used in several ways, depending on your organization 

and what evaluation stakeholders deem appropriate. Generally, findings can be 

used in the following ways: 

For internal improvement. Evaluation results can be used in internal 

agency decision-making to improve ongoing implementation of a 

program or refine strategies based on lessons learned about challenges and 

successes.

For accountability. Findings can demonstrate to stakeholders that the 

program is having a positive impact or is particularly cost-effective. 

Evaluations often are used to secure grants or convince supporters that 

the program is functioning as planned. For example, to maintain or 

secure funding, a granting agency might ask for proof that your program 

is achieving its stated goals. Findings can show that the program is 

successful in meeting its outcomes. If you conduct a process evaluation, 

the report also can document how funds or resources are being used.

For outreach. Evaluation results can be shared with the media and 

directly with the community to increase visibility for your initiatives. 

Strategies include releasing findings to the media, organizing town hall 

meetings to discuss results, providing information to school and faith 

community newsletters, and building an information web site. 

Challenges to Managing Evaluations 

While most evaluations run smoothly if well planned and monitored, there are 

challenges associated with managing an evaluation, whether internal or external. 

These include the following: 

time. Be clear about how much time you need for management activities 

and how your time can be spread across the various tasks for which you 

are responsible, including those not associated with the evaluation. As 

suggested earlier, if your time is limited, consider appointing a project 

manager to whom you can delegate a portion of the management 

activities. In addition to your time, also consider staff time. Do you have 

staff with time to implement the evaluation effectively? Generally, law 

enforcement officers are very busy individuals with little time to spare. If 

you do not have the internal capacity to conduct the evaluation, consider 

hiring someone from the outside. 

Resources. When budgets are prepared each year, every available dollar 

is assigned to some activity or function. If evaluation is not part of your 

agency’s annual budget already, funding your research will be the most 

challenging aspect of your management duties. In addition, you might 

find that funding the evaluation fully is not a one-time event, but requires 

attention throughout the study. 

°

°

°

°

°
Measuring Excellence 93

Competence. Whether managing internal or external evaluation staff, 

be confident they have the necessary skills to implement the evaluation 

effectively. Assessing the team’s competence is an issue not only before 

the evaluation begins, but also as it progresses. Evaluation staff require 

methodological and analytic skills and also must possess the capability 

to work cooperatively with other people, groups, and agencies. You 

might have a clear understanding of their evaluation skills in advance, 

but their ability to collaborate with others most likely will emerge over 

time; therefore, have mechanisms in place to monitor evaluation staff 

competence. In addition, you need good management skills to handle 

the evaluation appropriately. Have you ever managed a budget before? 

Supervised staff? If not, consider appointing someone to manage the 

evaluation for you. 

Support. In law enforcement agencies, any activity outside the realm of 

law enforcement calls for support from the administration. If your chief is 

not convinced of the value of evaluation, you will be hard-pressed to find 

the resources you need to implement your research. If evaluation is not an 

agency priority, you can promote it by taking advantage of opportunities 

to educate the administration and other officers about the benefits of 

evaluation. Be sure you have done your homework and be prepared 

with examples showing how evaluation findings increased an agency or 

program’s visibility or generated additional funds for the agency. 

While these challenges exist to some extent in most law enforcement agencies, 

they can be overcome or at least managed effectively. Think through these issues 

and ways to resolve them when planning the evaluation. If you do not plan for 

them, these challenges are more likely to interfere with your ability to garner the 

support you need to carry out your evaluation. 

Should I Hire an External Evaluator?

If you have decided to conduct an evaluation but your agency does not have an 

internal evaluation unit (or it does have such a unit, but the staff do not possess 

the necessary expertise), you might need to hire an external evaluator. This section 

of the guide offers tips for decision-making and selection when hiring an external 

evaluator and addresses four fundamental questions: 

How do I know if I should hire an external evaluator?

Why should I hire an external evaluator?

What do I look for in an external evaluator?

How do I find an external evaluator?

°

°

1.

2.

3.

4.
94 Managing Evaluations

How Do I Know if I Should Hire an External Evaluator?

During the planning stage, assess the evaluation capabilities within your agency to 

determine if you can use internal staff or if you need to hire an external evaluator. 

Exhibit IV-1 presents questions to help you decide if it is in your agency’s best 

interest to hire an external evaluator for a specific project. 

If the majority of the answers to these questions are “no,” then you should 

consider hiring an external evaluator to assist with planning and implementation. 

Even if you answered “yes,” review the sections in this guide that detail the 

competencies, roles, and responsibilities of the evaluator to identify the expertise 

within your internal team.

Why Should I Hire an External Evaluator?

To assist you further in making the decision, this section, including Exhibit 

IV-2, compares the advantages of hiring an external evaluator to the potential 

drawbacks of relying on internal staff. 

Capability Question Circle Yes or No

Does your agency have a person on staff who 

knows how to conduct needs assessments and/or 

evaluations? 

Yes No

Does a staff person from your agency know enough 

about evaluation to conduct one with the support of 

resources (e.g., manuals, toolkits, textbooks, guides, 

informal partners) that are available in the field?

Yes No

Do staff have enough time to devote to conducting a 

meaningful evaluation?

Yes No

Does your administration (e.g., chief or executive) 

support using internal staff to conduct an 

evaluation staff?

Yes No

Exhibit IV-1

Assessing Evaluator Capabilities
Measuring Excellence 95

Despite the advantages, there are potential challenges associated with hiring 

an external evaluator. Among these, expense and fit are discussed here along 

with suggestions for dealing with these possible pitfalls.

Expense. Conducting a thorough evaluation can be costly and 

time-consuming, especially for an evaluator who is not familiar with 

your community or program. To alleviate the start-up factor for an 

external evaluator, consider hiring someone from your community 

or field who has a basic understanding of the issues associated with 

law enforcement and your program or strategy. The 10 percent 

set-aside for evaluation that often is associated with grant awards can 

°

Advantage External Evaluator Internal Staff

Specialized 

Knowledge

Comes with specialized knowledge and 

ability; understands how to assess the 

needs of a community, document program 

outcomes, and collect and analyze. 

Often does not have expertise 

in the scientific approach to 

evaluation.

Reduces 

Burden

Reduces the burden on the internal staff 

who have limited time to devote to a 

meaningful evaluation.

May be focused on law 

enforcement activities and 

not have time to conduct a 

meaningful evaluation also. 

Objective 

Viewpoint

Remains unbiased about the program 

being studied; a good evaluator will point 

out both positive and negative aspects of a 

program, offering suggestions on how best 

to achieve the program’s goals.

Can be committed to the work 

and believe unequivocally 

in the effectiveness of the 

program, thus running the 

risk of looking at evaluation 

findings in a biased way.

Higher Level of 

Credibility

More knowledgeable and objective about 

the evaluation process; conclusions and 

recommendations tend to carry more 

weight with funding institutions, the field, 

and the public.

Not as knowledgeable about 

evaluation; findings often 

viewed as less credible to the 

field.

Exhibit IV-2

Advantages of Hiring an External Evaluator Compared to 

Using Internal Staff
96 Managing Evaluations

be used to help cover the expense. In the long run, using the 10 percent 

set-aside to hire an external evaluator with specific expertise could be 

less expensive and produce better results than trying to do the work with 

existing staff.

Poor Match. Hiring an unqualified evaluator, or one who does not work 

well with your organization, can produce potentially damaging results 

such as alienating staff, intruding on the community and staff-community 

relationships, and drawing incorrect conclusions about the community 

and/or program. To find a proper match, take the time necessary to 

select the best evaluator for the job to ensure that the evaluation is not 

burdensome and that evaluation findings address research questions about 

how the program and services are working.

Carefully weigh all these factors when deciding to hire an external evaluator or 

rely on evaluation capabilities within your agency or partnering organizations to 

develop and carry out the evaluation plan.

What Do I Look for in an External Evaluator?

After careful deliberation, if you decide to hire an external evaluator, it often is 

helpful to organize a hiring committee. The committee should be large enough 

to address the issue under study and the dynamics of your agency adequately, but 

small enough so decision-making is efficient and not overly troublesome. The 

hiring committee should report to the appropriate parties (via periodic e-mails, 

newsletters, meetings, etc.) to keep everyone informed of the selection process.

The following sections provide guidance for considering the evaluator’s 

competencies, roles, and responsibilities. These also can be applied to internal staff 

when determining their competencies to carry out internal evaluation work.

Evaluator Competencies

Hiring committee members should become familiar with the qualifications 

necessary for an external evaluator. Evaluators should possess the following skills:

Identify data that can be used for planning, managing, and evaluating 

program policies and practices

Use multiple research methods and collect various types of data from a 

variety of sources

Establish data-collection and analysis priorities that are informed by the 

experiences, capabilities, and resources of program staff and participants

Produce reliable interpretation of data

Link data to program development and management.

°

°

°

°

°

°
Measuring Excellence 97

In addition to these basic skills, evaluators must be willing to work within the 

realities of everyday program operations. Strong skills in communication, team 

building, group processes, and negotiation are vital to a successful relationship 

between the evaluator and your agency. Given the focus of your evaluation, it 

also is important to consider the following competencies when hiring an external 

evaluator:

Proficiency in facilitating an evaluation process that includes the 

experiences of the participants

Ability to build capacity in program staff to use evaluation findings

Willingness to develop and maintain a collaborative relationship based on 

cooperation and trust with program staff, managers, and stakeholders

Willingness to involve all partners in the evaluation 

Ability to design and implement methods to provide evaluation 

information to the program for use in planning, monitoring, and 

improvement

Experience in evaluating the type of programs, activities, or strategies that 

your agency is implementing 

Experience in conducting evaluations with law enforcement agencies 

Experience in working in culturally diverse settings with individuals of 

varying backgrounds and languages.

Find an evaluator with aptitude in as many of these competencies as possible. 

While evaluation success is not contingent on finding an evaluator with a high 

level of expertise in each of these competencies, it will benefit the project overall 

if you take time to secure an evaluator with the skills that meet your needs. Doing 

so will increase the probability of a meaningful evaluation. 

 How Do I Find an External Evaluator?

Now that you have a better understanding of the type of external evaluator who 

would be right for your project, you are ready to search for the person to fill that 

position. This section of the guide provides information on where to look and 

what to include in a job announcement.

Where to Look for an External Evaluator

When hiring an external evaluator, you might consider an individual or 

organization that either is referred to you or that you have worked with in the 

past. Alternatively, there are several sources you can consult to find a qualified 

evaluator. Potential sources, and the advantages and disadvantages of using each, 

are summarized in Exhibit IV-3.

°

°

°

°

°

°

°

°
98 Managing Evaluations

Evaluator 

Source Pros Cons

College or 

university

Educational institutions offer the expertise of faculty 

members or graduate students at low to moderate 

overhead costs. Many small, not-for-profit organizations 

have found low-cost data-collection assistance through 

colleges or universities. With proper training, graduate 

students (the more inexpensive option) can create 

instruments and collect and analyze data. Contact college 

or university behavioral/social science, criminal justice, 

social work, or public health departments to learn if they 

have faculty and/or graduate students to assist you.

Independent researchers not working 

within a larger organization might 

be challenged to meet the variety of 

expertise and time commitments to 

complete the job within a grant cycle.

Independent 

researcher

Researchers who work alone or with minimal staff can 

provide high-level expertise at low overhead costs. They 

commonly have the flexibility to be located on site, 

enabling frequent interaction with initiative participants.

May be focused on law enforcement 

activities and not have time to 

conduct a meaningful evaluation. 

Research and 

evaluation 

organization

Organizations with research and evaluation departments 

offer a range of expertise. They are able to meet the 

multiple needs of an initiative by offering experts across 

disciplines and having state-of-the-art technology in 

house.

These organizations typically are 

associated with higher overhead 

costs.

Exhibit IV-3

Sources for Finding an External Evaluator
Measuring Excellence 99

What to Include in a Job Announcement

The evaluator job announcement should include an overview of your project, 

the roles and responsibilities of the evaluator, suggested timeline for completion 

of work, estimated budget information, delineation of preferred skills, and a 

request for applicant background and education information. Preparing the job 

announcement should be the responsibility of the hiring committee or at least it 

should be reviewed and approved by those involved in developing and managing 

the evaluation. A sample job announcement is provided in Appendix K.

Following the application submission deadline, the hiring committee should 

meet to review applications and select candidates to interview. Appendix I offers 

examples of interview questions. After committee members select a candidate they 

believe is right for the job, they should solicit feedback on their recommendation 

from all partners. Scheduling follow-up interviews to assess the personal style and 

characteristics of the potential evaluator helps ensure that the individual hired 

is acceptable to program staff, administrators, and key stakeholders. Interview 

more than one applicant in case your first choice is unavailable or does not accept 

your offer. Once an offer is accepted, engage the evaluator as soon as possible in 

evaluation planning. If you already have developed an evaluation plan, review 

the document with the external evaluator and make any changes necessary. 

Once your plan is finalized, you and your evaluator are ready to proceed with 

implementation.

Additional Resources

Two additional resources for finding an external evaluator are the American 

Evaluation Association (AEA) and the Evaluation Center at Western Michigan 

University. See also Appendix L.

American Evaluation Association (www.eval.org) 

The AEA is an international professional association of evaluators devoted to 

the application and exploration of program evaluation, personnel evaluation, 

technology, and many other forms of evaluation. AEA provides links to help you 

find an evaluator and information on training opportunities.

The Evaluation Center at Western Michigan University

(www.wmich.edu/evalctr)

The Evaluation Center’s mission is to advance the theory, practice, and utilization 

of evaluation. The center’s principal activities include research, development, 

dissemination, instruction, and national and international leadership in 

evaluation. The center provides service to a selected group of clients—including 

school districts, nonprofit agencies, government agencies, foundations, and 

colleges—to evaluate their programs. The center also provides evaluation training 

to the field.

Chapter V:

Emerging Topics in 

Evaluation Mangement

Measuring Excellence 103

Emerging Topics in Evaluation 

Management

Three emerging topics in evaluation and program accountability are presented in 

this section: 

Collaborative evaluation. An evaluation conducted jointly by more than 

one office, agency, or partner. 

Participatory evaluation. An evaluation that features active involvement 

by all stakeholders.

Performance measurement. An ongoing process of documenting 

indicators of progress toward goal achievement. 

Collaborative and participatory evaluations have similarities, particularly because 

both involve stakeholders in the process. Performance evaluation provides an 

opportunity for police departments to apply program evaluation results to 

internal decision-making on critical issues such as staffing.

Collaborative Evaluation

“If the process of evaluation is carried out regularly and openly, with all group 

members participating, the answers they arrive at are in themselves not so 

important as what is learned from the discussion and from the process of reaching 

consensus on what questions should be used to evaluate outcomes and capacity.” 

(Uphoff, 1991) 

Many of the barriers to evaluation, including a lack of skills, tight budgets, and 

limited access to data, can be addressed through a collaborative approach. Such 

an approach entails forming a team that shares decisions about the purpose 

of evaluation, how it will be conducted, and how results will be interpreted 

and used. Generally, the team includes individuals, agencies, or organizations 

that have some vested interest in the program, strategy, or activity that will be 

evaluated. For example, if your law enforcement agency is planning to implement 

a drug use prevention program in schools, your collaborative evaluation team 

might include representatives from your agency and the funding source in 

addition to representatives from the local school district (administrators and 

teachers), parents, substance abuse service providers, and other groups that might 

be involved in planning or implementing the evaluation. In addition, if you do 

not possess the expertise to facilitate evaluation planning and implementation, 

be sure to include an evaluation person. The key to the collaborative approach to 

evaluation is that no single person, agency, or organization owns the evaluation; it 

belongs to the team of individuals who have come together to support it. 

1.

2.

3.
104 Emerging Topics in Evaluation Management

In developing this guide, we asked law enforcement representatives if they 

had used collaborative approaches to evaluation before and, if so, what their 

experiences had been. Most reported using this approach with positive results. 

In fact, they claimed several benefits, including access to more financial and staff 

resources (across agencies), the capacity to conduct more sophisticated evaluations 

(because of combined talents), and the opportunity to develop relationships with 

various organizations, agencies, and community and neighborhood groups. In 

addition to those reported by local law enforcement agencies, a collaborative 

approach to evaluation could have other benefits. According to Practical 

Evaluation of Public Health Programs (The University of Texas-Houston Health 

Science Center, 2001), a collaborative approach to evaluation accomplishes the 

following: 

Reduces suspicion and fear. Including stakeholders in decision-making 

about what to evaluate and how the evaluation will be conducted is likely 

to prompt support, whereas those who do not participate can become 

suspicious about the project and fearful of the outcome. They might not 

trust the results or those involved in the evaluation. People often fear that 

evaluation will result in termination of the program or their jobs, but 

involving them helps them understand that the focus of the evaluation is 

to improve the program.

Increases awareness and commitment. Through a collaborative 

approach, participants become more aware of, and committed to, the 

evaluation process. If people are involved, they have an interest in the 

results. A candid presentation and discussion of the program’s data can 

spur development of a consensus about interpretation and follow-up 

action.

Increases the possibility of achieving objectives. If people understand 

what is being evaluated and why, they are more likely to work toward 

improving those elements of the program. The collaborative approach 

promotes ownership of the process and responsibility for the outcomes.

Broadens knowledge. A collaborative approach draws on broader 

knowledge, skills, and experience in evaluation. In choosing people for the 

evaluation team, the role each person will play and the knowledge, skills, 

and experience they bring to the evaluation should be considered. This is 

a good opportunity to look both inside and outside the organization to 

tap resources needed for the type of evaluation planned. The team should 

count people from a variety of backgrounds, including those with front-line 

service experience, statistical and epidemiological expertise, management 

and policy perspectives, and planning skills. All these skills might not be 

available within the program or even the organization, so it is often helpful 

to rely on a community college or university in your area for assistance 

either from faculty or student interns.

°

°

°

°
Measuring Excellence 105

Teaches evaluation skills. When people work together, they share ideas, 

knowledge, skills, and abilities. Evaluation team members can learn 

about program objectives, data-collection methods, making evaluation 

decisions, and even how to work on a team. Team members learn by 

doing and come away from the process with an enhanced set of skills. 

Teaches stakeholders. A collaborative approach can teach community 

stakeholders about policing efforts. Involving people from outside the 

program increases their knowledge about your program and its role in 

law enforcement. This is an excellent way to involve the community in a 

discussion of law enforcement problems and possible interventions. For 

example, you might solicit participation from local elected officials and 

representatives of community-based organizations, all of whom would 

finish the evaluation with a better understanding of your program’s 

strengths, the constraints under which law enforcement programs 

operate, what is needed for improvement, and how the program relates to 

their interests.

Increases the possibility that findings will be used. Involving 

stakeholders increases the possibility that the findings will be used 

and implemented. When a variety of staff are involved in conducting 

evaluations, identifying problems, and determining solutions to problems, 

they are more likely to accept implementation of the solutions decided on 

by the group.

Allows for different perspectives. Recalling the old adages, “two heads 

are better than one” and “the whole is greater than the sum of its parts,” 

the collaborative approach includes people from outside the program who 

have unique points of view.

°

°

°

°
106 Emerging Topics in Evaluation Management

How They Did It: 

Minneapolis Police Department

The Hawthorne Huddle, a 1999 winner of the Herman 

Goldstein Award for Excellence in Problem-Oriented 

Policing, is a community-based initiative in which citizens; 

the faith community; the Minneapolis Police Department; 

city, county, and federal agencies; nonprofit organizations; 

and the General Mills Foundation worked together 

to develop a strategy to combat crime and improve 

neighborhood conditions. The Hawthorne Huddle used 

evaluation strategies throughout the initiative to identify 

(e.g., reviewing crime statistics and community concerns) 

and analyze (e.g., ongoing review of crime statistics and 

other community indicators) the problem, and develop a 

response. Many of the evaluation activities were conducted 

by Huddle staff and volunteers in a collaborative manner. 

In the fifth year, an external evaluation was conducted that 

was instrumental in documenting the accomplishments, 

lessons learned, and future goals of the initiative. The 

evaluation of this community initiative was managed by the 

General Mills Foundation, rather than by the Minneapolis 

Police Department. However, the results were used by all 

participating organizations, including the Minneapolis 

Police Department, to improve Huddle meetings, decide 

future strategic direction, and transition to new leadership.

The box below provides an example of a collaborative evaluation. As explained in 

this case study of the Hawthorne Huddle in Minneapolis, collaborative evaluation 

helped this community initiative demonstrate results, facilitate decision-making, 

and sustain its efforts. The evaluation managers stated that the participation 

of the Minneapolis Police Department in providing data throughout the 

implementation of the Huddle evaluation helped keep it focused on the needs 

of the community and the primary reason the Huddle was formed—to reduce 

community crime and violence. In addition, participation of the Minneapolis 

Police Department in Huddle activities and the evaluation helped strengthen 

working relationships among all participating agencies. These are only some of the 

benefits a law enforcement agency can gain 

from a collaborative evaluation. 

A more detailed description of the 

Hawthorne Huddle is on page 11. 
Measuring Excellence 107

Participatory Evaluation 

“When conducting [evaluation] in a collaborative mode, professionals and 

nonprofessionals become co-evaluators...Genuinely participatory approaches 

to evaluation require power sharing. One of the negative connotations often 

associated with evaluation is that it is something done to people. Participatory 

evaluation, in contrast, involves working with people. Instead of being research 

subjects, the people in the evaluation become co-investigators. The process is 

facilitated by the evaluator, but is controlled by the people in the program or 

community.” (Patton, 2002, pg. 183)

Participatory evaluation necessitates active involvement by all stakeholders and 

typically has several characteristics that set it apart from traditional evaluation 

approaches:

Participant focus and ownership. Participatory evaluations are oriented 

primarily to the information needs of program stakeholders rather 

than the law enforcement agency. The agency simply helps participants 

conduct their own evaluations, thus building their ownership and 

commitment to the results and facilitating follow-up action.

Flexible design. While some preliminary planning for the evaluation 

might be necessary, design issues are decided in the participatory process. 

Generally, evaluation questions and data-collection and analysis methods 

are determined by participants, not outside evaluators.

Participant negotiations. Participating groups meet to communicate 

and negotiate to reach a consensus on evaluation findings, solve problems, 

and make plans to improve performance.

Use of facilitators. Participants actually conduct the evaluation; however, 

external evaluators can provide a supporting role as mentors, group 

processors, or methodologists.

Learning process. Participatory evaluation emphasizes identifying 

lessons learned to help participants improve program operations and 

assess whether targets were achieved (USAID Center for Development 

Information and Evaluation, 1996).

 Each type of evaluation has its own strengths and limitations. Exhibit V-1 

presents some of the major differences between traditional and participatory 

approaches to evaluation. Some may be more appropriate than others, given 

circumstances and information needs. For example, if objectivity and credibility 

are key requirements, a more traditional external evaluation might be the right 

choice, with a hired evaluator who develops the plan and implements it with little 

input from you or your staff. If stakeholder ownership and program improvement 

are priorities, more collaborative or participatory approaches usually are better 

(USAID Center for Development Information and Evaluation, 1997). 

°

°

°

°

°
108 Emerging Topics in Evaluation Management

Some people criticize participatory evaluations as being less objective than more 

traditional approaches, while others suggest that a participatory approach can 

interfere with rigor when the evaluation requires complex analysis. Finally, some 

evaluators believe these approaches require more resources than more traditional 

approaches, because of the time and 

effort associated with bringing all 

stakeholders into the process. Regardless 

of the criticisms, your agency should 

determine the best approach to meet its 

information needs. 

How They Did It: 

Boston Police Department

The emergence of crack cocaine in the mid-1980s fueled 

an escalating crisis of youth violence in urban centers such 

as Boston. The Boston Gun Project was one of almost 20 

community-based prevention initiatives designed to reduce 

youth violence that were evaluated by a participant team from 

Harvard University. While other initiatives focused on issues 

ranging from drug treatment to job placement, the Boston Gun 

Project involved a law enforcement problem-solving approach 

that specifically targeted guns. The initiative used a crackdown 

approach (“pulling every [legal] lever”) to focus police attention 

on illicit gun sellers and the children who bought those 

weapons. The strategy included quickly arresting high-profile 

gangsters in the targeted high-crime district, even for trivial gun 

violations, making this crackdown a hot topic of conversation 

on the street. The idea was to send a loud and clear message 

that the only way to get the police off your back was to disarm. 

Operation Ceasefire officially began in May 1996 and within 

weeks there was a dramatic decline in youth violence—overall 

a 63 percent drop in the monthly number of youth homicides, 

a 25 percent decrease in the monthly number of citywide allage gun assault incidents, and a 44 percent reduction in the 

monthly number of youth gun assault incidents in the target 

district. At roughly the same time, the city adopted a strategic 

plan that included a neighborhood police initiative called 

“Same Cop, Same Neighborhood.” According to the Harvard 

researchers, in a 29-month period ending in January 1998, there 

had not been a single teenage homicide, something virtually 

unheard of in a major U.S. city at the time.
Measuring Excellence 109

An example of the participatory approach to evaluation comes from 

Harvard University’s David M. Kennedy, Anthony A. Braga, and 

Anne M. Piehl, who developed and implemented Operation Ceasefire 

in Boston. Elin J. Waring conducted the evaluation. Their report, 

Reducing Gun Violence: The Boston Gun Project’s Operation Ceasefire

(NCJ 188741, 2001), demonstrates the benefits of having the research 

team immersed in the initiative from the beginning. Instead of reading 

like a dry research report, the publication tells a magnificent story of 

how the police department became an integral and essential part of a 

citywide effort to save children’s lives.

The researchers helped design the intervention so they could determine 

whether any drop in gun violence was the result of Operation Ceasefire 

or another factor. The project launch was immediate and intense to 

ensure that the target community got the message. The official start date 

of Operation Ceasefire was May 15, 1996, the post-implementation 

phase began only 2 weeks later, on June 1, when the data already showed 

a dramatic drop in youth gun violence (see sidebar). The researchers 

identified and controlled for rival causal factors such as changes in the 

unemployment rate or the number of people classified as youth, as 

well as citywide trends in violent crime, homicides among older (age 

25+) residents, and youth involvement in street-level drug dealing (as 

measured by Boston Police Department arrests). Research participants 

credit this new kind of police-researcher partnership with enriching the 

entire process, from planning through evaluation. 

Participatory Evaluation Tradition Evaluation

Participant focus and ownership of the 

evaluation

Agency focus and ownership of the 

evaluation

Broad range of stakeholders participate Stakeholders have limited participation

Focus is on learning Focus is on accountability

Flexible design Predetermined design

Evaluators are facilitators Evaluators conduct the evaluation

Exhibit V-1

Difference between Participatory Evaluation and 

Traditional Evaluation
110 Emerging Topics in Evaluation Management

Performance Measurement

“Performance measurement is the ongoing monitoring and reporting of program 

accomplishments, particularly progress toward preestablished goals, and is 

typically conducted by program or agency management. Performance measures 

may address the type of level of program activities conducted (process), the 

direct products and services delivered by a program (outputs), and the results of 

those products or services (outcomes). A ‘program’ may be any activity, project, 

function, or policy that has an identifiable purpose or set of objectives” (Artley 

and Stroh, 2001).

Performance measurement is an ongoing process of assessing performance 

indicators to determine whether a program, activity, or strategy is achieving 

its objectives. It differs from program evaluation in several important ways, 

which are presented in Exhibit V-2. The main difference between performance 

measurement and evaluation is that the former is management-focused, while the 

latter is research-focused. Each is designed to answer different questions and is 

implemented for different reasons. 

Performance Measurement Program Evaluation

Management-focused activity Research-based activity

Continuous activity, tracking many projects 

across multiple fiscal years

Discrete activity, focusing on fewer projects for 

a limited period

Complexity at a minimal cost Complexity and cost are not major issues

General impact of grant program efforts Specific impact of grant program efforts

Evaluators are facilitators Evaluators conduct the evaluation

Exhibit V-2

Difference between Performance Measurement 

and Program Evaluation
Measuring Excellence 111

For example, a performance measurement system for a demand reduction 

education program in which law enforcement officers participate might include 

indicators such as number of juveniles in programs, number of adults in 

programs, and number of officers providing demand reduction training. Similarly, 

for multijurisdictional task force programs that integrate federal, state, and/or 

local drug law enforcement agencies and prosecutors to enhance interagency 

coordination and intelligence and facilitate multijurisdictional investigations, 

performance indicators might include number of offenders arrested, number of 

offenders prosecuted, number of drug seizures, quantity by weight (e.g., ounces, 

grams, dose units), drug type, and total value of funds and assets forfeited. 

Finally, programs designed to target the domestic sources of controlled and illegal 

substances, such as precursor chemicals, diverted pharmaceuticals, clandestine 

laboratories, and cannabis cultivation might have performance indicators such as 

number of offenders arrested, number of offenders prosecuted, number of labs 

eliminated, number of drug seizures, quantity of seizure by weight (e.g., ounces, 

grams, dose units), and drug type. 

The Colorado Springs Police Department’s (CSPD) performance monitoring 

system is based on the PASS (Police Accountability and Service Standards) 

model. Police service standards parallel the high-level organizational outcome 

goals for police service that the Colorado Springs community expects. CSPD 

service standards reflect the city’s strategic plan and the police department’s vision, 

mission, and values. Police service standards are described in terms of quality, 

quantity, cost, or time, and the seven categories are as follows:

Response times (e.g., response to emergency calls, answering telephone 

calls for police service). 

Officer deployment (e.g., scheduling and deployment efficiency). 

Traffic (e.g., crashes at top 25 targeted intersections). 

Clearance rates (e.g., violent and property crimes compared with national 

average). 

Drug and vice activity (e.g., response to methamphetamine labs, response 

to narcotics tips). 

Neighborhood policing (e.g., time consumed in proactive policing, 

partnerships). 

Citizen satisfaction with police services (e.g., customer satisfaction with 

officer contact, community perceptions of fear, safety, and disorder).

As noted by the chief of police and the city’s planning manager: 

“To achieve true accountability to the community, the police and the 

community must define police services and the level at which the services 

are provided. They must also identify measures of success for those services 

as viewed by their local communities. And finally, the police must be open 

1.

2.

3.

4.

5.

6.

7.
112 Emerging Topics in Evaluation Management

to being held accountable by the community for the measures of success 

agreed upon. Such an open dialogue between the police and the community 

will help gain the trust and confidence of all of our stakeholders, police and 

citizens alike” (Velez and Paine, 2002).

Based on data gathered through the performance system, informed decisions 

are made about service delivery, staffing levels, support personnel, manpower 

allocation, and deployment. The performance standards should reflect and guide 

realistic community expectations for police services. In commenting on the PASS 

model, the CSPD says the following:

“The other thing that this model (PASS) does is it keeps the pulse of the 

community. . . I think departments that don’t are missing out on a huge 

benefit, not only on monitoring their performance internally but also using 

that information to quantify why they need more officers, or that they need 

more police cars.” 

Performance measurement cannot replace data on program costs, political 

judgments about priorities, creative solutions, or common sense. A major purpose 

of performance measurement is to raise fundamental questions; the measures, by 

themselves, provide definitive answers.

Characteristics of a good performance measurement system include the following:

Quality over quantity. Performance goals should be relevant to the core 

mission of the program and the result the program is intended to achieve.

Importance to budget decisions. Achievement of performance goals 

should inform funding decisions.

Public clarity. Performance goals should be understandable to the users of 

what is being measured.

Feasibility. Performance goals should be feasible and based on relevancy 

of the outcomes.

Collaboration. Often agencies must work together to achieve a single 

outcome.

°

°

°

°

°
Measuring Excellence 113

Chapter Summary

Particularly as police departments embrace community policing, the benefits of 

collaborative and participatory evaluations become obvious. These approaches 

involve strategies that engage the community in the evaluation process, building 

opportunities to improve communication and promote trust, in addition 

to improving the scope and quality of evaluations themselves. Performance 

measurement is proving to be a valuable tool to tie program evaluation to 

municipal and police department decision-making about critical issues such as 

budgeting and staffing. Police departments that learn about and experiment with 

these techniques now will be fully prepared if evaluations are mandated in the 

future.

Police departments face mounting pressure to conduct quality evaluations and 

many federal agencies and private foundations want a strong commitment 

to evaluation demonstrated in competitive grant applications. While few 

municipalities so far tie funding to evaluation, this could change in the future. 

Police agencies that embrace evaluation now and experiment with a variety of 

techniques ensure that they will be ready for the future, while learning valuable 

lessons today. 

Appendixes

Measuring Excellence 117

Appendix A

Sample Needs Assessment Action Plan Template

1. Who needs to be included? Chief of police

Upper and middle management law enforcement 

personnel

Front-line police officers

Prosecutors

Court staff

Corrections officers

Women’s advocacy groups.

2. What do we want to learn 

about our community?

Rate of domestic violence within a jurisdiction.

3. What do we already know 

about our community?

Official police data including number of calls for service and 

arrests.

4. Do we need to collect 

new data?

Sources: Who will have these data?

Front-line police officers who work closely with the 

community, professionals and volunteers who work with 

victims at community hot lines, personnel at domestic 

violence shelters and hospital emergency rooms, and 

batterer intervention providers. 

Methods: How will we gather the information?

Focus groups, surveys, and interviews with beat cops. 

Strategies also might include targeted outreach to 

professional women’s groups. 

Instruments: What tools shall we use?

Focus groups, surveys, and interview protocols.

5. How will the data be 

analyzed?

Assess trends over time: Use data from calls for service and 

arrest as well as information collected from focus groups, 

surveys, and interviews with beat cops to determine 

whether the problem is getting better or worse. Use these 

data to examine what factors could be influencing changes 

over time. 

6. How will the findings be 

reported?

Create a brief write-up summarizing major findings. 

Communicate findings to community leaders, police 

personnel, and domestic violence advocacy groups. 

7. How will the data be used? Gain insight into factors that could be influencing the rate 

of domestic violence within a community. 

Needs Assessment Action Plan

Task/Activity Persons Responsible Due Date Anticipated Result/ 

Deliverable

Ex: Identify needs 

assessment questions

Ex: Law enforcement 

middle manager

Ex: October 15, 2006 Ex: List of needs 

assessment questions

Anticipated Needs Assessment Budget $__________
118 Appendixes

Appendix B

Preparing Measurable Goals and Objectives

The ABCDEs of writing measurable goals and objectives cover the who, what, to 

what degree, and by when information of your program.

Audience—Who?

The population/target audience for whom the desired outcome is intended.

Behavior—What? What is to happen?

A clear statement of the behavior change/results expected.

Condition—By when?

The conditions under which measurements will be made. This may refer to 

the time frame and/or implementation of a specific intervention.

Degree—By how much?

The quantification of, or the level of, results expected. This often involves 

measuring change compared to an identified baseline.

Evidence—As measured by?

The definition of the method of measuring expected change. The degree 

of change (set forth above) will be measured using a specific instrument or 

criterion.

Using the ABCDE method, these are the steps for developing goals and objectives: 

To develop measurable goals—

Step 1: Identify the longer term, global outcomes you wish to achieve.

Step 2: Identify each element (A, B, C, D, E).

Step 3: Formulate the goal statement using each necessary element. 

To develop measurable objectives—

Step 1: Identify the shorter term, more immediate outcomes you

 wish to achieve.

Step 2: Identify each element (A, B, C, D, E). 

Step 3: Formulate the objective statement using each necessary element.
Measuring Excellence 119

Sample Worksheet for Writing 

Measurable Program Goals and Objectives

Purpose: 

To develop measurable goals and objectives using the ABCDE method.

Directions: 

1. Identify the outcomes/impacts you wish to achieve. 

2. Identify each element of your goal and objective (A, B, C, D, E)

3. Formulate the goal and objective statements using each necessary 

element. 

GOAL: The longer term, global effects you wish to achieve.

Identify the elements:

A. Audience is... Citizens within a community.

B. Behavior to be changed is... Community response to domestic violence.

C. Conditions under which change will occur is (are)... Greater 

communication and collaboration among community organizations that 

work with domestic violence victims and offenders.

D. Degree of change expected is... Enhanced safety of domestic violence 

victims and their families.

E. Evidence of change is... Decreased domestic violence mortality rate 

within a community.

Goal Statement: 

To implement a Domestic Violence Enhanced Response Team (DVERT) to 

improve the safety of citizens within a community.

OBJECTIVES: The more immediate, direct results you wish to achieve.

 Identify the elements:

A. Audience is... Domestic violence victims and offenders and the agencies 

that work with these populations.

B. Behavior to be changed is ... Community response to high-risk-forlethality domestic violence victims.
120 Appendixes

C. Conditions under which change will occur is (are)... Enhanced 

coordinated community response to reports of domestic violence by 

creating a collaborative made up of agencies that work with domestic 

violence victims and offenders.

D. Degree of change expected is... Improve response time to reports of 

domestic violence and increase communication and collaboration 

among partners.

E. Evidence of change is... Appropriate containment of high-risk domestic 

violence offenders, improved response time to reports of domestic 

violence, faster intervention by domestic violence service providers.

Objective Statements: 

To increase communication and coordination between law enforcement, domestic 

violence service providers, batterer intervention service providers, the courts, 

probation and parole, and the district attorney’s office.

To produce greater batterer accountability by increasing batterer enrollment and 

attendance in offender treatment.

To connect domestic violence victims with services and resources immediately 

following a domestic violence incident.
Measuring Excellence 121

Appendix C

Planning Model Template

Draw seven columns on a sheet of paper and name them Column 

A, B, C, D, E, F, G.

Step 1: In Column A, list all background factors and conditions you think 

might influence the relationship between your program activities and goals. For 

example, lack of knowledge about the rights and needs of domestic violence 

victims would be a background factor. 

Step 2: In Column B, list program inputs. For example, the amount of funds 

that will go toward increasing the knowledge of the rights and needs of domestic 

violence victims would be an input. 

Step 3: In Column C, list program activities. For example, training law 

enforcement officers would be an activity.

Step 4: In Column D, list program outputs. For example, the number of law 

enforcement officers trained on the rights and needs of domestic violence 

victims would be an output. 

Steps 5 and 6: In Columns E and F, list all events occurring during or after your 

program activities that could affect how or whether you accomplish your goals. 

For example, an immediate outcome of the law enforcement officer training 

would be a better understanding of domestic violence. An intermediate outcome 

among law enforcement would be improved ability to assist domestic violence 

victims in obtaining a personal protection order.

Step 7: In Column G, list program goals. Listing program goals helps you see 

the overall effects of your inputs as well as the changes resulting from them. For 

example, one goal might be to decrease the incidence of domestic violence in your 

county. It is best to start with short-term goals because it often is difficult to measure 

and document long-term ones. 
122 Appendixes Sample Planning Model Template

Contextual Factors1. Agency budgets.2. Staffing.3. Political factors.4. Community interest or media 

attention to this issue.

5. Legislation.

Outcomes – ImpactsImmediate, Intermediate, Long-termShort-term changes we expect:

Medium-term changes we expect:

Long-term changes we expect:

Increased knowledge among police officers on ways to respond to domestic violence. 

Increased communication and collaboration between partners involved with the project (e.g., police officers and victim and offender service providers).

Increasing batterer enrollment and attendance in offender treatment.Connecting domestic violence victims with services and resources immediately following a domestic violence incident.

Conditions InputsProblem or need: What we invest: Improve community response to incidents of domestic violence

Time, money, and staff.

Activities OutputsWhat strategies or services comprise our program?

What are the direct products of our program activities?

Collaborative made up of police officers and victim/offender service providers.Training for police officers on how to respond to incidents of domestic violence.

DVERT collaborative meetings held two times/month.150 police officers trained on ways to respond to reports of domestic violence.
Measuring Excellence 123

Appendix D

Community Satisfaction Survey

The Fictitious Police Department wants to know about your attitudes, opinions, and 

experiences regarding our performance. All responses will remain confidential. Thank 

you for your time and support.

Please circle the answer closest to your views:

1. Do you feel the Fictitious Police Department has improved its police service 

during the past 5 years? 

 Yes No NA

2. How would you rate the competence of the Fictitious Police Department?

Excellent Good Fair Poor NA

3. Do you feel that Fictitious police officers act in a professional and courteous 

manner?

 Yes No NA

4. Are Fictitious police officers actively involved in working with citizens to 

make neighborhoods safer?

 Yes No NA

5. How satisfied are you overall with Fictitious police service?

 Very satisfied 

 Somewhat satisfied 

Neither satisfied nor unsatisfied 

Somewhat unsatisfied 

Very unsatisfied 

NA
124 Appendixes

6. How safe do you feel in your home?

Very safe

Somewhat safe

Neither safe nor unsafe

Somewhat unsafe

Very unsafe 

7. How safe do you feel on the street?

Very safe

Somewhat safe

Neither safe nor unsafe

Somewhat unsafe

Very unsafe 

8. Have you or a member of your family ever been a victim of a violent crime?

Yes No NA

9. Have you or a member of your family ever been a victim of a property crime?

Yes No NA

10. Have you interacted with a Fictitious police officer in the past year?

Yes No NA

 If yes, please check all that apply:

q Victim of crime

q Traffic stop

q I called police because of an emergency/crime

q Others called police to my home

q Neighborhood/school meeting

q Other (please explain) _________________________________
Measuring Excellence 125

11. Are you? 

Male

Female

12. Please circle your age category: 

Under 18 18-24 25-34 35-44 45-54 55-64 65 and up

13. How would you characterize your neighborhood?

High crime 

Low crime

Average level of crime

NA

14. Would you be willing to volunteer your time and talents to work with the 

Fictitious Police Department on community-based efforts to make our 

neighborhoods safer?

Yes No NA

If yes, please call xxx-xxx-xxxx or visit our web site at www.fictitiouspd.gov to find 

out about community meetings in your area and to volunteer.
126 Appendixes

Appendix E

Sample Focus Group Checklist

Focus Group Checklist

Consider the following items when you plan focus groups and develop focus 

group guides.

Advance Notice

Contact participants 1 to 2 weeks before the session.

Send each participant a letter of invitation.

Give the participants a reminder prior to the session. 

Over-recruit by three to five participants the number of participants 

necessary for each focus group.

Logistics

Make sure the room is satisfactory (size, tables, comfort, lighting, temperataure, etc.)

Arrive early to make necessary changes. 

Eliminate background noise as much as possible.

Bring nametags for participants.

Bring extra supplies like tapes, batteries, note pads, and extension cords.

Seat loud/disruptive participants next to the moderator.

Seat shy and quiet participants directly across from the moderator.

Serve refreshments before discussions begin.

Bring enough copies of handouts and/or visual aids. 

Set up a tape recorder and make sure that it works.

Questions

Introductory questions should be answered quickly and not identify personal 

characteristics of participants.

Questions should flow in a logical sequence.

Key questions should focus on the critical issues of concern.

Consider probe or follow-up questions.

Use open-ended questions.

Provide a summary of the discussion and invite comments.

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o
Measuring Excellence 127

Moderator Skills

Be well rested and alert for the focus group session.

Practice the introduction without referring to notes.

Make sure participants are comfortable and relaxed.

Ask questions with minimal reference to notes.

Be careful to avoid head nodding (i.e., showing support or favoritism 

for answers).

Avoid comments that signal approval (i.e., “Excellent,” “Great,” “Wonderful”)

Avoid giving personal opinions. 

Immediately After the Session

Prepare a brief written summary of key points as soon as possible.

Check to see if the tape recorder captured the comments. 

o

o

o

o

o

o

o

o

o
128 Appendixes

Appendix F

Instrument Development Checklist

Consider the following items when you develop your evaluation instrument.

Instrument Title

Use clear and concise words. 

State the program name. 

Indicate the type of service provided (e.g., counseling, shelter). 

Reflect the instrument method and content (e.g., survey, interview). 

Introductory Statement

Include information about the instrument’s purpose. 

Include information about how the data will be used. 

Include information about the level of confidentiality that will be 

arranged (e.g., who will see their responses, how responses will be 

reported). Always provide the “confidentiality” you promise. 

Demographics

Include questions that ask respondents for relevant information about 

themselves and their backgrounds (e.g., victim’s name, country of origin, 

and language). 

If necessary, include questions that ask about the person administering 

the instrument, if not you or program staff (e.g., observer’s name). 

If appropriate, identify the length of respondent participation in the 

program. 

Directions

Include general directions on how to complete the instrument itself (e.g., 

when, where, and how to return the instrument). 

Include specific directions on how to complete each section of the 

instrument. 

Make sure specific directions appear before each section. 

o

o

o

o

o

o

o

o

o

o

o

o

o
Measuring Excellence 129

Questions

Use language that respondents understand (e.g., “way to teach” vs. 

“pedagogical”). 

Avoid “double-barreled” questions (e.g., “Has there been an increase in 

program resources and the number of program participants?”). 

Allow enough space for respondents to write when using open-ended 

questions. 

Avoid biased and value-laden words or phrases. 

Include only questions asking for needed information. 

Keep question and answer options on the same page. 

Allow space for comments, concerns, or suggestions. 

Format

Use icons or graphics as clarifiers (e.g., “Please place a check in the 

appropriate box.”). 

Use a clearly legible font (e.g., Arial, Times New Roman). 

Lay out text and graphics using an entire page. 

Allow enough space between questions. 

Develop an instrument that it is pleasing to the eye (e.g., not “busy”). 

Indicate the date of instrument administration. 

Identify whether it is a pre-, post-, or ongoing survey. 

Note the name of the program/organization that developed the instrument in 

the footer (at the bottom of the page). 

Include a computer file location path of where to find a copy of the 

instrument in the footer (e.g., X:\Trafficking Victim Survey), if appropriate. 

Include the date of each new version in the header/footer. 

Pilot Testing

Clearly label a draft instrument “DRAFT.” 

Be mindful that advance permission to conduct the pilot test might be 

necessary. 

Arrange for pilot test participants and conditions to be as close to actual 

administration conditions as possible (e.g., time of day, location, methods, 

respondents). 

Source: The University of Texas at Arlington, School of Social Work. (n.d.). Instrument development checklist. 

Retrieved June 10, 2004 from www2.uta.edu/sswmindel/S6324/Class%20Materials/measurement/instrument 

development_checklist.htm

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o

o
130 Appendixes

Appendix G

Topics to Cover in an Informed Consent 

Statement for Research Participants

Invitation to Participate and Description of Project

Introduce yourself and describe the scope and purpose of your research.

Explain what agencies or groups are involved in funding and/or 

conducting the evaluation.

Tell the reader who is being targeted for participation in the research 

project.

Risks and Inconveniences

Describe the risks associated with participating in the research, if any.

Tell participants how much of their time will be required if they choose to 

participate.

Benefits

Explain why the research is important.

Note the benefits of participating in this research. 

Description of Procedures

Tell respondents what to expect if they agree to participate (e.g., how 

many interviews will be conducted with each person, where and when the 

focus group will be held; how much time it will take, whether any followup will be necessary).

Voluntary Participation and Withdrawal

Make sure respondents know that participation is voluntary and they can 

withdraw at any time without penalty. 

“Your participation is completely voluntary. You are free to choose not to 

participate in the study or to stop your participation at any time. You don’t have 

to answer any questions that you don’t want to, and you can withdraw your 

participation at any time without consequences or penalties.” 

o

o

o

o

o

o

o

o

o
Measuring Excellence 131

Confidentiality

Explain the steps you will take to protect the confidentiality of respondents.

“The information you provide will be kept confidential and will be used only for 

research purposes. We will not connect your name with your responses to the research 

questions. Research team members will not release any of the information you 

provide on this survey outside the confines of this research project. Your responses to 

the interview questions will be kept in a locked cabinet and will be seen only by the 

research team members. Your answers will be grouped with the other responses and 

summarized later in a report.” 

“Also, we would like your permission for the interview to be audiotaped. Please 

understand that the transcripts will be used only for research purposes and that no 

names will be associated with your transcript or audiotape.”

Questions

Please feel free to ask any questions and to think about whether you want to 

participate before you make a decision.

Do you agree to participate in this research study? 0. No 1. Yes

Can we audiotape this interview? 0. No 1. Yes

If you have further questions about this project or if you have a research-related problem, 

you may contact...

o
1

32 Appendixes Appendix H Sample Evaluation Plan Template 1. Focus What Will We 

Evaluate (Which Program Or Aspect Of A Program)? 

2.QuestionsWhat Do You Want To Know?

3. IndicatorsEvidenceHow will we know it? 

4. TimingWhen should we collect data?

Examine whether a police department’s Domestic Violence Enhanced Response Team (DVERT) is enhancing the safety of highrisk-for-lethality victims. 

1. Is DVERT improving safety of children and families?2. Is DVERT helping to ensure appropriate containment of high-risk offenders?

1. a. Client satisfaction surveysb. Interviews with DVERT partner agency staff (e.g., child welfare and domestic violence agencies) and focus groups with clients.2. a. Case records from District Attorney’s Officeb. Interviews with DVERT partner agency staff (e.g., child welfare and domestic violence agencies) and focus groups with clients.

Work with DVERT program management to determine a timeline for data collection that will not interfere with program activities.

5. Data CollectionSourcesWho will have this information?

MethodsHow will we gather the information? 

SampleWho will we question?

InstrumentsWhat tools shall we use?

Meet with District Attorney’s Office to determine accessibility of case records.

Client satisfaction surveysInterviews with DVERT partner agenciesDistrict Attorney’s Office Case Record ReviewFocus groups with DVERT clients.

Work with DVERT program management to identify staff to participate in interviews and staff who will be responsible for administering surveys to clients.

Develop client satisfaction survey protocol, interview protocol, and focus group protocol.

6. How will the data be analyzed?

7. How will the data interpreted?

Data from case records and satisfaction surveys will be analyzed using quantitative methods (e.g., descriptive analysis such as frequencies distributions and cross-tabulations).Data from interviews and focus groups will be analyzed using qualitative methods (e.g., descriptive analysis such as thematic coding).

Compare analyses from surveys, interviews, focus groups, and case reviews. 

8. How will the results be communicated?To Whom? When? Where? How?DVERT program managerHigh-level law enforcement administratorsBeat cops.

Create a brief summary of findings to be discussed during role calls and during DVERT management meetings.

Sample Evaluation Plan Template
Measuring Excellence 133

Appendix I

Sample Interview Questions Checklist

The following checklist contains a series of questions to help you interview and select 

an external evaluator. The questions are designed to help you judge the qualifications 

and characteristics of prospective evaluators, including basic training in evaluation, 

previous evaluation experiences, and some personal characteristics that fit your 

initiative and the community in which you will be implementing your program. In 

addition to the checklist, it will be important to look at work samples and contact 

references for any prospective evaluator.

Interview Questions

Evaluator appears to be: (Check one for each item)

Well-Qualified Not WellQualifiedCannot 

Determine

What formal training have you had in conducting needs 

assessment studies? What practical experience (ask for 

examples of needs assessment studies they have led or 

directed)?

What formal training have you had in conducting evaluation 

studies? What practical experience (ask for examples of 

studies they have led or directed)?

What experience do you have in evaluating policing 

programs?

What experience do you have in conducting evaluations of 

community initiatives?

What experience do you have in integrating evaluation 

activities with community planning processes?

What experience do you have in conducting evaluations 

with law enforcement (you may also ask about their 

experience working with any other agencies with whom 

you are partnering and with whom they would have to 

work)?

How familiar are you with the issue of _____ (insert the 

program/strategy area you will be evaluating, e.g., domestic 

violence)?

What indicators would you use for this type of evaluation? 

Source: Adapted from the W. K. Kellogg Foundation Evaluation Handbook and Evaluation 

Handbook for Projects Funded by S.T.O.P. Formula Grants.
134 Appendixes

Appendix J

Glossary

Archival data – Information that has already been collected by a department 

or organization, such as a government or nonprofit agency. These documents 

may include program records or materials such as proposals, annual or monthly 

progress reports, budgets, organization charts, memoranda, policies and 

procedures, operations handbooks, and training materials.

Baseline – Initial data on program participants or other program aspects 

collected prior to receipt of services or program intervention. Baseline data are 

often gathered through intake interviews and observations and are used later for 

comparing measures that determine changes in participants or a program.

Causality –A cause and effect relationship. The causality of two events describes 

the extent to which one event is caused by the other. When there is causality, 

there is a measure of predictability between the two events. 

Cleaning data – The process by which quantitative data are reviewed to identify 

missing or incorrect responses.

Comparison group – Individuals whose characteristics are similar to those of 

program participants or the treatment group. The individuals in the comparison 

group may not receive any services, or they may receive a different set of services, 

activities, or products. The experimental (or treatment) group and the comparison 

group are assessed to determine which type of services, activities, or products 

provided by a program produced the expected changes. 

Control group – A group of individuals whose characteristics are similar to those 

of program participants or an experimental group, but do not receive the program 

being evaluated. Participants are randomly assigned to either the treatment (or 

program) group or the control group. A control group is used to assess the effect 

of a program on participants compared to similar individuals not receiving the 

activities being evaluated. 

Cost benefit – An assessment of whether program costs are achieving the 

intended program impact. This analysis compares costs and benefits using the 

same monetary units. Determining the dollar value of a benefit is arguably the 

primary challenge in cost-benefit analysis.

Cost effectiveness – An assessment of whether money expended on a program 

was well-spent in reaching the program outcomes. This is the relationship 

between agency costs and its effectiveness in its mission. 

Data – Information or facts that are collected to answer specific research 

questions.
Measuring Excellence 135

Document review – A process of data collection that involves examining existing 

records or documents. 

Evaluability assessment – A process used to identify whether a program is ready to 

be evaluated. This is an assessment of whether or not a program has clearly defined 

activities and measurable goals and objectives that lend themselves to identifying data 

that can be collected. This process includes reviewing the program history, design, 

and operation, perhaps by watching the program in action; determining the program’s 

capacity for data collection; and assessing the likelihood that the program will meet 

its goals and objectives based on the activities that are actually being implemented. 

Executive summary – A condensed version of a longer report’s content. 

Executive summaries are written for someone who most likely does not have time 

to read a lengthy report. An executive summary may be called an abstract when it 

accompanies a scholarly document.

Experimental design – Individuals are randomly assigned to a test and control 

group and both are measured before and after the intervention, then the results are 

compared and analyzed.

External evaluation – An external is conducted by an agency or individuals not 

directly involved in or responsible for the program or activities being evaluated.

Evaluation design – The overall plan and approach for an evaluation. A description of 

how you plan to measure program components and how you plan to use the resulting 

measurements. The evaluation design is determined by the type of evaluation that is 

used and on the basic questions you want answered about your program.

Evaluation plan – A description of the overall approach or design used to guide 

an evaluation. The evaluation plan should include information on what evaluation 

activities will take place, a methodology for executing these activities, a timeline for 

completing planned evaluation tasks, a list of the individuals who will be responsible 

for completing each evaluation task, and a description of the purpose of the 

evaluation.

Formal evaluation – Rigorous and comprehensive assessments of data that are used to 

make decisions within a department, agency, or organization.

Frequency – A measure used to determine the number of times a given response 

occurs. This measure can be expressed using whole numbers or percentages.

Goal – A measurable statement of the desired long-term impact of the program.

History – Any event that happens between the time of your first measurement (pretest) and your second (post-test) that can affect the measurement. This can pose a 

threat to the validity of evaluation results. 

Impact evaluation – This evaluation is designed to assess whether a program or 

intervention had a long-term impact on a specified goal.
136 Appendixes

Informal evaluation – Less rigorous or comprehensive assessments of data that are 

used to make decisions within a department, agency, or organization.

Instrumentation – Changes over time can take place in the instruments used 

to measure participants’ performance. Changes caused by instrumentation 

can threaten internal validity if they cannot be separated from the effect of the 

treatment. Any flaws in a measurement device can skew evaluation results. 

Internal evaluation – An internal is conducted by the agency or individuals 

who are directly involved with or responsible for the program or activities being 

evaluated.

Logic model – A graphic representation that clearly identifies and lays out the 

logical relationships among program conditions, activities, outcomes (objectives), 

and impacts (goals).

Maturation – This describes the normal processes of development that occur 

over time that could affect evaluation outcomes independent of the evaluator’s 

intervention, and can pose a threat to the validity of evaluation results.

Mean – An average score that is calculated by dividing the sum of all responses to 

a particular question by the number of responses to that question.

Median – The midpoint or value below which half the values in a distribution 

fall. This score can be generated by arranging all responses in a list. The middle 

number is the median. If there are two middle numbers, then add the two 

numbers and average them for a median. 

Mixing methods – An approach that consists of intentionally combining different 

evaluation tools and techniques, not only to observe and to gather quantitative 

and qualitative information, but also to structure and analyze this information 

Mode – the number or value that occurs most often in a distribution of values.

Multiple methods – See definition of mixing methods.

Needs assessment – This evaluation is designed to assess problems in the 

community and perceptions about why they occur and persist.

Objective – A specific, measurable statement of the desired immediate or direct 

outcomes of the program that support the accomplishment of a goal.

Outcome – Refers to a change that results from a program or its activities.

Outcome evaluation – This evaluation is designed to assess whether the 

intervention achieved its stated short-term goals and objectives.

Outcome data – Information that is collected to measure the effectiveness 

of a program.

Output – Units of service that are provided by the program.
Measuring Excellence 137

Post-test only design – A type of evaluation design in which the target group is 

measured only after the intervention and those results are analyzed.

Pre- and post-test design – A type of evaluation design in which the target group 

is measured before and after the intervention and those results are compared and 

analyzed.

Pre- post- post-test design – A type of evaluation design in which the target group is 

measured before the intervention, immediately afterward, and then at a later time and 

those results are compared and analyzed.

Process data – This evaluation is designed to assess whether the intervention was 

implemented as intended and, if not, why.

Process evaluation – This evaluation is designed to assess whether an intervention was 

implemented as intended and, if not, why.

Program assessment – A disciplined way of assessing the merit, value, and worth of 

projects and programs.

Qualitative data – Data that are collected through activities such as observations, 

interviews, focus groups, and conversations. These data describe the attributes 

of an object without referring to quantity and are expressed in words. Instead of 

measuring the extent and direction of change or program impact, qualitative data are 

better suited to describing the nature of the change or impact that has taken place. 

Qualitative data are used frequently in process evaluations to describe how a program 

functions. 

Quantitative data – Data that are a measurement of something that can be counted 

and assigned a number. These data can be collected by using several methods, 

including simple counts, surveys, and tests. 

Quasi-experimental design – An evaluation design in which the target group and a 

similar group used as a control are measured before and after the intervention, then 

the results are compared and analyzed.

Questionnaire – See the definition for survey.

Random assignment – The assignment of individuals in the pool of all potential 

participants to either the experimental (treatment) or control group in such a manner 

that their assignment to a group is determined entirely by chance. 

Reliable data – The extent to which a measurement produces consistent results 

over repeated observations or administrations of the instrument under the same 

conditions each time. 
138 Appendixes

Research questions – A data-collection tool used to gather information on the 

focus of an evaluation effort.

Selection bias – Any determination, except random placement, by which you 

place some subjects in a treatment group, and other subjects in a control group. 

Subjects assigned to groups in any way other than random placement opens the 

possibility that the groups differ in important ways that can affect the results of a 

study. 

Stakeholder – People who will be affected by a project or can influence it but who 

are not directly involved with doing the project work. 

Statistical regression – A general tendency for extreme scores on any measure to 

move toward a less extreme score at a different time. Statistical regression occurs 

when study participants are selected based on their extreme scores. When an 

evaluator re-measures, there is a tendency for these extreme scores to move back 

toward the overall group average regardless of the effects of any intervention 

provided.

Strategic management – The process of specifying an organization’s objectives, 

developing policies and plans to achieve these objectives, and allocating resources 

to implement these plans.

Survey – A data-collection method that involves collecting information directly 

from individuals. Surveys can be self-administered, conducted face-to-face, over 

the telephone, or mailed. 

Target group – Also called a treatment group, a target group is composed of 

individuals receiving the activities or interventions that you are evaluating.

Testing – The process of test taking can pose a threat to the validity of evaluation 

results. Taking a pre-test may influence the behavior of subjects as much or more 

than the intervention does. The evaluator cannot be certain that the effects they 

see are because of the intervention and not caused by the administration of a pretest.

Treatment group – Also called a target group, a treatment group is composed of 

individuals receiving the activities or interventions that you are evaluating. 

Valid data – Data that are collected using a measurement instrument or test that 

accurately measures what they are supposed to measure. 
Measuring Excellence 139

Appendix K

Sample Job Announcement

Overview of Project

The Fictitious Police Department is seeking the services of a qualified evaluator to 

assist in an assessment of its 1-year project to reduce drug dealing in the city’s public 

housing developments. The department has been awarded a grant from the U.S. 

Department of Justice’s Bureau of Justice Administration and the U.S. Department 

of Housing and Urban Development’s Office of Resident Initiatives to explore 

how police can engage community coalitions in targeted efforts to reduce the open 

distribution and sale of illicit drugs. The grant includes funding for an evaluation 

designed to assess the impact of various strategies on reducing the sale of drugs to 

residents of the complex and also how those strategies affect sales to outsiders who 

come there to purchase drugs.

Evaluator Roles/Responsibilities

The evaluator is responsible for developing and implementing the evaluation plan, 

including the methodology, types of data be gather and analyze, and the management 

plan for completing the evaluation according to the guidelines set forth in the grant 

agreement. The evaluator will work closely with top command in charge of the 

project to develop a task list and timeline for the department’s participation in the 

evaluation process.

Tasks and Timeline

Applicants must outline an evaluation plan designed to produce the final report by 

the end of the 1-year grant-reporting period, as well as fulfill the quarterly reporting 

requirement specified in the grant agreement. Applicants must describe how they will 

meet the grant requirements, including a detailed discussion of the tasks and who will 

carry them out within the timeline detailed in the grant agreement.

Management Plan and Key Personnel

Applicants also must provide a proposed management plan designed to ensure that a 

comprehensive and valid evaluation will be completed within the timeline. The plan 

also must include a description of the roles and responsibilities of key personnel, as 

well as the role and responsibilities of the evaluator in oversight.
140 Appendixes

Preferred Skills and Experience

Applicants must demonstrate appropriate skills and experience in developing and 

implementing an evaluation strategy to assess a police initiative in a real-world 

setting. Applicants must demonstrate strong program evaluation and impact 

evaluation skills, in addition to the management skills required to complete a 

complex and multifaceted assessment within a specified time.

Background/Education

Applicants must submit a current curriculum vita or résumé for the lead evaluator 

and additional key personnel, as well as a brief description of relevant experience 

in police evaluation. Preferred candidates will exhibit educational credentials that 

certify broad experience in evaluation methodology and implementation, as well 

as the practical skills required to work alongside police in a diverse environment.

Complete applications along with references and examples of relevant work should 

be submitted to [Contact Person] by [Due Date] at [Address of the Agency].
Measuring Excellence 141

Appendix L

Evaluation Resources

Evaluation resources with practical applications for people with little or 

no evaluation experience

Eck, John E. Assessing Responses to Problems: An Introductory Guide for Police ProblemSolvers. Washington, D.C.: U.S. Department of Justice Office of Community 

Oriented Policing Services, 2002. www.cops.usdoj.gov/txt/pop/e08064504.txt

Eck, John E. “Learning from experience in problem-oriented policing and situational 

prevention: The Positive Functions of Weak Evaluations and the Negative 

Functions of Strong Ones.” In Evaluation for Crime Prevention, Nick Tilley, ed. 

Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. 

Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.). Beverly Hills, 

California: Sage Publications, 1990.

Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). Newbury 

Park, California: Sage Publications, 1993. 

Salant, P. and D. Dillman, D. How to Conduct Your Own Survey. New York: Wiley 

Publishers, 1994.

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement

Aagencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice 

Statistics and the Office of Community Oriented Policing Services, 1999. 

www.ojp.usdoj.gov/bjs/pdf/ccspglea.pdf

Web sites that offer useful information on evaluation resources, 

tools, and articles:

American Evaluation Association: The American Evaluation Association is an 

international professional association of evaluators devoted to the application 

and exploration of program evaluation, personnel evaluation, technology, and 

many other forms of evaluation. Evaluation involves assessing the strengths and 

weaknesses of programs, policies, personnel, products, and organizations to 

improve their effectiveness. www.eval.org

Bureau of Justice Assistance (BJA) Evaluation Web Site: The evaluation web site, 

produced by the BJA Center for Program Evaluation, provides state 

administrative agency staff, criminal justice planners, researchers and evaluators, 

and local practitioners with a variety of resources for evaluating criminal justice 

programs. http://www.ojp.usdoj.gov/BJA/evaluation/index.html
142 Appendixes

Center for Problem Oriented Policing: The Center for Problem-Oriented Policing 

is a nonprofit organization composed of police practitioners, researchers, and 

universities dedicated to advancing the concept and practice of problem-oriented 

policing in open and democratic societies. www.popcenter.org

COPS Office: The Office of Community Oriented Policing Services 

(the COPS Office) was created as a result of the Violent Crime Control and 

Law Enforcement Act of 1994. As a component of the Justice Department, the 

mission of the COPS Office is to advance community policing in jurisdictions 

of all sizes across the country. COPS provides grants to tribal, state, and local 

law enforcement agencies to hire and train community policing professionals, 

acquire and deploy cutting-edge crime-fighting technologies, and develop and 

test innovative policing strategies. The COPS Office also offers publications, 

CD-ROMs, videos, toolkits, and training resources to law enforcement agencies 

nationwide. www.cops.usdoj.gov

International Association of Chiefs of Police (IACP): The IACP is the world’s 

oldest and largest nonprofit membership organization of police executives, with 

more than 19,000 members in more than 89 different countries. Its goals are to 

advance the science and art of police services; develop and disseminate improved 

administrative, technical, and operational practices; foster police cooperation 

and exchange of information; bring about recruitment and training of qualified 

persons in the police profession; and encourage adherence of all police officers to 

high professional standards of performance and conduct. www.theiacp.org/

Juvenile Justice Evaluation Center Online (JJEC): The JJEC, a segment of 

the Justice Research and Statistics Association, is a tool for assisting juvenile 

justice practitioners, policymakers, and state agency administrators with the 

assessment and evaluation of programs and initiatives. JJEC Online is composed 

of several sections, including JJEC Information, State Information, Juvenile 

Justice Evaluation Program Areas, Evaluation Resources, State Information, and 

Technical Assistance and Training. www.jrsa.org/jjec/index.html

Police Executive Research Forum (PERF): PERF, a national organization of 

progressive police executives, is dedicated to improving policing and advancing 

professionalism through research, public policy debate, provision of management 

services and executive development, training, and publishing. www.policeforum.org
Measuring Excellence 143

Police Foundation: Since its founding in 1970, the Police Foundation has conducted 

seminal research in police behavior, policy, and procedure. The foundation has 

established and refined the capacity to define, design, conduct, and evaluate 

controlled experiments testing ways to improve the delivery of police services.

www.policefoundation.org/

Technical assistance resources:

The American Evaluation Association’s (AEA) web site contains a list of individuals and 

firms that are available for evaluation consulting. www.ovcttac.org/aboutus.cfm

The Office for Victims of Crime Training and Technical Assistance Center’s (OVC 

TTAC) training and technical assistance activities are coordinated through three core 

functions: 

Needs assessment. OVC TTAC uses a variety of measurement tools—surveys, 

stakeholder discussions, and literature reviews—to assess the needs of key 

constituencies and to identify available technical assistance and training 

resources to support their needs. 

Capacity building. By developing and delivering training and technical 

assistance resources, OVC TTAC helps organizations foster professional 

development of their staff, enhance services to their communities, and reach 

unserved and underserved victims of crime. 

Evaluation. OVC TTAC monitors customer satisfaction and measures the 

resulting changes of its training and TA activities over time. 

www.ovcttac.org/aboutus.cfm

The What Works Clearinghouse provides a “Registry of Outcome Evaluators” 

that allows web site users the opportunity to search for evaluators (both 

individuals and organizations) who conduct research on the effects of 

educational interventions. The Registry allows users to search by geographic 

region, individual/organization, content area, and target populations. 

www.whatworks.ed.gov/technicalassistance/EvlSearch.asp

Examples of evaluation reports related to policing:

Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: The Boston 

Gun Project’s Operation Ceasefire. Washington, D.C.: U.S. Department of 

Justice, National Institute of Justice, 2001.

Lurigio, A.J. and W.G. Skogan. “Winning the Hearts and Minds of Police Officers: 

An Assessment of Staff Perceptions for Community Policing in Chicago,” Crime 

& Delinquency, 40 (3) (1994): 315–330.

Independent Commission on Policing for Northern Ireland. (1999). A New 

Beginning: Policing in Northern Ireland. Retrieved November 24, 2004 from 

www.belfast.org.uk/report.htm

o

o

o

o

o
144 Appendixes

Evaluation journals:

American Journal of Evaluation

The Evaluation Exchange

Evaluation and Program Planning, ISSN 0149-7189

The Evaluation Review

New Directions for Evaluation

Practical Assessment Research and Evaluation, a peer-reviewed electronic journal, 

ISSN 1531-7714

Technical reports and articles:

Burt, M.R., et al. Evaluation Guidebook. Washington DC: National Institute 

of Justice, Office of Justice Programs, U.S, Department of Justice, Urban 

Institute, 1997.

Clark, R.V. and J. Eck. Becoming a Problem Solving Crime Analyst in Fifty-Five 

Small Steps. Jill Dando Institute of Crime Science. Ad Hoc Publications, 2002. 

Coldren, J.R., T. Bynum, and J. Thome. Evaluating Juvenile Justice Programs: 

A Design Monograph for State Planners. Office of Juvenile Justice and 

Delinquency Prevention, 1991. 

www.jp.usdoj.gov/BJA/evaluation/guide/documents/documents/documentg.html

The Compiler. (1995). Evaluation as an Important Tool in Criminal Justice 

Planning. Retrieved January 20, 2004.

Eck, J. Assessing Responses to Problems: An Introductory Guide for Police 

Problem-Solvers. Washington, D.C.: U.S. Department of Justice Office of 

Community Oriented Policing Services, 2002. 

www.cops.usdoj.gov/txt/pop/e08064504.txt /www.cops.usdoj.gov

Joppe, M. The Research Process. Ontario Hostelry Institute, AMEX Canada, and 

Ryerson University, n.d.. Retrieved January 19, 2004, from 

www.ryerson.ca/~mjoppe/ResearchProcess/RequestforProposal.htm

Juvenile Justice Evaluation Center. Hiring and Working with an Evaluator: 

Program Evaluation Briefing Series No. 2. Washington, D.C.: Justice 

Research & Statistics Association, 2001.

Juvenile Justice Evaluation Center, Justice Research and Statistics Association. 

Evaluability Assessment: Examining the Readiness of a Program for Evaluation.

Program Evaluation Briefing Series No. 6. Washington, D.C.: Office of 

Juvenile Justice and Delinquency Prevention, 2003. 

www.jrsa.org/jjec/about/briefing_evaluability-assessment.html
Measuring Excellence 145

Melvin M., G.T. Henry, and G. Julnes. Evaluation: An Integrated Framework for 

Understanding, Guiding, and Improving Public and Nonprofit Policies and 

Programs. San Francisco: Jossey-Bass, 2002. 

Muraskin, L.D. Understanding Evaluation: The Way to Better Prevention Programs. 

Washington, D. C.: U.S. Department of Education, 1993. 

www.jrsa.org/jjec/resources/publications/westat-handbook.pdf

Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. “Juvenile Mentoring 

Program: A Progress Review,” Juvenile Justice Bulletin. Washington, DC: U.S. 

Department of Justice, Office of Justice Programs, Office of Juvenile Justice and 

Delinquency Prevention, 2000.

Priest, S. “A Program Evaluation Primer,” Journal of Experiential Education, 24 (1) 

(2001), 34–40.

Sechrest, L., and A.J. Figuerdo. “Program Evaluation.” Evaluation Group for Analysis 

of Data, Department of Psychology, University of Arizona. Annual Review of 

Psychology. 44 (1993), 645–74.

Shadish, Jr., W.R., T.d. Cook, and L.C. Leviton. Foundations of Program Evaluation, 

Theories of Practice. London: Sage Publications, 1995. 

Taylor-Powell, E., S. Steele, and M. Douglah. Planning a Program Evaluation

(G-3658-1). Madison: University of Wisconsin-Extension, Cooperative 

Extension, 1996.

Todevski, D. Public Administration Evaluation. Ministry of Finance, Republic of 

Macedonia, 2001. 

United States Air Force Family Advocacy Program. A guide to family intervention 

and prevention program evaluation: Conducting an evaluability assessment. In G. 

K. Kantor, K. Kendall-Tackett, C. M. Allen, eds. Durham: University of New 

Hampshire, 2000. 

United States Department of Health and Human Services, Administration for 

Children & Families. The Program Manager’s Guide to Evaluation. 

United States Department of Health and Human Services, Substance Abuse and 

Mental Health Services Administration, Center for Substance Abuse Prevention. 

Achieving Outcomes: A Practitioner’s Guide to Effective Prevention. Washington, 

DC, 2002. www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf

Public Technology, Inc. Program Evaluation and Analysis: A Technical Guide for State 

and Local Governments. Washington, D.C.: U.S. Department of Housing and 

Urban Development, Office of Policy Development and Research, 1978.

United States Department of Housing and Urban Development, Office of Policy 

Development and Research. A Guide to Evaluating Crime Control of Programs in 

Public Housing. Prepared for the United States Department of Housing and Urban 

Development by KRA Corporation. Washington, D.C., 1997.
146 Appendixes

United States General Accounting Office. Designing Evaluations. Washington, 

D.C.: United States General Accounting Office, Program Evaluation and 

Methodology Division, 1991.

Weidman, D. R., et al. Intensive Evaluation for Criminal Justice Planning Agencies. 

Washington, D.C.: U.S. Department of Justice, 1975.

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement 

Agencies. U.S. Department of Justice, Bureau of Justice Statistics and the 

Office of Community Oriented Policing Services. 

www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf

Wholey, J.S., H.P. Harty, and K. E. Newcomer. The Handbook of Practical 

Program Evaluation. San Francisco, California: Jossey-Bass, 1994.

W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. 

Kellogg Foundation, 1998.

W.K. Kellogg Foundation. Logic Model Development Guide: Using Logic Models 

to Bring Together Planning, Evaluation, and Action. Battle Creek, Michigan: 

W.K. Kellogg Foundation, 2001.

Evaluation Books

Berk, R.A. and P. H. Rossi. Thinking About Evaluation (2nd ed.). Newbury Park, 

California: Sage Publications, 1999. 

Bickman, L., ed. Using Program Theory in Evaluation. San Francisco, California: 

Jossey-Bass, 1987. 

Boruch, R. F., P.M. Wortman, and D.S. Cordray, eds. Reanalyzing Program

Evaluations. San Francisco: Jossey-Bass, 1981. 

Boyle, R. and D. Lemaire. Building Effective Evaluation Capacity: Lessons from 

Practice. New Brunswick, New Jersey: Transaction Publishers, 1999. 

Bradley, V. J. Assessing and Enhancing the Quality of Services: A Guide for the 

Human Services Field. Boston: Human Services Research Institute, 1984. 

Braybrooke, C.B., and C.B. Lindblom. A Strategy of Decision: Policy Evaluation as 

a Social Process. New York: Free Press, 1963. 

Brinkerhoff, R.O., D.M. Brethower, T. Hluchyj, and J.R. Nowakowski. 

Program Evaluation: A Practitioner’s Guide for Trainers and Educators. Boston:

Kluwer-Nijhoff, 1983. 

Brinkerhoff, R.O., ed. Evaluating Training Programs in Business and Industry, 

No.44, Winter. San Francisco: Jossey-Bass, 1989. 
Measuring Excellence 147

Bryk, A.S., ed. Stakeholder-Based Evaluation. San Francisco, California: Jossey-Bass, 1983.

Campbell, D.T., and J.C. Stanley. Experimental and Quasi-Experimental Designs for

Research. Chicago: Rand McNally, 1963. 

Caro, F.C., ed., Readings in Evaluation Research. New York: Russell Sage Foundation, 1971. 

Chambers, D.E., K.R. Wedel, and M.K. Rodwell. Evaluating Social Programs. Boston: 

Allyn & Bacon, 1992. 

Chen, H. Theory-Driven Evaluations. Newbury Park, CA: Sage Publications, 1990. 

Conner, R., and M. Hendricks, eds. International Innovations in Evaluation

Methodology. San Francisco: Jossey-Bass, 1989. 

Cook, T.D., and D.T. Campbell. Quasi-Experimentation: Design and Analysis Issues for 

Field Settings. Chicago: Rand McNally, 1979. 

Cronbach, L.J. Designing Evaluations of Educational and Social Programs. 

San Francisco: Jossey-Bass, 1982.

Datta, L., ed. Evaluation in Change: Meeting New Government Needs. Newbury Park, 

California: Sage Publications, 1981. 

Denzin, N.K., and Y.S. Lincoln, eds. Handbook of Qualitative Research. Thousand 

Oaks, California: Sage Publications, 1994. 

Eck, J. “Learning from Experience in Problem-Oriented Policing and Situational 

Prevention: The Positive Functions of Weak Evaluations and the Negative 

Functions of Strong Ones.” In Evaluation for Crime Prevention, Nick Tilley, ed. 

Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. 

Eisner, E.W. The Enlightened Eye: Qualitative Inquiry and the Enhancement of

Educational Practice. Upper Saddle River, New Jersey: Prentice Hall, 1998. 

Fetterman, D.M., S. J. Kaftarian, and A. Wandersman, eds. Empowerment Evaluation:

Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, 

California: Sage Publications, 1996. 

Grinnell, R.M. Social Work Research and Evaluation (4th ed.). Itasca, Illinois: F.E. 

Peacock, 1993. 

Guba, E.C. and Y.S. Lincoln. Effective Evaluation: Improving the Usefulness of 

Evaluation Results through Responsive and Naturalistic Approaches. 

San Francisco: Jossey-Bass, 1981. 

Guba, E.G. and Y.S. Lincoln. Fourth Generation Evaluation. Newbury Park, 

California: Sage Publications, 1989. 

Guttentag, M. and E.L. Struening, eds. Handbook of Evaluation Research. Newbury 

Park, California: Sage Publications, 1975.
148 Appendixes

Hausman, J. and D. Wise, eds. Social Experimentation. Chicago: University of 

Chicago Press, 1985. 

Herman, J.L., L.L. Morris, and C.T. Fitz-Gibbon. Evaluator’s Handbook. 

Newbury Park, California: Sage Publications, 1987. 

House, E.R., ed. School Evaluation: The Politics and Process. Berkeley, California: 

McCutchan, 1973. 

House, E.R., ed New Directions in Educational Evaluation. London: Farmer, 1986.

King, J., L. Morris, and C.T. Fitz-Gibbon. How to Assess Program Implementation. 

Newbury Park, California: Sage Publications, 1987.

LaVigna, G.W., T. J. Willis, J.F. Shaull, M. Abedi, and M. Sweitzer. The Periodic 

Service Review: A Total Quality Assurance System for Human Services and 

Education. Baltimore: Paul H. Brookes, 1994.

Levin, H.M. Cost-Effectiveness: A Primer. Newbury Park, California: Sage 

Publications, 1983. 

Levine, R.A., M.A. Solomon, C.M. Hellstern, and H. Wollmann. Evaluating 

Action Programs: Readings in Social Action and Education. Boston: Allyn & 

Bacon, 1981. 

Lincoln, Y.S. and E.C. Cuba. Naturalistic Inquiry. Newbury Park, California: Sage 

Publications, 1985. 

Lofland, J. and L.H. Lofland. Analyzing Social Settings: A Guide to Qualitative 

Observation and Analysis. Belmont, California: Wadsworth, 1984.

Love, A.J. Internal Evaluation: Building Organizations from Within. Applied 

Social Research Methods Series, Vol. 24. Newbury Park, California: Sage 

Publications, 1991 

Madaus, C.F., M.J. Scriven, and D.L. Stufflebeam, eds. Evaluation Models: 

Viewpoints on Educational and Human Services Evaluation. Boston: KluwerNijhoff, 1983. 

Miles, M.B.and A.M. Huberman. Qualitative Data Analysis: A Sourcebook of New

Methods. Newbury Park, California: Sage Publications, 1984. 

Mishan, E.J. Cost-Benefit Analysis (4th ed.). London: Unwin Hyman, 1988. 

Mohr, L.B. Impact Analysis for Program Evaluation (2nd ed.). Thousand Oaks, 

California: Sage Publications, 1995. 

Nathan, R.P. Social Science in Government: Uses and Misuses. New York: Basic 

Books, 1989. 

Nay, J.N. and P. Kay. Government Oversight and Evaluability Assessment. 

Lexington, Massachusetts: D.C. Heath, 1982.
Measuring Excellence 149

Paine, S.C., G.T. Bellamy, and B.L. Wilcox, eds. Human Services that Work: From

Innovation to Standard Practice. Baltimore, Maryland: Paul H. Brookes, 1984. 

Palumbo, D., ed. The Politics of Program Evaluation. Newbury Park, 

California: Sage Publications, 1991. 

Patton, M.Q. Practical Evaluation. Newbury Park, California: Sage Publications, 1982.

Patton, M.Q. Creative Evaluation (2nd ed.). Newbury Park, California: Sage 

Publications, 1987. 

Patton, M.Q. How to Use Qualitative Methods in Evaluation. Newbury Park, 

California: Sage Publications, 1987. 

Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.). Beverly Hills, 

California: Sage Publications, 1990. 

Patton, M.Q. Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, California: 

Sage Publications, 1997. 

Popham, I.W., ed. Evaluation. in Education: Current Applications. Berkeley, 

California: McCutchan, 1974. 

Pressman, I.L. and A. Wildavsky. Implementation (3rd ed.). Berkeley: University of 

California Press, 1984. 

Rossi, P.H., ed. Standards for Evaluation Practice. San Francisco: Jossey-Bass, 1982. 

Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). 

Newbury Park, California: Sage Publications, 1993. 

Royce, D. . Program Evaluation. Chicago: Nelson Hall, 1992. 

Rutman, L., ed. Evaluation Research Methods: A Basic Guide. Newbury Park, 

California: Sage Publications, 1977.

Salant, P. and D. Dillman. How to Conduct Your Own Survey. New York: Wiley 

Publishers, 1994.

Saxe, L. and M. Fine. Social Experiments: Methods for Design and Evaluation. 

Newbury Park, California: Sage Publications, 1981. 

Schmidt, R.E., J.W. Scanlon, and J.B. Bell. Evaluability Assessment: Making Public

Programs Work Better. Rockville, Maryland: U.S. Department of Health and 

Human Services, Project Share, 1979. 

Schulberg, H.C. and J.M. Jerrell, eds. The Evaluator and Management. Newbury Park, 

California: Sage Publications, 1979. 

Scriven, M. The Logic of Evaluation. Inverness, California: Edgepress, 1980. 

Scriven, M. Evaluation Thesaurus (4th ed). Newbury Park, California: Sage 

Publications, 1991. 
150 Appendixes

Sechrest, L., I. Bunker, I., and B. Perrin, eds. Improving Methods in Nonexperimental

Research. Newbury Park, California: Sage Publications, 1991. 

Shadish, W., T. Cook, and L. Leviton. Foundations of Program Evaluation: Theories 

of Practice. Newbury Park, California: Sage Publications, 1991. 

Smith, N.L. ed. New Techniques for Evaluation. Newbury Park, California: Sage 

Publications, 1981. 

Smith, N.L., ed. Communication Strategies in Evaluation. Newbury Park, 

California: Sage Publications, 1982. 

Stake, R. B., ed. Evaluating the Arts in Education. A Responsive Approach. 

Columbus, Ohio: Merrill, 1975.

Stufflebeam, D.L. and A.J. Shinkfield. Systematic Evaluation: A Self-Instructional 

Guide to Theory and Practice. Boston: Kluwer-Nijhoff, 1985. 

Suchman, B.A. Evaluative Research: Principles and Practice in Public Service and 

Social Action Programs. New York: Russell Sage Foundation, 1967. 

Thompson, M.S. Benefit-Cost Analysis for Program Evaluation. Newbury Park, 

California: Sage Publications, 1980. 

Trochim, W.M.K. Research Design for Program Evaluation: The RegressionDiscontinuity Approach. Newbury Park, California: Sage Publications, 1984.

Veney, J.E. and A.D. Kaluzny. Evaluation and Decision-Making for Health Services

(3rd ed.). Ann Arbor, Michigan: Health Administration Press, 1998. 

Webb, B.J., D.T. Campbell, R.D. Schwartz, and L.B. Sechrest. Unobtrusive 

Measures: Nonreactive Research in the Social Sciences. Chicago: Rand 

McNally, 1966. 

Weiss C.H. Evaluation: Methods for Studying Programs and Policies (2nd ed.). 

Englewood Cliffs, New Jersey: Prentice Hall, 1997. 

Weiss, C.H. Evaluation Research: Methods for Assessing Program Effectiveness. 

Englewood Cliffs, New Jersey: Prentice-Hall, 1972. 

Weiss, H.B. and F. Jacobs. Evaluating Family Programs. New York: Aldine 

DeGruyter, 1988. 

Welch, W., ed. Case Study Methodology in Educational Evaluation. Minneapolis: 

Minnesota Research and Evaluation Center, 1981. 

Wholey, J.S. Evaluation and Effective Public Management. Boston: Little, 

Brown, 1983. 
Measuring Excellence 151

Williams, W. The Implementation Perspective. Berkeley: University of California Press, 

1980.

Yin, R.K. Case Study Research: Design and Methods. Newbury Park, California: Sage 

Publications, 1984. 

Zweig, F.M., ed. Evaluation in Legislation. Newbury Park, California: Sage 

Publications, 1979. 

Zweig, F.M. and K.E. Martin, eds. Educating Policymakers for Evaluation. Newbury 

Park, California: Sage Publications, 1981 

References

Measuring Excellence 155

References

Artley, A. and S. Stroh, eds. The Performance-Based Management Handbook: A SixVolume Compilation of Techniques and Tools for Implementing the Government 

Performance and Results Act of 1993 (GPRA), Volume 2: Establishing an 

Integrated Performance Measurement System. Oak Ridge Institute for Science 

and Education, Training Resources and Data Exchange, The PerformanceBased Management Special Interest Group, 2001. 

www.orau.gov/pbm/pbmhandbook/Volume%202.pdf

Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: 

The Boston Gun Project’s Operation Ceasefire. Washington, D.C.: U.S. 

Department of Justice, National Institute of Justice, 2001. 

Center for Substance Abuse Prevention (CSAP). Building a Successful Prevention 

Program: A Planning Tool for Substance APrevention Professionals. CSAP 

Western Center for the Application of Prevention Technologies (CAPT). 

Reno: University of Nevada, 2003. 

Community Policing Consortium Resources. Strategic Planning for Community 

Policing, Section 5: Evaluation. Washington, D.C.: Community Policing 

Consortium. www.communitypolicing.org/strategic/sec6.html

Fiedler, M. The Colorado Springs Police Department’s Police Accountability and 

Service Standard (PASS) Mmodel. Colorado Springs: Management Services 

Division, Planning Section, Colorado Springs Police Department, 2002.

Juvenile Justice Evaluation Center, Justice Research and Statistics Association.

Evaluability Assessment: Examining the Readiness of a Program for Evaluation. 

Program Evaluation Briefing Series No. 6. Washington, DC: Office of 

Juvenile Justice and Delinquency Prevention, 2006.

www.jrsa.org/jjec/about/briefing_evaluability-assessment.html

Loe, H., Jr. Practical Evaluation of Public Health Programs Workbook. Houston, 

TX: The University of Texas-Houston Health Science Center, School of 

Public Health and The Texas Department of Health, 2001.

www.phppo.cdc.gov/phtn/Pract-Eval/workbook.asp

Mertinko, E., L.C. Novotney, T. Baker, and J. Lange. Evaluating Your Pprogram: 

A Beginner’s Self-Evaluation Workbook for Mentoring Programs. Washington, 

D.C.: Office of Juvenile Justice and Delinquency Prevention, Office of 

Justice Programs, and the U.S. Department of Justice. 
156 References

Miles, M.B. and A.M. Huberman. Qualitative Data Analysis. Thousand Oaks, 

California: Sage Publications, 1994. 

Patton, M. Q. Qualitative Evaluation and Research Methods (2nd ed.). 

Beverly Hills, California: Sage Publications, 1990.

Patton, M.Q. Qualitative Research and Evaluation Methods. Thousand Oak,s 

California: Sage Publications, 2002.

Salant, P., D. Dillman. How to Conduct Your Own Survey. New York: Wiley 

Publishers, 1994.

Shadish, W. R., T.D. Cook, and D.T. Campbell. Experimental and QuasiExperimental Designs for Generalized Causal Inference. New York: Houghton 

Mifflin, 2002.

United States Department of Health and Human Services, Substance Abuse and 

Mental Health Services Administration (SAMHSA), Center for Substance 

Abuse Prevention. Achieving Outcomes: A Practitioner’s Guide to Effective 

Prevention. Washington, D.C.: SAMHSA, 2002. 

www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf

The University of Texas at Arlington, School of Social Work. Instrument

Development Checklist. www2.uta.edu/sswmindel/S6324/

Class%20Materials/measurement/instrument_development_checklist.htm

The University of Texas-Houston Health Science Center, School of Public 

Health and The Texas Department of Health. Practical Evaluation of Public 

Health Programs. Atlanta, Georgia: The Centers for Disease Control and 

Prevention and The Association of Schools of Public Health Cooperative 

Agreement, 1998.

University of Wisconsin-Extension: Program Development and Evaluation. 

Enhancing Program Performance with Logic Models. 

www.uwex.edu/ces/lmcourse/

Uphoff, N. “A Field Methodology for Participatory Self-Evaluation,” Community 

Development Journal, 26, (1991) 271–286. 

Urban Institute. Evaluation Guidebook for Projects Funded by S.T.O.P. Formula 

Grants under the Violence Against Women Act. Washington, D.C.: National 

Institute of Justice, 1997.

USAID Center for Development Information and Evaluation. Performance 

Monitoring and Evaluation Tips: Conducting a Participatory Evaluation. 

Washington, D.C.: United States Agency for International Development, 1996.

USAID Center for Development Information and Evaluation. Performance 

Monitoring and Evaluation Tips: The Role of Evaluation in USAID. Washington, 

D.C.: United States Agency for International Development, 1997.
Measuring Excellence 157

Velez, L. and T. Paine. Service Standards in the Colorado Springs Police Department. 

Retrieved May 1, 2007 from www.calea.org/Online/newsletter/No83/

servicestandards.htm

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement

Agencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice 

Statistics and the Office of Community Oriented Policing Services. 

www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf

W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. 

Kellogg Foundation, 1998.


For More Information

U.S. Department of Justice

Office of Community Oriented Policing Services

1100 Vermont Avenue, N.W.

Washington, DC 20530

To obtain details on COPS programs, 

call the COPS Office Response Center at 800.421.6770

Visit COPS Online at the address listed below

www.cops.usdoj.gov

e06072996