PROGRAM 

EVALUATION 

GUIDE 

Tribal Motor Vehicle Injury Prevention
3 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

 

TRIBAL MOTOR VEHICLE INJURY PREVENTION 

PROGRAM EVALUATION GUIDE 

2022 

DEVELOPED FOR: 

National Center for Injury Prevention and Control
 

Centers for Disease Control and Prevention
 

DEVELOPED BY: 

Albuquerque Area Southwest Tribal Epidemiology Center
 

Albuquerque Area Indian Heath Board, Inc.
 

This document was produced by the Albuquerque Area Southwest Tribal Epidemiology Center 


through an Interagency Agreement between the National Center for Injury Prevention and Control,
 

Centers for Disease Control and Prevention, and the Indian Health Service,
 

Office of Public Health Support.
 
4 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

 

 

TABLE OF CONTENTS 

Introduction 5
 

How to Use this Guide: Why? What? When? Who? How? 

Framework for Tribal Motor Vehicle Injury Prevention Program Evaluation 11
 

Step 1 Engaging Partners 13
 

Step 2 Describing the Program 15
 

Step 3 Focusing the Evaluation Design 22
 

Step 4 Gathering Credible Evidence 26
 

Step 5 Justifying Conclusions 41
 

Step 6 Ensuring Use and Sharing Lessons Learned 45
 

Appendix A Evaluation Standards 49
 

Appendix B Additional Resources 53
 

Glossary 55
 

Acknowledgements 59
 
5 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

HOW TO USE THIS GUIDE 

WHY WAS THIS GUIDE DEVELOPED? 

This guide was developed to share best practices for evaluating Tribal Motor Vehicle Injury Prevention 

(also called Tribal Traffic Safety) Programs. It is a companion document to the Tribal Motor Vehicle 

Injury Prevention (TMVIP) Best Practices Guide 2016, which focuses on evidence-based strategies 

to reduce the burden associated with motor vehicle crashes in American Indian and Alaska Native 

communities. These strategies include increasing seat belt use, increasing child safety seat use, 

and decreasing alcohol-impaired driving. This guide aims to assist Tribes and Tribal Organizations 

that are implementing these proven strategies with key methods for evaluating their programs. 

Specific examples are provided throughout the guide that pertain directly to Tribal Traffic Safety 

Programs. At the same time, the evaluation principles and framework throughout the guide are 

universal, and may also be useful for evaluating other public health programs in Indian Country. 

HOW TO NAVIGATE THROUGH THIS GUIDE 

This guide was developed to be an easy-to-use electronic document. Several hyperlinks—words 

highlighted in color and underlined that can be clicked on—are included throughout the document. 

Clicking on these hyperlinks will take you to a variety of external resources (e.g., websites, PDF 

documents, or other resources) for more in-depth information about a specific topic, method, or 

technique. 

There is also a list of resources at the end of this guide that can be used to help strengthen 

knowledge and skills in program evaluation. 
6 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When? 

WHAT IS PROGRAM EVALUATION? 

Definition: Program evaluation is the systematic collection of information about the activities, 

characteristics, and outcomes of programs to make judgments about the program, improve program 

effectiveness, and/or inform decisions about future program development. 

Tribal Traffic Safety Programs are invaluable in American Indian/Alaska Native communities to 

protect all generations from the harms of motor vehicle crash injuries, which continue to be a 

leading cause of mortality across the lifespan. But how do we know if these programs are working? 

And even if they are, how can we improve them to make them even better for tribal communities? 

Good program evaluation can help us answer these questions, and provides results that can be 

used to make our programs the best programs possible! 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When? 

 

WHY DO PROGRAM EVALUATION? 

Most Tribal Traffic Safety Programs aim to change behavior in one or more populations of focus. 


At the same time, they also strive to create an environment that reinforces sustained adoption of 


these changes, with the intention that changes in environments and behaviors will prevent injuries. 


Through evaluation, programs can track these changes and, with careful evaluation designs, assess 


the effectiveness and impact of a particular intervention, strategy, and/or activity for producing these 


changes.
 

Program evaluation enables programs to: 

Create the best possible programs 

Identify what is working well 

Learn from mistakes 

Modify program strategies and activities as needed 

Monitor progress toward program goals 

Assess the success of the program in achieving its short-term, intermediate,
 

and long-term outcomes 


The ultimate purpose of program evaluation is to monitor progress toward your program’s goals and 

use the information to improve your programs. The evaluation results can be used to demonstrate 

the effectiveness of your program, identify ways to improve your program, modify program elements, 

demonstrate accountability, and justify investment of resources. 

✔ To demonstrate that resources are 

being well spent and that the program 

is effective 

✔ To compare outcomes with those of 

previous years 

✔ To compare actual outcomes with 

intended outcomes 

✔ To justify the need for additional resources 

and support 

✔ To support annual and long-range planning 

✔ To focus attention on issues important 

to your program 

✔ To promote your program 

✔ To retain or increase resource investment 

✔ To provide direction for program staff 

✔ To identify training and technical 

assistance needs 

✔ To find opportunities for continuous 

quality improvement 

MORE REASONS TO PRIORITIZE PROGRAM EVALUATION 

7 

•

•

•

•

•

•
8 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When? 

WHO SHOULD DO PROGRAM EVALUATION? 

Program evaluation is an essential component of all Tribal Traffic Safety Programs. There are several 

options that should be considered when determining who will lead evaluation activities in your 

program. 

Internal/External Evaluators: Traditionally, program evaluation was led by an external individual who 

was not affiliated with your program. The rationale for this approach was to ensure that the evaluator 

would be objective and not introduce bias into your evaluation results. However, over the past 

decade there have been new movements in program evaluation, such as empowerment evaluation, 

which provides communities with the tools and knowledge that allow them to monitor and evaluate 

their own performance. Neither approach is right or wrong, but you should know if your funding 

source has specific requirements that determine who can evaluate your program. 

Team Approach: Good evaluation requires a combination of skills that are rarely found in one 

person. A preferred approach is to build an evaluation team that includes internal program staff, 

external partners, and possibly consultants or contractors with evaluation expertise. 

An initial step in the formation of an evaluation team is to decide who will be responsible for 

planning and implementing evaluation activities. One or two persons can be selected as the 

lead evaluator (or co-leaders) to coordinate program evaluation efforts. Leader(s) should be 

responsible for overseeing evaluation activities, including planning and budgeting for evaluation, 

developing program objectives, addressing data collection needs, reporting findings, and working 

with consultants. Although the leader(s) should have the skills necessary to competently coordinate 

evaluation activities, he or she can also look elsewhere for technical expertise to design and 

implement specific tasks. 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When? 

✔ Has experience in the type of evaluation 

needed 

✔ Can do both quantitative and qualitative 

data collection and analysis 

✔ Is aware of existing data sources 

✔ Works successfully with a wide variety of 

partners 

✔ Incorporates evaluation into all program 

activities 

✔ Listens to your needs and ideas 

✔ Educates program personnel in designing 

and conducting the evaluation 

✔ Honors important cultural considerations 

for your program/community 

✔ Keeps all data confidential (consider using 

a MOA or data sharing agreement) 

✔ Gives staff and partners a full report of 

findings 

✔ Agrees that the Tribal Traffic Safety Program 

has sole ownership of the evaluation data 

✔ Maintains neutrality and understands that 

the ultimate purpose of program evaluation 

is to make our programs the best programs 

possible 

CHARACTERISTICS OF A GOOD EVALUATOR 

If you plan to include an external evaluator as part of your evaluation team, important factors to 

consider include their level of professional training, experience, and ability to meet your needs. Overall, 

it is important to find an evaluator whose approach to evaluation best fits your program’s needs and 

goals. Be sure to check all references carefully before you enter into a contract with any evaluator. Also 

consider executing a Memorandum of Agreement (MOA) or data sharing agreement that outlines how 

the confidentiality of your data will be protected. Some places to find experienced evaluators include 

health departments, universities, non-profit organizations, and Tribal Epidemiology Centers. 

TRIBAL EPIDEMIOLOGY CENTERS TIP 

There are 12 Tribal Epidemiology Centers (TECs) across the country. 


TECs provide training and technical assistance to tribes and urban Indian 


organizations in a variety of program evaluation related activities, including:
 

Developing evaluation plans Collecting new data Interpreting results 

Designing survey instruments Analyzing data Developing reports 

Gathering existing data 

TECs can be a valuable partner in your program evaluation activities. If you are not familiar 

with the TEC in your area, you can learn more at www.tribalepicenters.org. 

9 

•

•

• •

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When? 

 

WHEN SHOULD I EVALUATE? 

Program evaluation should occur before, during, and after your program. Plans for evaluation should 

be a part of planning for the implementation of the program. The evaluation program helps establish 

a baseline from which changes in participants or the impact on the community can be tracked. 

Evaluation during program implementation can help you make immediate changes/adjustments in 

the program as it is progressing. This helps staff find the strengths and weaknesses in the program 

while it is still going on and is helpful for program improvement. Evaluation after the completion of 

a program is valuable to sum up what has occurred, assess the extent to which you have achieved 

your target outcomes, identify successes and lessons learned, and reflect on the value of the 

program. 

BEFORE 

Conduct needs 

assessment 

Establish baselines 

and targets 

Create evaluation 

plan that aligns with 

your program goals, 

objectives, 

and outcomes 

AFTER 

Assess achievement 

of outcomes 

Compare actual 

results to targets 

and standards 

Identify successes 

and lessons learned 

DURING 

Process evaluation 

Monitor if program 

activities are being 

implemented as intended 

Assess progress toward 

short-term outcomes 

Make immediate 

adjustments as the 

program is progressing 

Improve quality 

continuously 

10 

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Framework for Evaluating Tribal Traffic Safety Programs 

DESCRIBE 

THE PROGRAM 

JUSTIFY

CONCLUSIONS 

ENSURE USE 

AND SHARE 

LESSONS 

EVALUA

FOCUS 

DESIGN

TION 

HOW TO EVALUATE YOUR TRIBAL TRAFFIC SAFETY PROGRAM 

The remainder of this guide is designed to help you answer this final question: “How do I evaluate 

my Tribal Traffic Safety Program?” The framework described below is a practical, non-prescriptive tool 

that summarizes the important elements of program evaluation. It was developed by the Centers for 

Disease Control and Prevention (CDC), and offers a way to understand and improve your program 

evaluation using methods that are useful, feasible, proper, and accurate. The framework contains 

two related dimensions: 1) steps in evaluation practice, and 2) standards for good evaluation. 

A FRAMEWORK FOR PROGRAM EVALUATION 

ENGAGE 


PARTNERS
 

STANDARDS
 

UTILITY
 

FEASIBILITY
 

PROPRIETY
 

ACCURACY
 

GATHER 

CREDIBLE 

EVIDENCE
 

11 
12 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Framework for Evaluating Tribal Traffic Safety Programs 

The six connected steps of the framework are actions that should be a part of any evaluation: 

1. Engage partners

2. Describe the program

3. Focus the evaluation design

4. Gather credible evidence

5. Justify conclusions

6. Ensure use of findings and share lessons learned

These steps are meant to be adaptable, not rigid. They are intended to serve as starting points 

around which Tribal Traffic Safety Programs can tailor an evaluation to best meet their needs. 

Sensitivity to each program’s unique context (e.g. culture, history, capacity, and organizational 

climate) is essential for impactful evaluation. 

The second part of the framework is a basic set of standards to assess the quality of evaluation 

activities. There are 30 specific standards, organized into the following four groups: 

Utility: Who needs the evaluation results? Will the evaluation provide relevant information in a 


timely manner for them? 


Feasibility: Are the planned evaluation activities realistic given the time, resources, and 


expertise at hand? 


Propriety: Does the evaluation protect the rights of individuals and the welfare of those involved? 

Does it engage those most directly affected by the program, such as participants or the 

surrounding community? 

Accuracy: Will the evaluation produce findings that are valid and reliable, given the needs of 


those who will use the results? 


Each of these steps and standards will be explored in detail throughout the remainder of this guide. 

•

•

•

•
13 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 1: Engaging Stakeholders

STEP 1: ENGAGING PARTNERS 

Partners are people or organizations that are invested in the program or that are interested in the 

results of the evaluation or what will be done with results of the evaluation. 

Key partners for evaluations of public health programs typically fall into three major groups: 

Those involved in program operations, i.e., program management, staff, and partnering programs 

Those served or affected by the program, i.e., tribal members 

Those who are intended users of the evaluation findings, i.e., tribal leaders, program partners,
 

funding agencies, and other persons in a position to make decisions about the program
 

WHY INVOLVE PARTNERS IN 

EVALUATION? 

Evaluation cannot be done in isolation. Almost 

everything done in programs to improve 

community health and wellness involves 

partnerships. Therefore, program evaluation 

must consider the different values and interests 

held by the partners. Partners must be part 

of the evaluation to ensure that their unique 

perspectives are understood. When partners are 

not appropriately involved, evaluation findings 

are likely to be ignored, criticized, or resisted. 

However, if they are part of the process, people 

are likely to feel a good deal of ownership for the 

evaluation process and results. That’s why this 

evaluation cycle begins by engaging partners. 

GIVE PRIORITY TO PARTNERS WHO: 

Can increase the credibility of your efforts or 

your evaluation. 

Are responsible for day-to-day implementation 

of the activities that are part of the program. 

Will advocate for or authorize changes to the 

program that the evaluation may recommend. 

Will support or authorize the continuation or 

expansion of the program. 

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 1: Engaging Stakeholders

TYPES OF PARTNERS IN TRIBAL TRAFFIC SAFETY PROGRAMS 

Although each Tribal Traffic Safety Program is unique, some common partners may include, but are 

not limited to: 

Tribal leaders I/T/U Health Facilities Emergency Medical Services 

Schools Environmental Health State and Government 

Head Start Community Members Women Infants and Children (WIC) Program 

Day Care Centers Bureau of Indian Affairs Community Health Representatives (CHR) 

Law Enforcement Indian Health Service Transportation Department 

TIPS FOR ENGAGING PARTNERS 

✔ Include a diverse group of partners. 

✔ Identify key areas for partner input. 

✔ Create a plan for partner engagement. 

✔ Involve partners at the beginning, middle, and end of your program evaluation activities. 

✔ Include partners at various levels based on their preferences, interests, and expertise: 

Direct involvement in designing and conducting the evaluation 

Advisory role providing feedback and guidance on program evaluation activities and 

reports through routine communication 

✔ Keep partners informed regarding the progress of the evaluation through periodic meetings, 

reports and other preferred means of communication. 

✔ Partners are more likely to support the evaluation and act on results and recommendations 

if they are involved in the evaluation process. 

TIP 

14 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
15 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

STEP 2: DESCRIBING THE PROGRAM 

This step centers upon developing a comprehensive description of your Tribal Traffic Safety Program 

that outlines all of its key components and intended outcomes. Completing this step will help you 

focus your evaluation on the most central and important questions. This step can either follow the 

partner engagement step or precede it. It is important to note that in this step you are describing 

the program and not the evaluation. You can use a tool called a logic model for this purpose (see 

page 19), but a program description can be developed without using this or any tool. Either way, a 

comprehensive program description includes the following components: 

Need. What is the big public health problem you aim to address with your program? 

Focus. Which individual groups, programs, departments, or organizations need to change or take 

action to ensure progress on the public health problem? 

Outcomes. How and in what way do these groups, programs, departments, or organizations need 

to change? What action specifically do they need to take? 

Activities. What will your program and its staff do to move these groups, programs, departments, 

or organizations to change/take action? 

Outputs. What tangible capacities or products will be produced by your program’s activities? 

Resources/Inputs. What is needed from the larger environment in order for the activities to be 


successful? 


Each of these components will be further described in the remainder of this section. 

NEED FOR PROGRAM 

The need is the public health or other problem addressed by the program. For most Tribal Traffic 

Safety Programs the public health problem is motor vehicle crash injuries and deaths. You 

might define the need, in terms of its consequences for your tribe or community, the size of the 

problem overall, the size of the problem in various segments of the population (e.g., children and 

adolescents), and/or significant changes or trends in the rate of motor vehicle crash injuries. For 

example, the need for a particular Tribal Traffic Safety Program might be to reduce the high rate of 

motor vehicle crash injuries to American Indian children due to insufficient child passenger safety 

restraint use (i.e., use of weight, height, and age appropriate car seats, booster seats, and seat 

belts among children). 

FOCUS 

Focus on the various audiences the program needs to engage in order to make progress on the 

public health problem. For Tribal Traffic Safety Programs this might include tribal members (e.g., 

parents of infants, children, adolescents, adults, elders, or the whole community), as well as 

key programs or departments (e.g., law enforcement, Community Health Representatives, tribal 

leadership, etc.). Reducing motor vehicle crash injuries will require not only individual-level actions by 

tribal members (e.g., using seat belts, car seats, and booster seats), but also community, system, 

and environmental level actions such as improvements to roads, increased signage, enforcement of 

laws, distribution of child safety seats, community education, etc. 

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

 

OUTCOMES 

Outcomes are the changes in someone or something (other than the program and its staff) that 

you hope will result from your program’s activities. For Tribal Traffic Safety Programs, the ultimate 

outcome is often ambitious and long-term, such as reducing motor vehicle crashes with injuries or 

decreasing motor vehicle-related fatalities. A strong program description provides details not only 

on the intended long-term outcomes, but also on the short-term and intermediate outcomes that 

precede it. 

SAMPLE SHORT-TERM, INTERMEDIATE-TERM, AND LONG-TERM 

OUTCOMES FOR TRIBAL TRAFFIC SAFETY PROGRAMS 

OUTCOME TYPE DEFINITION EXAMPLES 

Short-Term 

The initial expected changes in 

your population(s) of focus after 

implementing certain activities 

or interventions (e.g., changes in 

knowledge, skills, and attitudes). 

Increased tribal member skills in 

installing car seats 

Increased knowledge about safe driving 

and the harms of distracted and alcoholimpaired driving 

Enhanced collaboration/communication 

among program partners (e.g., Tribal 

Traffic Safety Program, law enforcement, 

CHR program, and schools) 

IntermediateTerm 

The interim results that provide 

a sense of progress toward 

reaching the long-term objectives 

(e.g., changes in behavior, 

practices, actions, and policy) 

Intermediate outcomes specify 

what individuals do with the 

short-term outcomes. 

Increased use of seat belts among tribal 

members 

Increased use of car seats and booster 

seats for children 

Increased citations for speeding, 

impaired driving, and failure to use 

seat belts 

Primary seat belt law passed and 

implemented in the tribal community 

Long-Term 

Long-term outcomes are those 

that result from the achievement 

of your short- and intermediateterm outcomes. Typically, but 

not always, these mirror the goal 

statement, and are achieved only 

after the program has been in 

place for some time. 

Decreased motor vehicle crash-related 

mortality 

Fewer motor vehicle crashes that result in 

injury 

Improved quality of life 

Increased life expectancy 

16 

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

ACTIVITIES 

Activities are the actions taken by the program and its staff to achieve the desired outcomes in 

the groups of focus. Typical program activities might include outreach, training, education, service 

delivery, collaborations and partnerships, and health communication. Most Tribal Traffic Safety 

Programs will engage in numerous activities to achieve their desired outcomes, such as those 

highlighted in the text box below. 

SAMPLE ACTIVITIES OF TRIBAL TRAFFIC SAFETY PROGRAMS 

✔ Conduct car seat installations 

✔ Conduct community education/ 

presentations 

✔ Conduct observational seat belt checks 

✔ Provide classroom education in high 

schools about the hazards of distracted 

driving 

✔ Review existing tribal laws and policies that 

relate to traffic safety 

✔ Recruit partners to participate in a 

community traffic safety coalition 

✔ Implement a community scan for road and 

signage improvement needs 

✔ Identify training opportunities for 

program staff 

✔ Coordinate safe driving courses 

✔ Gather existing data on motor vehicle 

crashes and injuries in the community 

✔ Advertise program events 

✔ Recruit participants 

✔ Implement sobriety checkpoints 

✔ Create and disseminate small media (e.g., 

posters, digital stories, brochures, fact 

sheets, radio PSAs, etc.) 

✔ Conduct surveys 

✔ Share information on evidence-based 

traffic safety policy interventions. 

17 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

SAMPLE OUTPUTS FOR TRIBAL TRAFFIC SAFETY PROGRAMS 

✔ Consultations with partners or other 

partners around evidence-based traffic 

safety policies 

✔ Seat belt, DUI, and speeding checks 

conducted 

✔ Car seats checked 

✔ Coalition meetings conducted 

✔ Community presentations conducted and 

participants reached 

✔ Car seats installed 

✔ Outreach events/health 

fairs conducted 

✔ Tribal members participating in safe driving 

course 

✔ Memoranda of Agreement executed among 

partners 

✔ Programs/departments represented on 

coalition 

✔ Staff trained as child passenger safety 

technicians 

✔ Materials developed and disseminated 

Outputs are the direct products of activities, usually some sort of tangible deliverable. 

Outputs can be viewed as activities redefined in tangible or countable terms. For example, 

the number of: 

18 
19 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

RESOURCES/INPUTS 

These are the people, money, infrastructure, and other resources needed—usually from outside 

the program—to implement program activities effectively. It is important to include inputs in the 

program description because accountability for resources to funders and partners is often a focus 

of evaluation. Just as important, the list of inputs is a reminder of the type and level of resources 

on which the program is dependent. If intended outcomes are not being achieved, look to the 

resources/inputs list for one reason why program activities could not be implemented as intended. 

Examples of Resources/Input for Tribal Traffic Safety Programs 

Funding 

Program staff 

Partners (e.g., police, tribal leaders, schools, CHRs, etc.) 

Infrastructure 

Materials & equipment 

Data 

LOGIC MODEL 

Once the components of the program description have been 


identified (i.e., need, focus, outcomes, activities, outputs,
 

resources/inputs), a visual depiction might help to summarize the relationship among the 


components of your Tribal Traffic Safety Program. The logic model requires no new thinking about the 


program; rather, it converts the raw material in the program description into a picture of the program. 


This visual tool can help with both strategic planning and program evaluation. 


•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

In other words, logic models are graphic depictions of the relationship between a program’s activities 

and its intended outcomes. Two words in this definition bear emphasizing: 

Relationship: Logic models convey not only the activities that make up the program, but also 


how they relate to each other, as well as their relationship with your outputs and outcomes. 


Intended: Logic models depict intended outcomes of a program’s activities. As the starting 

point for evaluation and planning, the model serves as an “outcomes road map” that shows the 

underlying logic behind the program, i.e., why it should work. 

LOGIC MODEL COMPONENTS 

Logic models come in many different shapes and sizes. At a minimum, they should include 

the following components from your program description: 

Inputs: Resources that go into the program and on which it is dependent to execute 

its activities 

Activities: Actual events or actions done by the program and its staf 

Outputs: Direct products of program activities, often measured in countable terms 

(e.g., the number of education sessions held, number of car seats installed, etc.) 

Outcomes: The changes that result from the program’s activities and outputs, often 

in a sequence expressed as short-term, intermediate-term, and long-term outcomes 

20 

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 2: Describing the Program

 

SAMPLE LOGIC MODEL FOR TRIBAL TRAFFIC SAFETY PROGRAMS 

INPUTS ACTIVITIES OUTPUTS OUTCOMES 

• Program Staff • Car seat clinics • Number of car

Short-Term Intermediate Long-Term 

• Increased • Increased use • Decrease in

• Partners • Safe driving seats inspected number of of car seats motor vehicle

(police, school, classes & installed vehicles with • Increased use crashes

WIC, CHR, • Media • Number appropriate car of seat belts • Decrease in

transportation) campaign of classes seats • Increased motor vehicle

• Grant funding • Seat belt conducted • Increased enforcement of crash-related

• Data (State observation • Number knowledge seat belt and injuries and

Vital Records,

EMT logs, IHS • Car seat

observation

• Policy

development

of class

participants

• Number of

community

members

about safe

driving

practices

• Increased

community

car seat use death

• Increased life

expectancy

and quality of

life for AI/AN

reached awareness people

through media about traffic

campaign safety

• Number of • Improved

observations surveillance of

completed seat belt and

• Number of car seat use

policies drafted • Enhanced tribal

traffic safety

policies

21 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 3: Focusing the Evaluation Design

STEP 3: FOCUSING THE EVALUATION DESIGN 

After completing Steps 1 and 2, you and your partners should have a clear understanding of your 

Tribal Traffic Safety Program and are ready to focus the evaluation. This includes determining the 

most important evaluation questions and the appropriate design for the evaluation. Rather than 

waiting to conduct evaluation when the program ends and asking, “Did the program work?”, the 

framework outlined here views evaluation as an ongoing activity over the life of a program that asks, 

“Is the program working?” 

EVALUATION TYPES 

There are several types of evaluations. Some of the most common types include: formative, 

summative, process, outcome, and impact. In this guide, we will focus on two complementary 

evaluation types that all Tribal Traffic Safety Programs should conduct: process and outcome. 

PROCESS EVALUATION 

Process evaluation assesses whether or not a program is operating as intended and identifies 

opportunities to strengthen it. In process evaluation, you examine whether the planned activities 

are taking place, who is conducting the activities, who is reached through the activities, and whether 

sufficient inputs have been allocated or mobilized. You will also examine whether program activities 

have resulted in certain outputs. Remember, outputs are the direct products of the program activities 

which provide evidence that the activities were accomplished. 

Process evaluation should be conducted throughout the life of your program. 

The results of your process evaluation will strengthen your ability to report on your program 

accomplishments and can be used to improve future activities. 

PROCESS EVALUATION 

Process evaluation helps you to track program information related to who, what, when, 

where, and how? Such as: 

What has your program done? 

When did your program activities take 

place? 

Where did your program activities take 

place? 

Who received services? 

How much or for how long did they 

receive these services? 

What are the barriers/facilitators to 

implementating your program activities? 

22 

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 3: Focusing the Evaluation Design

OUTCOME EVALUATION 

Outcome evaluation measures the degree to 

which your Tribal Traffic Safety Program is being 

effective in meeting it’s objectives. Outcome 

evaluation measures a program’s results and 

helps determine whether a program or strategy 

produced the changes it intended to achieve. 

In other words, the main goal of outcome 

evaluation is to determine whether or not the 

program contributed to beneficial effects on 

your specific population. Outcome evaluation 

assesses the short-term, intermediate-term, and 

long-term effects of your program, and might 

include things such as: 

Changes in people’s attitudes and beliefs 

Changes in risk or protective behaviors 

Changes in the environment, including 


policies, enforcement of laws, roads and 


signage, and programs/departments 


Changes in morbidity and mortality 

PROCESS AND OUTCOME EVALUATION GO TOGETHER! 

It is important to note the usefulness of conducting process evaluation while you are also 

implementing outcome evaluation. For example, if the outcome evaluation shows that 

your Tribal Traffic Safety Program did not produce the expected results, it may be due to 

program implementation issues. These potential issues will be much easier to identify 

if you have been measuring them through process evaluation activities. Likewise, if your 

process evaluation demonstrates that your program is only reaching one subgroup of your 

population of focus (e.g., females), you would be less likely to expect change for other 

subgroups (e.g., males) in the community during outcome evaluation. Therefore, process 

and outcome evaluation are complementary, and it is recommended that all Tribal Traffic 

Safety Programs conduct both simultaneously! 

23 

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 3: Focusing the Evaluation Design

DEVELOPING EVALUATION QUESTIONS 

At this point, it is important to develop evaluation questions to focus and guide your program 

evaluation activities. Your evaluation questions should reflect the basic question of whether or not 

your program worked as intended. 

Many different questions can be selected to guide your program evaluation. Most can fit into four 

main categories: 

QUESTION CATEGORIES POSSIBLE QUESTIONS 

PLANNING AND IMPLEMENTATION: 

How well was your Tribal Traffic Safety 

Program planned, and how well was that 

plan put into practice? 

Were your program’s activities put into 

place as originally intended? 

How many events were conducted? 

Who participated? 

Was there diversity among participants? 

Why do participants enter and leave your 

program? 

Do those most in need of help receive 

services? 

Are community members satisfied with 

your program? 

ASSESSING ATTAINMENT OF OBJECTIVES: 

How well has the Tribal Traffic Safety 

Program met its stated objectives? 

How many people participated? 

Were activities being implemented 

according to your planned timelines? 

Did program activities yield your 

expected outputs? 

IMPACT ON PARTICIPANTS: 

How much and what kind of a difference has 

the Tribal Traffic Safety Program made for its 

population of focus? 

How has behavior changed as a result of 

participation in the program? 

What new knowledge or skills have been 

obtained by participants? 

IMPACT ON THE COMMUNITY: 

How much and what kind of a difference has 

the Tribal Traffic Safety Program made in the 

community as a whole? 

What new policies resulted from the 

program? 

Were there any changes in infrastructure 

or the environment? 

What new resources or infrastructure 

has been created by the program? 

24 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 3: Focusing the Evaluation Design

 

PRIORITIZED EVALUATION QUESTIONS 

Are important to your program and partners. 

Address important program needs. 

Can be answered with available resources, including funds and staff capacity. 

Reflect program goals and objectives. 

Are realistic and can be answered within the available timeframe. 

Provide information to make program improvements. 

TIP 

SAMPLE EVALUATION QUESTIONS FOR TRIBAL TRAFFIC SAFETY PROGRAMS 

Did the program follow the basic plan for implementation? 

What services were provided through the program (e.g., outreach, education, car seat 

installations/distribution, seat belt checks, coalition building, policy development, 

new signage, safe driving classes, DUI checks, etc.)? 

What was the quality of the services provided through the program? 

How much did the community participate in the program? 

What are the characteristics of the individuals who participated in the program? 

What is the participants’ satisfaction with the program? 

Who still needs to be recruited to participate? 

What are staff members’ perceptions of the program? 

What are the planning products of the programs (e.g., logic models, strategic plan, etc.)? 

What materials were developed through the program (e.g., video, brochures, curriculum, etc.)? 

How many program materials have been distributed? 

What were the barriers to participation in the program? 

What were the barriers to the implementation of the program? 

What resources are still needed to implement the program as planned? 

How much progress did the program make toward short-term, intermediate-term and 

long-term outcomes? 

What changes occurred among individuals that participated in the program? 

What community-level/system-level changes occurred as a result of the program? 

What educational opportunities were used to discuss evidence-based policy options 

around traffic safety with partners and policymakers? 

25 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

STEP 4: GATHERING CREDIBLE EVIDENCE 

Now that you have created your evaluation questions, your next task is to gather the evidence to 

answer them. Your evaluation should strive to collect information that will convey a well-rounded 

picture of your Tribal Traffic Safety Program. Therefore, you should plan to gather multiple types of 

evidence to evaluate your program. 

THE TWO MOST IMPORTANT STEPS TO GATHERING CREDIBLE EVIDENCE ARE TO: 

1. Select meaningful indicators (sometimes called performance measures).

2. Select appropriate data collection methods and sources to track your indicators.

Indicators are specific, observable, and measurable characteristics or changes that show the 

progress a program is making toward achieving a specified outcome. In other words, your indicators 

are the specific item(s) that you will measure to answer your evaluation questions (e.g., attendance, 

events conducted, meetings convened, people trained, new materials developed, changes observed, 

etc.). Indicators reflect aspects of the program that are meaningful for monitoring. They can be 

related to your process or outcome evaluation. Some examples of indicators that can be defined and 

tracked by your Tribal Traffic Safety Program are highlighted in the table below. 

SAMPLE INDICATORS FOR TRIBAL TRAFFIC SAFETY PROGRAM 

Process Indicators Outcome Indicators 

Attendance/participation Changes in participant behavior 

Participant characteristics (age, sex, job title, etc.) Changes in knowledge, attitudes, and beliefs 

Number of events/activities conducted Changes in policies 

Number of meetings held Changes in health status 

Number of data use agreements established 

Changes in enforcement of laws (seat belt 

warnings/citations, child safety seat warnings/ 

citations, DUI arrests) 

Number of assessments/observations 

conducted Changes in quality of life 

Number of materials developed & distributed Changes in services or practices 

Number of staff trained Changes in road conditions 

Number of classes held Changes in traffic signage 

26 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

 

 

 

 

 

Indicators should be developed for 

activities (process indicators) and for 

outcomes (outcome indicators). 

There can be more than one indicator for 

each activity or outcome. 

The indicator must be clear and specific in 

terms of what it will measure. 

Indicators should meaningfully address 

your evaluation questions. 

Indicators should align with your program 

goals and objectives. 

Select a mix of indicators at the 

individual-level (e.g., change in participant 

knowledge), community-level (e.g., 

number of car seats distributed throughout 

community), and systems/environmental 

levels (e.g., change in policies, 

availability of safe driving classes, roads 

improvements/signage, enforcement of 

seat belt, car seat, speeding, and impaired 

driving). 

TIP KEEP THE FOLLOWING TIPS IN MIND WHEN SELECTING YOUR INDICATORS: 

Another helpful tip for selecting indicators is to look at the SMART objectives you have established 

for your program. If your program objectives were written to be specific, measurable, action-oriented, 

realistic, and time-bound (so-called “SMART” objectives), they will likely include indicators as well. 

SMART OBJECTIVES 

Have you ever wondered why so many grants require you to include SMART objectives? 

S = Specific 

M = Measurable 

A = Action-oriented (achievable) 

R = Realistic 

T = Timebound 

Not only do SMART objectives provide details of your program plan, they also lay the foundation 

for your program evaluation by: 

indicating specific approaches that will be implemented. 

identifying key indicators or performance measures. 

establishing baselines and targets for your indicators. 


outlining timeframes for completion of key program components. 


27 

•

•

•

•

•

•

•

•

•

•
28 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

 

SELECTING DATA COLLECTION METHODS AND SOURCES 

Now that you have determined the activities and outcomes that you want to measure and the 

indicators that you will use to measure progress on them, you need to select data collection 

methods and sources for your indicators. A key decision is whether there are existing data 

sources—secondary data—that can be gathered to measure your indicators or whether you need to 

collect new data—primary data. Most likely you will use both! 

SECONDARY DATA SOURCES 

Secondary data are data that have already been collected by someone else. Depending on your 

evaluation questions and indicators, some secondary data sources might be appropriate. For example, 

existing data sources that might be particularly useful for Tribal Traffic Safety Programs include: 

Local: 

EMT/EMS logs Tribal Departments of Transportation 

Police logs, reports, records Community surveys and needs assessments 

IHS/Tribal/Urban Indian Health facility Maps 

electronic health records 

State: 

Vital records/statistics (mortality, death certificates) 

Hospital discharge data (motor vehicle crashes that resulted in hospitalization) 

Emergency department data (motor vehicle crashes that resulted in an ED visit) 

State Departments of Transportation/State Highway Safety Offices 

National: 

NHTSA’s Fatality Analysis Reporting System (FARS), data regarding fatal injuries suffered in 

motor vehicle traffic crashes 

CDC’s Web-based Injury Statistics Query and Reporting System (WISQARS), an interactive, 

online database that provides fatal and nonfatal injury data 

CDC’s Behavioral Risk Factor Surveillance System (BRFSS), a national survey of health-related 

risk behaviors among adults, including seat belt use and drinking and driving data 

CDC’s Youth Risk Behavior Surveillance System (YRBSS), a national survey of health-related 

risk behaviors among adolescents, including seat belt use, texting and driving, and drinking and 

driving data 

Inventory of National Injury Data System, a comprehensive list of 45 different federal data 

systems operated by 16 different agencies and 3 private injury registry systems that provide 

nationwide injury-related data 

CDC’s Pregnancy Risk Assessment Monitoring System (PRAMS), a system that collects statespecific, population-based data on maternal attitudes and experiences before, during, 

and shortly after pregnancy 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
29 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

Reach out to your Tribal Epidemiology Center or 


State Department of Health for technical support 


TIP in accessing secondary data that may be helpful 


to evaluate your Tribal Traffic Safety Program. 


PRIMARY DATA SOURCES 

Primary data are data that are collected by your evaluation team to answer your evaluation 

questions. Primary data can be quantitative or qualitative. It is important that data are collected in 

an ethical manner. Please see Propriety Standards on Page 50 for additional information. 

Quantitative data are data that can either be counted or compared on a numeric scale. These 


data are used to measure a problem or address the questions of “what?” or “how many?”. 


Qualitative data are data concerned with descriptions, which can be observed but cannot 

be computed. Qualitative data describes qualities or characteristics. It is collected using 

questionnaires, interviews, or observation, and frequently appears in narrative (words) form. Data 

collected include transcripts of interviews and focus groups and field notes from observation of 

certain program activities. 

Primary data collection methods for program evaluation typically fall into several broad categories. 

Among the most common that are used to evaluate Tribal Traffic Safety Programs include: 

Surveys 

Key Informant Interviews 

Focus Groups 

Observation 

Document Review 

The remainder of this section will provide definitions of each primary data collection type and tips for 

how you can incorporate them into the evaluation of your Tribal Traffic Safety Program. 

•

•

•

•

•

•

•
30 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

1. SURVEYS

A survey is a way of collecting information to understand the views of the community or group in 

which you are interested. Survey instruments or questionnaires ask questions in a standardized 

format that allows consistency and the ability to aggregate (combine) responses. Most survey 

questions are quantitative and closed-ended, requiring participants to select an answer from a 

given list. However, surveys may also include some open-ended (qualitative) questions that allow 

participants to provide an answer in their own words. Some common types of surveys used in 

program evaluation include: 

Pre-Post Test – A survey instrument given to participants before and after a training, an 

intervention, or a particular period of time to assess changes in participant knowledge, attitudes, 

beliefs, and/or intentions. The same survey is administered at two different time points, and the 

changes are evaluated to determine impact. 

Satisfaction Survey – A satisfaction survey is typically administered only after an event, training, 

or intervention to assess participant satisfaction with program services and activities. 

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

WHO SHOULD I SURVEY? 

Deciding who you will survey (your sample) 

depends on the purpose of the survey. If you are 

trying to evaluate the impact of your project on 

people who have participated in your activities, 

you should administer the survey to those individuals who have participated in one or more of 

your program activities. Similarly, if you are trying 

to determine the impact of your project on specific groups of people, such as youth, parents, 

adults, women, etc., you may want to just survey 

people in the community with those characteristics. If you are trying to understand the impact 

of your program on the entire community, you 

should administer the survey to a much broader 

group of participants, including those who did 

not participate in project specific activities. This 

could be a random sample of community members, where everyone in your community has an 

equal chance of being selected to participate. 

Random samples are often drawn from existing 

lists of community members (e.g., voter registration, tribal enrollment/census, mailing lists, IHS/ 

Tribal/Urban Indian Health facility users, etc.), or community maps (i.e., random selection of homes 

throughout the community). Other options include: 1) census, where all community members are 

selected to participate; 2) convenience sample, where survey participants that are easy to reach 

are selected (e.g., surveying participants gathered at an event, at the post office, in the clinic, etc.); 

and/or 3) volunteers, where individuals self-select to participate. 

SAMPLE TYPE PROS CONS 

Census or Random Sample More representative of the 

whole community 

More expensive/resource 

intensive 

More time consuming 

Convenience Sample 

or Volunteers 

Quicker 

Less expensive/resource 

intensive 

Usually not representative 

of the whole community 

Results cannot be 

generalized 

31 

•

•

•

•

•

•

•
32 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

10 BEST PRACTICES FOR CREATING EFFECTIVE SURVEYS
 

1.	 Keep your survey short. In most cases, people are doing you a favor by taking your survey.

What better way to respect their time than by not taking up too much of it? You’ll be rewarded

with a higher completion rate as well as more thoughtful responses for the questions you end

up including.

2.	 Keep your questions clear and simple. Be specific and concrete in developing your questions

and be sure to avoid technical words or jargon that are commonly used among your Tribal Traffic

Safety Program staff and colleagues, but might not be well known to your community members.

3.	 Ask only questions that will help you meet your goal. Every question you include should have a

well-defined purpose and a strong reason for being there. Remove any questions that appear to

duplicate one another or will not provide information that is helpful to your evaluation.

4.	 Place the more personal questions to the end. Structure your survey like a conversation.

Would you start an exchange by asking someone how old they are? Probably not. Instead, you’d

engage in small talk first, and gradually move on to more personal topics. Similarly, keep your

early set of questions light and straightforward and then slowly move towards more personal

questions (often taking the form of demographic questions).

5.	 Focus on using closed-ended questions. What do we mean by closed-ended questions? We’re

talking about questions that use pre-populated answer choices for the respondent to choose

from—like multiple choice or checkbox questions. These questions are easier for respondents

to answer and provide you with quantitative data to use in your analysis. Open-ended questions

(qualitative) ask the respondent for feedback in their own words. Since open-ended questions

can take much longer to answer, try to only include 1-2 of them at the end of your survey.

6.	 Consider including a survey incentive. If you’re keen on getting a lot of responses, an incentive

in some form can prove helpful. Potential incentives range from entering respondents into a

drawing to giving respondents a gift card if they answer all of your questions.

7.	 Don’t ask leading questions. In other words, try not to put your own opinion into the question.

Doing so can influence the responses in a way that doesn’t reflect participants’ true experiences.

For example, a leading question might ask: You are satisfied with our trainings, aren’t you?

The right question to ask in this context would be:

How satisfied are you with our training?

F Very Satisfied

F Somewhat Satisfied

F Neither Satisfied nor Dissatisfied

F Somewhat Dissatisfied

F Very Dissatisfied
33 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

8.	 Keep your answer choices balanced. Using answer choices that lean a certain way can result in

respondents providing inauthentic feedback. Let’s revisit the prior example. Here’s how a set of

unbalanced answer choices (that lean towards being too positive) can look for that question:

F Very Satisfied

F Somewhat Satisfied

F Neither Satisfied nor Dissatisfied

And here is how they would look once balanced: 

F Very Satisfied 

F Somewhat Satisfied 

F Neither Satisfied nor Dissatisfied 

F Somewhat Dissatisfied 

F Very Dissatisfied 

9.	 Stay away from asking double-barreled questions. Double-barreled questions ask for feedback

on two separate things within a single question.

Here’s an example: “How would you rate the quality of our training and brochures?”

How would the respondent answer this question? Would they address the quality of the

training? The quality of the brochures that you developed? Maybe they’d skip the question out of

confusion. Also, how would you interpret their answer? Were they responding about the training,

the brochures, or both? You can fix a double-barreled question by either choosing one thing to

ask or by breaking the question up into two separate questions.

10.	 Preview and pilot test your survey before you use it. Imagine using your survey only to realize

that you forgot to add a question. Or that you didn’t include a few essential answer choices for

one of the questions you asked. In either case, you’ll probably end up being frustrated and get

results that fall short of what you need. To prevent any mishaps in your survey design, preview

your survey and share it with others so they can catch any mistakes you might not find on your

own. Even better, pilot test your survey instruments with a small group of participants similar to

the group of focus to assess its clarity, comprehensibility, and cultural appropriateness.

TIP When developing a new survey, always look for existing questions or instruments 

that have been previously tested and can be adapted or used as-is! 
34 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

2. KEY INFORMANT INTERVIEWS

Key informant interviews are a qualitative data collection method that, like open-ended questions 

in a survey, allow you to obtain an individual’s response in their own words. Interviews differ from 

surveys in that they elicit more detailed qualitative data and allow you to interact with the person 

to better understand their response. Interviews may be conducted in-person or over the phone. 

Interviewing is useful when you want more in-depth information about a person’s knowledge, 

attitudes/beliefs, or behaviors. Key informant interviews can be useful for Tribal Traffic Safety 

Programs looking to gather in-depth information from a select number and type of individuals, such 

as tribal leaders, program administrators, law enforcement, partnering agencies, etc. 

3. FOCUS GROUPS

Like interviews, focus groups allows you to collect qualitative data. However, unlike interviews, in 

which data are collected by one-on-one interactions, focus groups provide data about a particular 

topic through small group discussions. Focus groups are an excellent method for obtaining opinions 

about programs and services. They produce information from many people in a short period of time, 

so they can be an effective method when information is needed quickly. A facilitator guides the group 

based on a predetermined set of topics. The facilitator creates an environment that encourages 

participants to share their perceptions and points of view. Focus groups typically are conducted 

with participants who share common traits (e.g., gender, adolescents, parents, people who have 

participated in your project activities, people who have not participated in your project activities). 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

Develop a guide with key themes or 

questions that you would like to explore in 

each interview or focus group. This will be 

used by the facilitator or interviewer, not 

shared with the participants. 

Keep the guide to no more than 10 key 

themes or questions. 

List subthemes or probes under each 

theme or question. You can use these 

probes if the topics you want to cover 

do not emerge on their own during the 

interview or focus group. 

Audio record each session (with permission 

from participants) to make sure that you 

capture everything that was said. 

Take notes even if you are recording so that 

you have a backup. 

Aim to keep interviews to no more than one 

hour in length, and focus groups under 1.5 

hours. 

Have at least two people assist with the 

focus group if possible—one to moderate 

the session and the other to take notes 

and handle logistics. 

TIP TIPS FOR CONDUCTING KEY INFORMANT INTERVIEWS AND FOCUS GROUPS 

4. OBSERVATION

Observation is a way of gathering data by watching behavior and events and noting physical 

characteristics in their natural setting. Observation allows you to gather information about a program 

as the program’s activities occur. It can collect quantitative data, qualitative data, or a mix of both. 

Examples could be observing services being provided, training sessions, meetings, community 

member behaviors (e.g., seat belt/car seat use), the environment (e.g., traffic patterns), etc. It is 

best for observation to be done in an unobtrusive manner to ensure that you will witness what really 

takes place. Some types of observation that are useful for evaluating Tribal Traffic Safety Programs 

include: 

a.	 Seat belt/car seat checks These observational activities use a formal protocol to find out

how many people in your community are using seat belts or car seats.

b.	 Windshield/walking survey Windshield surveys are systematic observations made from

a moving vehicle. Walking surveys are systematic observations made on foot. Either can

help you better understand specific conditions that your Tribal Traffic Safety Program might

be trying to impact such as traffic patterns, road conditions, signage, road safety, and

infrastructure needs.

Prior to conducting an observation, always develop a checklist of what you 

plan to observe so if you become distracted, you won’t forget what to look for! 

35 

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

5. DOCUMENT REVIEW

Document review might also be helpful when evaluating your Tribal Traffic Safety Program. You can 

review meeting minutes, sign-in sheets, progress reports, medical records, logs, maps, laws, and 

policies to learn more about the activities of your program and its reach and impact. 

To ensure that you have plenty of documents to review, be sure to collect data at each of your Tribal 

Traffic Safety Program activities and events, including sign-in sheets, meeting minutes, satisfaction 

surveys, etc. It’s also a good idea to debrief with your staff shortly after each major activity to 

discuss what worked well and what needs improvement for the future. Try using the plus/delta 

technique, an easy-to-use feedback tool. 

TIPS FOR DOCUMENT REVIEW OF POLICIES/LAWS 

Rather than just evaluating whether or not specific laws and policies exist in your community, it is 

helpful to review the specific elements of these laws and policies to determine their alignment with 

proven, evidence-based practices. The list below shows some examples of criteria that can be used 

to evaluatie laws and policies. 

DOCUMENTING LAWS FOR IMPAIRED DRIVING 

F Is there a law? Is it a Tribal law, or does it defer to state law? 

F How is the term “impaired” defined (that is, alcohol, drugs, prescription medication)? 

F Does the law describe how “impairment” must be measured, and if so, are those measurements 

different from state laws and best practices? 

F What are the consequences for first-time offenders? To what degree do those consequences 

differ from state laws and best practices? 

F What are the consequences for repeat offenders? To what degree do those consequences differ 

from state laws and best practices? 

F Does the law include a “child endangerment” clause if children are in the vehicle at the time of 

arrest? 

F Does the law include “court diversion” or alternate sentencing options, and if so, who manages 

those programs? 

F Does the law stipulate how funds generated from fines or court fees are used (that is, do those 

funds support on-going traffic safety efforts)? 

36 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

DOCUMENTING LAWS FOR SEAT BELT AND CHILD SAFETY SEAT USE 

F Is there a law? Is it a Tribal law, or does it defer to state law? 

F Is the law a primary or secondary enforcement law? 

F Who is covered by the seat belt law (i.e., front seat and rear seat occupants, or front-seat only)? 

F What is covered by the child safety seat law (i.e., ages, location of seat, and/or types of seats)? 

F What are the fines or other penalties for not using a restraint and how do they compare to those 

at the state level and best practices? Do the fines differ for non-use of a child safety seat versus 

improper use of a child safety seat? 

F Does the law describe increased consequences for repeat offenders? 

F Does the law include a “court diversion” program and if so, who manages the court diversion 

program? 

F Does the law stipulate how funds generated from citations or court fees are used (that is, do 

those funds support on-going traffic safety efforts)? 

37 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

PROS & CONS OF DATA COLLECTION/SOURCE TYPES
 

DATA COLLECTION/ 

SOURCE TYPE PROS CONS 

Surveys 

Standardized questionnaires 

that ask predetermined 

questions 

• Easy to do with a large number of people

• Easy to conduct analysis

• Easy to repeat at a later time (pre-post)

• Allows control over the content that is

being measured

• Can be highly accurate

• Allows for comparisons with other/larger

populations when survey items come

from existing instruments

• Little opportunity to explore issues in

depth

• Low response rates are common

• Time-consuming if you need a large

number of participants

• Relatively high cost

• Relatively slow to design, implement,

and analyze

• Accuracy depends on who and how many

people participate

Key Informant • Possible to explore issues in depth • Interviewer’s presence and

Interviews • Personalized approach

• Low cost (assuming relatively few)

characteristics might bias results

• Small sample; not generalizable

Structured or unstructured 

one-on-one directed 

conversations with key 

• Allows respondents define what is

important and in their own words

• Can clarify responses through probes

• Can be time consuming to set up

interviews with busy informants

• Might be difficult to analyze and

summarize findings due to large

individuals or leaders in a • May lead you to other data sources and volume of transcripts and diverse

community other key informants perspectives

Focus Groups 

Structured interviews with 

small groups of like individuals 

using standardized questions, 

follow-up questions, and 

exploration of other topics that 

arise to better understand 

participants 

• Less expensive

• Rapid data collection

• Faster than one-on-one interviews

• Can clarify participant responses through

probing (follow-up questions)

• Often generate new/fresh ideas

• Allow respondents define what is

important and in their own words

• Requires a skilled facilitator to keep

conversation flowing and everyone

involved

• Can be time consuming to assemble

groups

• Less control over process than key

informant interviews

• Difficult to collect sensitive information

• Limited generalizability

• Might be difficult to analyze and

summarize findings

38 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

DATA COLLECTION/ 

SOURCE TYPE PROS CONS 

Observations • Natural setting

• Does not rely on people’s willingness

or ability to provide information

• Observer might bias behavior of

participants

• Difficult to generalize observations to Watching behaviors or • Allows you to directly see what people the larger populations

events or noting physical do rather than relying on what people • Does not increase your understanding

characteristics say they did of why people behave as they do

Document Review • Low cost

• Might be difficult to access

• Might not be current or updated

Reviewing documents, records, • Relatively quick

• Often allows for historical

• Data are limited to what has already

been collected

or archives held by your comparisons or trends • Might not answer your specific project or partnering agencies evaluation questions

Gathering 

Secondary Data 

• Less expensive

• Less time consuming

• Might be immediately available for use

• Data are limited to what has already

been collected

• Might not answer your specific

evaluation questions

• Data sharing limitations

• Data might not be tribe-specific

39 
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 4: Gathering Credible Evidence

SELECT SOURCES OF EVIDENCE FOR TRIBAL TRAFFIC SAFETY PROGRAM EVALUATION 


Who might you survey or interview? 

Program participants (or nonparticipants) 

Staff, program managers, administrators 

General public 

Partner agency staff 

Tribal leaders 

What might you observe? 

Coalition meetings 

Educational events/trainings 

Car seat checks/installations 

Seat belt use 

Signage 

Road conditions 

Traffic patterns 

Which documents might you analyze? 

Meeting minutes 

Sign-in sheets 

Event and activity logs 

Previous evaluation reports 

Road maps 

Photographs 

News articles 

Medical records or other files 

Law enforcement logs 

Tribal policies and protocols 

Progress reports 

USING MIXED METHODS APPROACHES FOR PROGRAM EVALUATION 

A mixed methods approach combines at least one qualitative and one quantitative data 

source or collection method for program evaluation. 

Selecting multiple data sources provides an opportunity to include different perspectives 

regarding the program and might enhance your evaluation’s credibility. Sometimes a single 

method is not sufficient to accurately measure an activity or outcome because the thing 

being measured is complex. Using multiple methods helps increase the accuracy of the 

measurement and the certainty of your conclusions when the various methods yield similar 

results. Combining methods maximizes the strengths and minimizes the limitations of 

each method, which will provide you and your partners increased confidence in the overall 

findings. 

40 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
41 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 5: Justifying Conclusions

STEP 5: JUSTIFYING CONCLUSIONS 

This step includes: 

Analyzing the evidence gathered in step 4

Making claims about your Tribal Traffic Safety Program based on the analysis 

Justifying claims by comparing the evidence against standards selected by your program 

Making recommendations for continuing, expanding, modifying, or terminating the program 

ANALYZYING EVIDENCE 

Data analysis is the process of evaluating data using analytical and statistical tools to discover 

useful information and aid in decision making. There are several data analysis methods, most of 

which can be categorized as quantitative or qualitative data analysis. 

QUANTITATIVE DATA ANALYSIS: KEY STEPS 

Enter the data into a database and check for errors. If you are using secondary data such as BRFSS 

or PRAMS, the data have already been checked, entered, and tabulated by those conducting the 

survey. Select the computer program you will use and determine who will enter and analyze the data, 

and determine who will enter, check, tabulate, and analyze the data. 

Tabulate the data to provide information (such as a number or percentage) for each indicator. Some 

basic calculations include determining: 

The total number of participants, trainings, The number of times a certain value 

activities, etc. appears (frequency) 

The number of participants achieving the The difference between the highest score 

desired outcome and lowest score observed (range) 

The percentage of participants achieving the The lowest/smallest score in a data set 

desired outcome (min.) 

The percent increase (or decrease) observed The highest/largest score in a data set 

The average (mean) score (max.) 

For more specific information about analyzing quantitative evaluation data click here or check out 

the additional resources listed in the appendix. 

•

•

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 5: Justifying Conclusions

QUALITATIVE DATA ANALYSIS: KEY STEPS 

Transcribe your interviews or focus groups from your notes and recordings. 

Become familiar with the data. It is important to read and re-read the data, writing down 

impressions, looking for meaning and determining which pieces of data have value. 

Focus the analysis. In this step, it is important to identify key questions that you want to answer 

through the analysis. One approach would be to focus the analysis on the answers to a particular 

question or topic. Another way to focus the analysis would be to examine the data as it relates to a 

particular group (e.g., elders, teens, women, etc.). 

Categorize the data and create a framework. This is often referred to as coding or indexing the 

data. Begin by identifying ideas, concepts, behaviors, key words, and phrases that frequently emerge 

in your data. A code is then assigned to those pieces of data in an effort to label the data and make 

it easier to organize and retrieve. A coding plan helps to provide a framework that will structure, label 

and, define the data. 

Identify patterns and make connections. Review codes and patterns of codes to identify themes 

and relationships between themes. 

Interpret the data and explain findings. After themes, patterns, connections, and relationships 

are identified, attach meaning and significance to the data. It can be helpful in this process to 

develop lists of key ideas, create diagrams, or use models to explain the findings. It is important to 

remember that qualitative data are not suited to make generalizations across a population. 

For more specific information about analyzing qualitative evaluation data, click here or check out the 

additional resources listed in the appendix. 

TRIANGULATION 

Triangulation involves combining different sources of information in your evaluation to reach 

a larger understanding of the impact of your Tribal Traffic Safety Program. These different 

sources might include: 

Quantitative data (surveys) 

Qualitative data (focus groups and interviews) 

Primary data (data that you collect) 

Secondary data (existing data that you use) 

One of the main benefits of triangulation is that it uses a range of data to provide more 

comprehensive insight into your program and that is particularly useful as a way to validate 

and strengthen your evaluation findings. 

42 

•

•

•

•
43 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 5: Justifying Conclusions

STANDARDS 

Once you have analyzed your evaluation data, the next step is to interpret the findings to make 

claims and judgements about your Tribal Traffic Safety Program. This step is accomplished by 

comparing the evidence against standards selected by your program. Standards establish a 

comparison by which your program can be judged and be considered successful. The box below lists 

possible standards you might use for determining the progress and/or success of your program: 

Needs of the participants/community Change in performance since baseline 

Program mission Professional or existing standards (e.g., 

Program SMART objectives Healthy People 2030, Countermeasures 


that Work: A Highway Safety 
 Performance by previous or similar Countermeasure Guide for State Highway programs Safety Offices, The Guide to Community Performance by a comparison group Preventive Services) Targets or fixed criteria of performance Mandates, policies, statutes, regulations, Change in performance over time laws 

Your standards for performance and outcomes should be determined at the beginning of your 

program, based on achievable targets, and aligned with you goals and SMART objectives. It is also 

important to note that multiple standards can be applied to a given program. 

INTERPRETING THE FINDINGS AND MAKING JUDGEMENTS AND RECOMMENDATIONS 

Interpretation is the effort of figuring out what your findings mean about your Tribal Traffic Safety 

Program. Evaluation evidence must be interpreted to determine the practical significance of what 

has been learned. Judgements are statements concerning the merit, worth, or significance of the 

program. They are formed by comparing the findings and interpretations regarding your Tribal Traffic 

Safety Program against one or more of your selected standards. Recommendations are actions for 

consideration resulting from the evaluation. Sharing draft recommendations, soliciting reactions from 

multiple partners, and presenting options instead of directive advice will increase the likelihood that 

recommendations are relevant and well-received. 

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 5: Justifying Conclusions

Interpret evaluation results with the goals 

of your program in mind. 

Keep your audience in mind. What do they 

need and want to know? 

Consider the limitations of the evaluation. 

Consider the possible biases of the 

individuals conducting the evaluation. 

Consider alternative explanations for your 

results. 

Compare program outcomes with those of 

previous years. 

Compare actual outcomes with intended 

outcomes. 

Explore how your results compare with 

those of similar programs. 

Check to see if the different data collection 

methods used to measure your progress 

show similar results. 

Evaluate whether or not your results 

are similar to what you expected. If not, 

consider why they may be different. 

Keep all recommendations relevant and 

feasible. 

TIPS TO REMEMBER WHEN INTERPRETING YOUR FINDINGS 

AND MAKING JUDGEMENTS & RECOMMENDATIONS TIP 

44 

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 6: Ensuring Use and Sharing Lessons Learned

STEP 6: ENSURING USE AND SHARING LESSONS LEARNED 

Lessons learned in the course of an evaluation do not automatically translate into informed decisionmaking and appropriate action. This final step is therefore critical to ensure that the evaluation 

processes and findings are used and disseminated appropriately. 

REMEMBER YOUR PURPOSE! 

The ultimate purpose of program evaluation is to use the information to improve your Tribal Traffic 

Safety Programs to maximize its benefits for the community. The purpose(s) you identified early in 

the evaluation process should guide the use of the evaluation results. The evaluation results should 

be used to demonstrate the effectiveness of your program, identify ways to improve your program, 

modify program planning, demonstrate accountability, and justify funding. 

ADDITIONAL PURPOSES OF PROGRAM EVALUATION 

Demonstrate to legislators or other 

partners that resources are being well 

spent and that the program is effective 

Aid in forming budgets and to justify the 

allocation of resources 

Compare outcomes with those of 

previous years 

Compare actual outcomes with intended 

outcomes 

Suggest realistic intended outcomes 

Support annual and long-range planning 

Focus attention on issues important 

to your program 

Ppromote your program 

Identify partners for collaboration 

Enhance the image of your program 

Retain or increase funding 

Provide direction for program staff 

Identify training and technical assistance 

needs 

Document the level of success in 

achieving objectives 

Identify areas of the program that need 

improvement 

Mobilize community support 

Redistribute or expand the locations 

where the intervention is carried out 

Focus program resources on a specific 

population 

45 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 6: Ensuring Use and Sharing Lessons Learned

WHO WILL USE THE EVALUATION RESULTS? 

Many different people and organizations will be interested in your evaluation findings, including: 

Partners – Remember your work in step 1 to identify those people or organizations that 

are invested in your Tribal Traffic Safety Program or that are interested in the results of the 

evaluation. Although each Tribal Traffic Safety Program is unique, some common partners might 

include, but are not limited to: tribal leaders, schools, head start, day care centers, Community 

Health Representatives (CHR), WIC, transportation departments, law enforcement, I/T/U health 

facilities, EMTs, environmental health, community members, BIA, IHS, and state and county 

partners. 

Funders – Most funding agencies want to know how their investment in your program is going. 

They might also use your evaluation findings to assess accountability, improve their grantmaking, 

disseminate successful programs to other grantees, increase the state of knowledge, assess 

the quality or impact of grants, and plan new programs. 

You – In addition to these important partners, you and your Tribal Traffic Safety Program 


team should use your evaluation findings to celebrate successes, troubleshoot challenges or 


unintended outcomes, and continuously improve the quality of your program.
 

CONTINUOUS QUALITY IMPROVEMENT (CQI) 

Continuous Quality Improvement (CQI) is the process of using your process and outcome 

evaluation data to determine what worked well, where there is room for improvement, and 

what changes may be needed. CQI takes advantage of what you have learned over time 

from your evaluation to improve the program for the future without starting over from the 

beginning. It helps all staff involved to keep your program fresh and a good fit for your 

participants and your community. CQI should be performed throughout the life of your 

program to assess what went well and what should be improved. You will continually use 

the results from your evaluation to determine whether you met your goal and desired 

short-, intermediate-, and long-term outcomes. The results will prepare you to decide 

whether and how to revise your goals and desired outcomes, reassess fit and capacity, 

and revise your work plan. The process then repeats itself over time to assess whether the 

changes you make prove to be effective. 

46 

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 6: Ensuring Use and Sharing Lessons Learned

 

DEVELOPING AN EVALUATION REPORT: 

Your evaluation report should clearly, succinctly, and impartially communicate all parts of the 

evaluation. The report should be written so that it is easy to understand by your audience. It need 

not be lengthy or technical. 

Common Evaluation Report Outline: 

Executive Summary a short summary of the 

report that can be shared separately or as 

part of the full evaluation report 

Background and Purpose 

• Program background

• Program description

• Key evaluation questions/focus

Evaluation Methods 

• Design

• Sampling procedures

• Measures or indicators

• Data collection procedures

• Analysis

• Limitations

Results 

Discussion and Recommendations 

TIPS FOR WRITING YOUR EVALUATION REPORT 

Tailor the report to your audience; you may 

need a different version of your report for 

each segment of your audience. 

Present clear and succinct results. 

Include a summary. 

Describe essential features of the program 

(e.g., logic models). 

Explain the focus of the evaluation and its 

limitations. 

Summarize the evaluation plan and 

procedures. 

List the strengths and weaknesses of the 

evaluation. 

Organize the report logically and include 

appropriate details. 

Discuss recommendations for action with 

their advantages, disadvantages, and 

resource implications. 

Verify that the report is unbiased and 

accurate. 

Remove technical jargon. 

Use examples, illustrations, graphics, and 

stories. 

Prepare and distribute reports on time. 

Distribute reports to as many partners as 

possible 

Acknowledge partners and participants. 

47 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

STEP 6: Ensuring Use and Sharing Lessons Learned

DISSEMINATION: SHARING RESULTS AND LESSONS LEARNED FROM EVALUATION 

Dissemination involves communicating evaluation findings or lessons learned to relevant audiences 

in a timely, unbiased, and consistent manner. Disseminating the evaluation findings for your Tribal 

Traffic Safety Program is a critical step in the overall process that is sometimes overlooked. To avoid 

making this common mistake, work with your team to create a dissemination plan that maps the 

following essential details: 

Who is the key audience? 

What approaches and formats will be used to disseminate findings—hard copy print, electronic, 

oral presentations, briefings, mailings, etc.? 

How, where, and when will findings be used? 

Who is responsible for dissemination? 

What resources are available to accomplish the work? 

What are the follow-up activities to ensure that the evaluation findings are used? 

How will follow-up activities be monitored? 

SOME METHODS FOR GETTING YOUR EVALUATION INFORMATION 

TO YOUR AUDIENCE INCLUDE: 

ORAL PRESENTATIONS – 

Tribal councils, partners 

meetings, community 

groups, etc. 

MAILINGS 

WEB SITES 

COMMUNITY FORUMS 


MEDIA – 


television, radio, newspaper 


PERSONAL CONTACTS 


EMAIL LISTSERVS 


ORGANIZATIONAL 

NEWSLETTERS 

48 

•

•

•

•

•

•

•
49 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix A

APPENDIX A: STANDARDS FOR EFFECTIVE EVALUATION
 

The steps in the CDC Evaluation Framework used throughout this guide are informed by a set 

of standards for evaluation. The standards provide practical guidelines to follow when having to 

decide among evaluation options at each step in the framework. The steps and standards are used 

together throughout the evaluation process. For each step, a subset of relevant standards should be 

considered. Furthermore, the standards can be applied while planning an evaluation and throughout 

its implementation. 

The four categories discussed below include a total of 30 specific standards. 

STANDARD 1: UTILITY 

Utility standards ensure that information needs of evaluation users are satisfied. Seven utility 

standards address such items as identifying those who will be impacted by the evaluation, the 

amount and type of information collected, the values used in interpreting evaluation findings, and the 

clarity and timeliness of evaluation reports. The following utility standards ensure that an evaluation 

will serve the information needs of intended users: 

•	 Partner identification. Persons involved in or affected by the evaluation should be identified so 


that their needs can be addressed.
 

•	 Evaluator credibility. The persons conducting the evaluation should be trustworthy and

competent in performing the evaluation for findings to gain maximum credibility and acceptance.

•	 Information scope and selection. Information collected should address pertinent questions 


regarding the program and be responsive to the needs and interests of clients and other 


specified partners.
 

•	 Values identification. The perspectives, procedures, and rationale used to interpret the findings

should be carefully described so that the bases for value judgments are clear.

•	 Report clarity. Evaluation reports should clearly describe the program being evaluated, including

its context and the purposes, procedures, and findings of the evaluation so that essential

information is provided and easily understood.

•	 Report timeliness and dissemination. Substantial interim findings and evaluation reports should

be disseminated to intended users so that they can be used in a timely fashion.

•	 Evaluation impact. Evaluations should be planned, conducted, and reported in ways that 


encourage follow-through by partners to increase the likelihood of the evaluation being used.
 
50 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix A

STANDARD 2: FEASIBILITY
 

Feasibility standards ensure that the evaluation is viable and pragmatic. The three feasibility 

standards emphasize that the evaluation should employ practical, nondisruptive procedures; that the 

differing political interests of those involved should be anticipated and acknowledged; and that the 

use of resources in conducting the evaluation should be prudent and produce valuable findings. The 

following feasibility standards ensure that an evaluation will be realistic, prudent, diplomatic, and 

frugal: 

•	 Practical procedures. Evaluation procedures should be practical so that you can keep disruption

of the program to a minimum while still being able to obtain needed information.

•	 Political viability. During planning and while the evaluation is being conducted, consideration 


should be given to the varied positions of interest groups so that their cooperation can be 


obtained and possible attempts by any group to curtail evaluation operations or to bias or 


misapply the results can be averted or counteracted.
 

•	 Cost-effectiveness. The evaluation should be efficient and produce valuable information to justify

expended resources.

STANDARD 3: PROPRIETY 

Propriety standards ensure that the evaluation is ethical (i.e., conducted with regard for the rights 

and interests of those involved and affected). Eight propriety standards address such items as 

developing protocols and other agreements for guiding the evaluation; protecting the welfare 

of human subjects; weighing and disclosing findings in a complete and balanced fashion; and 

addressing any conflicts of interest in an open and fair manner. The following propriety standards 

ensure that an evaluation will be conducted legally, ethically, and with regard for the welfare of those 

involved in the evaluation as well as those affected by its results: 

•	 Service orientation. The evaluation should be designed to assist organizations in addressing 


and serving effectively the needs of the participants.
 

•	 Formal agreements. All principal parties involved in an evaluation should agree in writing to their

obligations (i.e., what is to be done, how, by whom, and when) so that each must adhere to the

conditions of the agreement or renegotiate it.

•	 Rights of human subjects. The evaluation should be designed and conducted in a manner that 


respects and protects the rights and welfare of human subjects.
 

•	 Human interactions. Evaluators should interact respectfully with other persons associated with 


an evaluation, so that participants are not threatened or harmed.
 

•	 Complete and fair assessment. The evaluation should be complete and fair in its examination 


and recording of strengths and weaknesses of the program so that strengths can be enhanced 


and problem areas addressed.
 
51 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix A

•	 Disclosure of findings. The principal parties to an evaluation should ensure that the full

evaluation findings with pertinent limitations are made accessible to the persons affected by the

evaluation and any others with expressed legal rights to receive the results.

•	 Conflict of interest. Conflict of interest should be handled openly and honestly so that the 


evaluation processes and results are not compromised.
 

•	 Fiscal responsibility. The evaluator’s allocation and expenditure of resources should be prudent,

ethically responsible, and follow sound accounting procedures.

STANDARD 4: ACCURACY 

Accuracy standards ensure that the evaluation produces findings that are considered correct. Twelve 

accuracy standards include such items as describing the program and its context; articulating in 

detail the purpose and methods of the evaluation; employing systematic procedures to gather valid 

and reliable information; applying appropriate qualitative or quantitative methods during analysis and 

synthesis; and producing impartial reports containing conclusions that are justified. The following 

accuracy standards ensure that an evaluation will convey technically adequate information regarding 

the determining features of merit of the program: 

•	 Program documentation. The program being evaluated should be documented clearly and 


accurately.
 

•	 Context analysis. The context in which the program exists should be examined in enough detail

to identify probable influences on the program.

•	 Described purposes and procedures. The purposes and procedures of the evaluation should be

monitored and described in enough detail to identify and assess them.

•	 Defensible information sources. Sources of information used in a program evaluation should be

described in enough detail to assess the adequacy of the information.

•	 Valid information. Information-gathering procedures should be developed and implemented to 


ensure a valid interpretation for the intended use.
 

•	 Reliable information. Information-gathering procedures should be developed and implemented to

ensure sufficiently reliable information for the intended use.

•	 Systematic information. Information collected, processed, and reported in an evaluation should

be systematically reviewed and any errors corrected.

•	 Analysis of quantitative information. Quantitative information should be analyzed appropriately

and systematically so that evaluation questions are answered effectively.

•	 Analysis of qualitative information. Qualitative information should be analyzed appropriately and

systematically to answer evaluation questions effectively.

•	 Justified conclusions. Conclusions reached should be explicitly justified for partners’ 


assessment.
 
52 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix A

•	 Impartial reporting. Reporting procedures should guard against the distortion caused by

personal feelings and biases of any party involved in the evaluation to reflect the findings fairly.

•	 Meta-evaluation. The evaluation should be formatively and summatively evaluated against these

and other pertinent standards to guide its conduct appropriately and, on completion, to enable

close examination of its strengths and weaknesses by partners.

Source: Joint Committee on Standards for Educational Evaluation. Program evaluation standards: how 

to assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994. 
53 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix B

APPENDIX B - ADDITIONAL RESOURCES 

Selected Web-based Resources on Program Evaluation
 

Centers for Disease Control and Prevention: www.cdc.gov/eval/
 

Community Tool Box, University of Kansas: ctb.ku.edu/
 

RAND Corporation – Getting to Outcomes: 


www.rand.org/pubs/tools/TL259/introduction.html 

University of Wisconsin Cooperative Extension: Evaluation Resources: 

www.uwex.edu/ces/pdande/ 

W.K. Kellogg Foundation: 

www.wkkf.org/resource-directory/resource/2017/11/wk-kellogg-foundation-step-by-step-guideto-evaluation 

Selected Publications on Program Evaluation 

Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New approaches to evaluating community initiatives. 

New York, NY: Aspen Institute, 1995. 

Fawcett SB, Paine-Andrews A, Francisco VT, Schulz J, Ritchter KP, et al. Evaluating community 

initiatives for health and development. In: Rootman I, Goodstadt M, Hyndman B, et al., eds. 

Evaluating Health Promotion Approaches. Copenhagen, Denmark: World Health Organization (Euro), 

1999. 

Fawcett SB, Sterling TD, Paine Andrews A, Harris KJ, Francisco VT, et al. Evaluating community 

efforts to prevent cardiovascular diseases. Atlanta, GA: Centers for Disease Control and Prevention, 

National Center for Chronic Disease Prevention and Health Promotion, 1995. 

Fetterman DM, Kaftarian SJ, Wandersman A. Empowerment evaluation: Knowledge and tools for selfassessment and accountability. Thousand Oaks, CA: Sage Publications, 1996. 
54 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix B

Fetterman DM, Kaftarian SJ, Wandersman A. Empowerment evaluation principles in practice. New 

York: Guilford, 2005. 

Joint Committee on Standards for Educational Evaluation. Program evaluation standards: how to 

assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994. 

Patton MQ. Utilization-focused evaluation. Thousand Oaks, CA: Sage Publications, 1997. 

Rossi PH, Freeman HE, Lipsey MW. Evaluation: A systematic approach. Newbury Park, CA: Sage 

Publications, 1999. 
55 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix Glossary

APPENDIX GLOSSARY 

Accountability: The responsibility of program managers and staff to provide evidence to partners 

and funding agencies that a program is effective and in conformance with its coverage, service, 

legal, and fiscal requirements. 

Accuracy: The extent to which an evaluation is truthful or valid in what it says about a program, 

project, or material. 

Activities: The actual events or actions that take place as a part of the program using inputs, such 

as funds, technical assistance and other types of resources to produce specific outputs. 

Baseline Data: The initial information collected about the condition or performance of subjects 

prior to the implementation of an intervention, against which progress can be compared at strategic 

points during and at completion of the program. 

Bias: A point of view that inhibits objectivity. 

Coding: To translate a given set of data or items into descriptive or analytic categories to be used 

for data labeling and retrieval. 

Comparison group: A group not exposed to a program or treatment. Also referred to as a control 

group. 

Comprehensiveness: Full breadth and depth of coverage on the evaluation issues of interest. 

Continuous Quality Improvement (CQI): An ongoing process that evaluates how a program works 

and ways to improve its processes. 

Control group: In quasi-experimental designs, a group of subjects who receive all influences 

except the program in exactly the same fashion as the treatment group (the latter called, in some 

circumstances, the experimental or program group). Also referred to as a non-program group. 

Data Material: Gathered during the course of an evaluation which serves as the basis for 

information, discussion, and inference. 

Data collection method: The way facts about a program and its outcomes are amassed. Data 

collection methods often used in program evaluations include literature search, file review, natural 

observations, surveys, expert opinion, and case studies. 

Descriptive statistical analysis: Numbers and tabulations used to summarize and present 

quantitative information concisely. 

Dissemination: The communication of the actions--by written, oral, and/or audio-visual reporting--of 

evaluators to foster knowledge of the evaluation findings among all right-to-know audiences. 
56 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix Glossary

Evaluation design: The logical model or conceptual framework used to arrive at conclusions about 

outcomes. 

Evaluation plan: A written document describing the overall approach or design that will be used to 

guide an evaluation. It includes what will be done, how it will be done, who will do it, when it will be 

done, why the evaluation is being conducted, and how the findings will likely be used. 

Executive summary: A nontechnical summary statement designed to provide a quick overview of the 

full-length report on which it is based. 

Focus group: A group of people selected for their relevance to an evaluation that is engaged by a 

trained facilitator in a series of discussions designed for sharing insights, ideas, and observations 

on a topic of concern. 

Indicator: A specific, observable, and measurable characteristic or change that shows the progress 

a program is making toward achieving a specified outcome. 

Inputs: Resources that go into a program in order to perform the activities successfully. 

Interviews: Guided conversations between a skilled interviewer and an interviewee that seek to 

maximize opportunities for the expression of a respondent’s feelings and ideas through the use of 

open-ended questions and a loosely structured interview guide. 

Instrument: Device that assists evaluators in collecting data in an organized fashion, such as a 

standardized survey or interview protocol; a data collection device. 

Key informant: Person with background, knowledge, or special skills relevant to topics examined by 

the evaluation; sometimes an informal leader or spokesperson in the population of focus. 

Logic model: A systematic and visual way to present the perceived relationships among the 

resources you have to operate the program, the activities you plan to do, and the changes or results 

you hope to achieve. 

Mixed-method evaluation: An evaluation design that includes the use of both quantitative and 

qualitative methods for data collection and data analysis. 

Natural observation: A data collection method that involves on-site visits to locations where a 

program is operating. It directly assesses the setting of a program, its activities, and individuals who 

participate in the activities. 

Objective: A specific, measurable description of an intended outcome. 

Objectivity: The expectation that data collection, analysis and interpretation will adhere to standards 

that eliminate or minimize bias; objectivity insures that outcome or evaluation results are free from 

the influence of personal preferences or loyalties. 
57 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix Glossary

Outcome evaluation: The systematic collection of information to assess the impact of a program, 

present conclusions about the merit or worth of a program, and make recommendations about 

future program direction or improvement. 

Outcomes: Changes in attitudes, values, behaviors or conditions between baseline measurement 

and subsequent points of measurement. Changes can be immediate, intermediate or long-term; the 

results/effects expected by implementing the program’s strategies. 

Outputs: The direct products of program activities; immediate measures of what the program did. 

Partners: People or organizations who have a direct or indirect interest in the development 

intervention or its evaluation and/or what will be done with results of the evaluation. 

Population: All the persons in a particular group. 

Pre/post tests: Standardized methods used to assess change in subjects’ knowledge and capacity 

to apply this knowledge to new situations. The tests are administered prior to implementation of the 

program and after completion of the program’s activities. Determines performance prior to and after 

the delivery of an activity or strategy. 

Primary data: Data collected by an evaluation team specifically for the evaluation study. 

Process evaluation: The systematic collection of information to document and assess how a 

program was implemented and operates. 

Program evaluation: The systematic collection of information about the activities, characteristics, 

and outcomes of programs to make judgments about the program, improve program effectiveness, 

and/or inform decisions about future program development. 

Program goal: A statement of the overall mission or purpose(s) of the program. 

Qualitative data: Non-numerical data rich in detail and description that are usually presented in a 

textual or narrative format, such as data from case studies, focus groups, interviews or document 

reviews. Used with open-ended questions, these data have the ability to illuminate evaluation 

findings derived from quantitative methods. 

Quantitative data: Numeric information, focusing on things that can be counted, scored and 

categorized; used with close-ended questions, where participants have a limited set of possible 

answers to a question. Quantitative data analysis utilizes statistical methods. 

Questionnaire: Highly structured series of written questions that is administered to a large number 

of people; questions have a limited set of possible responses. 

Random assignment: A process by which the people in a sample to be tested are chosen at random 

from a larger population; a pool of eligible evaluation participants are selected on a random basis. 
58 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Appendix Glossary

Reliability: The extent to which a measurement, when repeatedly applied to a given situation 

consistently produces the same results if the situation does not change between the applications. 

Reliability can refer to the stability of the measurement over time or to the consistency of the 

measurement from place to place. 

Representative sample: A segment or group taken from a larger body or population that mirrors in 

composition the characteristics of the larger body or population. 

Resources: Assets available and anticipated for operations. They include people, equipment, 

facilities, and other things used to plan, implement, and evaluate programs. 

Sample: A segment of a larger body or population. 

Sample size: The number of units to be sampled. 

Secondary data: Data collected and recorded by another (usually earlier) person or organization, 

usually for different purposes than the current evaluation. 

Standard: A principle commonly agreed upon by experts in the conduct and use of an evaluation for 

the measure of the value or quality of an evaluation (e.g., accuracy, feasibility, propriety, utility). 

Statistical analysis: The manipulation of numerical or categorical data to predict phenomena, to 

draw conclusions about relationships among variables or to generalize results. 

Subjectivity: Subjectivity exists when the phenomena of interest are described or interpreted in 

personal terms related to one’s attitudes, beliefs or opinions. 

Surveys: A data collection method that involves a planned effort to collect needed data from a 

sample (or a complete census) of the relevant population. Uses structured questions from specially 

designed instruments to collect data about the feelings, attitudes, knowledge, and/or behaviors 

of individuals. The relevant population consists of people or entities affected by the program (or of 

similar people or entities). 

Triangulation: Using multiple data sources independent of one another to help verify the validity of 

evaluation findings so results are not informed (and possibly skewed) by only one or two sources. 

Having more sources helps ensure findings are replicable and unbiased. 

Validity: The degree of accuracy of a measurement. For survey instruments, validity refers to the 

degree to which the instrument measures what it is intended to measure. 
59 Tribal Motor Vehicle Injury Prevention Program Evaluation Guide

Acknowledgements

ACKNOWLEDGEMENTS 

The contents of this guide were developed and adapted from several existing sources including: 

Centers for Disease Control and Prevention 

CDC Program Performance and Evaluation Office 

Framework for Program Evaluation in Public Health 

Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide 

Tribal Motor Vehicle Injury Prevention Program Best Practices Guide 

Community Tool Box, University of Kansas 

Joint Committee on Standards for Educational Evaluation 

SUGGESTED CITATION: 

Albuquerque Area Southwest Tribal Epidemiology Center (2022). Tribal Motor Vehicle Injury 

Prevention: Program Evaluation Guide. Albuquerque, NM. 

This guide was developed by the Albuquerque Area Southwest Tribal Epidemiology Center via Indian 

Health Service Cooperative Agreement Number U1B1IHS0013-12-01, with support from the Centers 

for Disease Control and Prevention. The content is solely the responsibility of the authors and does 

not necessarily represent the official views of the Indian Health Service or the Centers for Disease 

Control and Prevention. 

For additional information about the guide please contact the Albuquerque Area Southwest Tribal 

Epidemiology Center (AASTEC) at aastecinfo@aaihb.org