Gather Credible

Evidence
STEP 1: ENGAGE STAKEHOLDERS

1.1 Determine how and to what extent to involve stakeholders in

program evaluation

STEP 2: DESCRIBE THE PROGRAM

2.1 Understand your program focus and priority areas

2.2 Develop your program goals and measurable (SMART) objectives

2.3 Identify the elements of your program and get familiar with 

logic models

2.4 Develop logic models to link program activities with outcomes

STEP 3: FOCUS THE EVALUATION

3.1 Tailor the evaluation to your program and stakeholders’ needs

3.2 Determine resources and personnel available for your evaluation

3.3 Develop and prioritize evaluation questions

u STEP 4: GATHER CREDIBLE EVIDENCE

4.1 Choose appropriate and reliable indicators to answer your

evaluation questions

4.2 Determine the data sources and methods to measure indicators

4.3 Establish a clear procedure to collect evaluation information

4.4 Complete an evaluation plan based on program description 

and evaluation design

STEP 5: JUSTIFY CONCLUSIONS

5.1 Analyze the evaluation data

5.2 Determine what the evaluation findings “say” about your program

STEP 6: ENSURE USE OF EVALUATION FINDINGS 

AND SHARE LESSONS LEARNED

6.1 Share with stakeholders the results and lessons learned from 

the evaluation

6.2 Use evaluation findings to modify, strengthen, and improve 

your program

Practical Use of Program Evaluation among

Sexually Transmitted Disease (STD) Programs

SUGGESTED CITATION: 

Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program

Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA):

Centers for Disease Control and Prevention; 2007.
T

he information you gather in your evaluation must be

reliable and relevant for your program and stakeholders.

Gathering credible evidence means that the data you collect

answer the evaluation questions you developed. Step 4 will

help you gather credible evidence to strengthen your evaluation

findings and the recommendations that follow to improve the

program component or activity you are evaluating. 

In Step 4 you will: 

• Choose appropriate and reliable indicators for your evaluation

questions (Tool 4.1).

• Determine data sources and methods for indicators (Tool 4.2).

• Establish a procedure to collect the information (Tool 4.3).

• Complete an evaluation plan based on your program description

and evaluation design (Tool 4.4).

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 173

Gather Credible

Evidence
TOOL 4.1: CHOOSE APPROPRIATE AND 

RELIABLE INDICATORS TO ANSWER YOUR

EVALUATION QUESTIONS

_____________________________________

INTRODUCTION

In previous tools, you practiced formulating goals and writing

Specific, Measurable, Achievable, Relevant, and Time-bound

(SMART) objectives, and developing a logic model to visually

describe how your program or program component works. You

learned to distinguish between short-term, intermediate, and longterm outcomes, and to develop and prioritize your evaluation

questions based on your program and stakeholders’ needs and

resources available. Also, you learned to articulate the purpose of

your evaluation and to identify its uses and users. 

The next step in your evaluation is to select the measures (i.e.,

indicators) you will use to determine the progress your program or

activity is making. The flowchart below illustrates where indicators

fit into your evaluation planning activities.

174

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS

PROGRAM GOALS

SMART

OUTCOME OBJECTIVES

SMART

PROCESS OBJECTIVES

ENGAGE

STAKEHOLDERS

THROUGHOUT

EVALUATION

EVALUATION PURPOSE, USERS, USES

EVALUATION QUESTIONS

EVALUATION DESIGN

PROCESS OUTCOME INDICATORS

LOGIC MODEL

INPUTS ACTIVITIES OUTPUTS

PROCESS

SHORT-TERM

OUTCOMES

INTERMEDIATE

OUTCOMES

LONG-TERM

OUTCOMES

OUTCOME
LEARNING OBJECTIVES

After completing this tool, you should be able to:

• Develop indicators that you will use to measure the outcomes

and process aspects pertaining to your evaluation questions.

WHAT IS AN INDICATOR?

An indicator is a specific, observable, and measurable

accomplishment or change that shows the progress made toward

achieving a specific output or outcome in your logic model.

Indicators are markers of accomplishment/progress. Some commonly

used indicators include participation rates, attitudes, individual

behaviors, community norms, policies, health status, and incidence

and prevalence. The indicators you select should answer your

evaluation questions and help you determine whether your program

objectives have been achieved.

WHAT ARE THE KEY ELEMENTS OF AN INDICATOR?

The indicators you select must be specific, observable and

measurable. This means that they can be seen (e.g., observed

behavior), heard (e.g. participant responses), or read (i.e., agency

records).

• An indicator that is specific provides a clear description of what

you want to measure. This is very similar to the “Specific”

criterion of SMART objectives discussed in tool 2.2. For

example, it is more specific to say “in-school adolescents aged 13-

18 who test positive for Chlamydia,” rather than “youth who

have an STD”. With the latter example, what is meant by the

terms “youth” and “STD” is unclear.

• An indicator that is observable focuses on an action or change.

For example, “the proportion of school clinic staff who can

identify the risk factors for Chlamydia” will be more difficult to

verify than the “proportion of school-clinic staff who can list two

risk factors for Chlamydia.” Remember to use verbs that relate to

observation of progress, such as “list,” “demonstrate,” “select,”

“show,” or “report.”

• An indicator that is measurable quantifies change and is usually

reported in numerical terms (count, percentage/proportion, or

ratio), which is very similar to the “Measurable” criterion of

SMART objectives discussed in tool 2.2. 

175

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
WHAT IS THE LINK BETWEEN INDICATORS 

AND PERFORMANCE MEASURES?

CDC’s Division of STD Prevention (DSTDP) has developed a set of

indicators that are referred to as performance measures, and each

project area receiving CDC funds is required to report on the

indicators that apply to them. 

These indicators can be linked to the outputs component of the

program logic model(s) that you develop for your overall program

(e.g., Comprehensive STD Prevention System-CSPS, Infertility

Prevention Program-IPP, and Syphilis Elimination Effort-SEE) and

can give you a notion of “where things may stand” and how to

improve performance. In Table 1, five of these measures are linked to

certain outputs in the generic CSPS program logic model presented

in Tool 2.4. 

176

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Example of an indicator:

The proportion of gonorrhea cases among women 14-49 years of age

interviewed within 7 days from the date of specimen collection.

• Specific: specifies the STD and the target population (gonorrhea

cases among women 14-49 years of age)

•Observable: indicates the action/change (interviews conducted

within 7 days from the date of specimen collection)

• Measurable: quantifies change which will be reported as a

“proportion of cases” where the numerator will be the total

number of gonorrhea cases among women 14-49 years of age

interviewed within 7 days from the date of specimen collection

and the denominator will be the total number of diagnosed and

reported gonorrhea cases among women 14-49 years of age.
Please note that even though performance measures are indicators,

they only partially determine what is going on in a program.

Therefore, collecting data on performance measures alone does not

constitute evaluation. To have a complete picture of what is

happening in your program and answer the ‘whys’ and ‘hows’, you

will need to conduct evaluation. Table 2 shows how a performance

measure (indicator) is combined with other process indicators to

measure outputs in an evaluation of County Z’s Chlamydia (Ct)

screening initiative in juvenile detention centers. 

177

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Medical and Lab Services

• Female admittees in

juvenile detention facilities

tested for chlamydia.

Partner Services

• Syphilis cases’ partners

identified.

PROCESS OUTPUTS 

• Proportion of female admittees to large juvenile

detention facilities who were tested for Chlamydia.

• Proportion of primary and secondary (P&S) syphilis

cases interviewed within 7, 14, and 30 calendar

days from the date of specimen collection.

• Number of associates and suspects tested, per case

of P&S syphilis.

• Number of associates and suspects treated for

newly diagnosed syphilis, per case of P&S syphilis.

• Proportion of ‘priority’ gonorrhea cases interviewed

within 7, 14, and 30 days from the date of

specimen collection.

LINKED DSTDP PERFORMANCE 

MEASURES (INDICATORS)

Table 1: Examples of CSPS Process Outputs Linked with

DSTDP Performance Measures (Indicators)

1

Juvenile detention facilities having booked 500 or more adolescent females annually For the

purpose of DSTDP reporting, programs with greater than 500 admitees booked must report

on all facilities, those with less than 500 may select one or more facilities to report on.

2

Priority population(s) is to be locally determined (e.g., pregnant women, women aged 15-19

years, women of child-bearing age, resistant gonorrhea, MSM, etc.).
HOW DO YOU DEVELOP APPROPRIATE INDICATORS? 

The following are some steps to guide you in developing indicators.

1. Involve your program’s stakeholders in indicator

development. 

In Tool 1.1 you identified relevant stakeholders to involve in

various stages of your evaluation. Bring these individuals together

to identify and develop indicators, using the steps outlined

below. In this way, they will identify indicators that are

meaningful and useful to them. This will help ensure the

credibility of, and buy-in for, your evaluation findings.

2.Review your evaluation questions and use your logic

model as a template for developing indicators.

If you want your evaluation questions (see Tool 3.3) to answer

whether your program is functioning as planned, develop

process/performance indicators that are logically related to your

logic model outputs (see Tools 2.3 and 2.4). If you want your

evaluation to answer whether changes in the target population

occurred as a result of your program activities, develop outcome

indicators linked with your logic model outcomes. The following

matrix can serve as a template for developing indicators based on

the identified evaluation questions. 

178

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

As a result of the Ct

screening initiative, were

adolescent females in

juvenile detention center

screened, counseled and

treated for Ct?

Female adolescents

screened, counseled

and treated for Ct.

PROCESS EVALUATION

QUESTION

PROCESS OUTPUTS

• Proportion of female admittees to

large juvenile detention centers

who were tested for Chlamydia

(CDC performance measure).

• Proportion of female admittees in

large juvenile detention centers

who received STD prevention

counseling.

• Proportion of female admittees to

large juvenile detention centers

who were treated presumptively

• Proportion of female admittees to

large juvenile detention centers

who were treated as a result of a

positive Ct test result.

PROCESS INDICATORS

Table 2: Examples of Indicators and Use of a CDC Performance Measure for

Evaluation of a Ct Screening Initiative 
3.When appropriate, use CDC performance measures as

process indicators linked with your logic model outputs. 

While you are required to report on relevant CDC performance

measures for your funded program, these measures can be used as

process indicators in your evaluation of a specific program

component or activity if applicable. 

4.Review your indicators to ensure they are specific,

observable, and measurable. 

Be sure your indicators are focused and clearly written, and that

they measure progress related with the program component or

activity you are interested in. 

5. Include an indicator related with data of previous

year/months/weeks, etc. for inputs and outcomes

intended to achieve change (e.g., increased adherence of

clinical staff to STD guidelines, decreased prevalence of

Chlamydia among adolescent females in juvenile detention

facilities). It is important to have baseline data when a program

wants to measure change. If this is not included, change cannot

be quantified/measured.

179

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

PROCESS EVALUATION

QUESTIONS

PROCESS OUTPUTS PROCESS INDICATORS

SHORT-TERM OUTCOME

EVALUATION QUESTIONS

SHORT-TERM

OUTCOMES

SHORT-TERM INDICATORS
6. Determine whether the indicators:

• Provide useful information that can measure your outcomes

or process aspects and answer your evaluation questions. 

• Are feasible in terms of whether the data are available, can be

made available, or can be collected in a timely manner with

the resources you have.

• Are adequate to capture the measure you need. Depending on

how each output/outcome is phrased, you may need to

develop more than one indicator to describe completely what

is being measured by that output or process aspect and/or

outcome. Remember that while multiple indicators are

sometimes needed for tracking the implementation and effects

of a program, defining too many indicators can detract from

the evaluation’s goals. 

180

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

c Involve your stakeholders in the development of and

selection of indicators.

c Review your evaluation questions and use your logic

model as a template for developing indicators. The

indicators should relate to the outputs and the shortterm, intermediate, and long-term outcomes of your

logic model.

c When appropriate, use CDC performance measures as

indicators linked with your logic model outputs.

c Review your indicators to ensure they are specific,

observable, and measurable.

c Include indicators related with data of previous

year/months/weeks, etc. for inputs and outcomes

intended to achieve change.

c Ask yourself if the indicators:

• Provide useful information.

• Are feasible.

• Are adequate in number.

SUMMARY CHECKLIST: Selecting Indicators
CONCLUSION AND NEXT STEPS

Gathering credible evidence is essential for a good evaluation. 

As you plan and conduct your evaluation, strive to collect information

that is relevant and convincing. In this tool you learned how indicators

can help you measure progress toward your program goals and objectives,

and how to develop them (specific, observable, and measurable). 

Indicators are used as the starting point for designing data collection and

reporting strategies. The next tool (4.2) will help you determine which

data sources and methods you will use to gather information for

measuring your indicators.

ACRONYMS USED IN THIS TOOL

CDC – Centers for Disease Control

CSPS – Comprehensive STD Prevention System

Ct – Chlamydia trachomatis

DSTDP – Division of STD Prevention

IPP – Infertility Prevention Program

MSM – Men who have sex with men

SEE – Syphilis Elimination Effort

SMART – Specific, Measurable, Achievable, Relevant, and Time-bound

STD – Sexually Transmitted Disease

KEY TERMS

Indicator: A specific, observable, and measurable accomplishment or

change that shows whether progress has been made toward achieving a

specific program output or outcome. 

Outcome indicators: These measure whether progress was made toward

achieving your short-term, intermediate, or long-term outcomes.

Performance measures: A set of indicators developed by CDC’s

Division of STD Prevention with input from members of NCSD, state

representatives of NCSD member grantees, and seven project areas where

the measures were pilot-tested. Each project area receiving CDC funds is

required to report on the measures (indicators) that apply to them. 

Process indicators: Indicators that measure whether progress is made

toward achieving implementation fidelity by your program. These

indicators measure whether your program is functioning as planned, and

relate to the outputs in your program logic model.

181

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
EXERCISE: 

DEVELOPING INDICATORS

Listed below are the evaluation questions and the logic model outputs

and short-term outcomes that you and your stakeholders want to

address in a process and outcome evaluation of a syphilis media

campaign targeting men who have sex with men (MSM) in City Y. 

Process evaluation questions:

• Were the campaign activities implemented as planned?

–Were press releases written and sent to media outlets?

–Were posters created and distributed to MSM business

venues?

–Were educational pamphlets developed and distributed 

to MSM?

Process Outputs:

• Press releases written and sent to media outlets

• Posters created and distributed to MSM business venues

• Educational pamphlets developed and distributed to MSM

Short-term Outcome Evaluation Questions:

• As a result of the syphilis media campaign, were MSM reached?

• As a result of the campaign, did MSM become aware of the

syphilis outbreak? 

• As a result of the campaign, did knowledge of syphilis prevention

increase among MSM?

Short-term Outcomes:

• MSM reached with prevention messages

• Awareness of syphilis outbreak among MSM as a result of the

campaign

• Knowledge of syphilis prevention among MSM as a result of the

campaign

Review your evaluation questions and corresponding logic model

components to develop appropriate indicators. Develop at least one

indicator per process output or short-term outcome. Your developed

indicators need to be relevant to the STD program goals and

objectives of the syphilis elimination campaign. Determine if the

developed indicators:

182

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
• Are specific, observable and measurable;

• Provide useful information that can measure your process and

outcomes aspects and answer your evaluation questions;

• Are feasible in terms of whether data is available or can be

collected in a timely fashion with the resources you have

available; and

• Are adequate to capture the measure you need. 

Put the information provided above in the corresponding columns in

the templates below. This will help you identify and develop the

appropriate indicators. The completed templates are presented at the

end of this tool. [Note: This exercise only includes short-term outcomes

since measuring longer term outcomes may require more time and

resources than are available in a typical four-year grant period. Tool 4.2

will discuss the sources and methods for collecting indicator

information.] 

183

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

PROCESS EVALUATION

QUESTIONS

PROCESS OUTPUTS PROCESS INDICATORS

SHORT-TERM OUTCOME

EVALUATION QUESTIONS

SHORT-TERM

OUTCOMES

SHORT-TERM INDICATORS
REFERENCES 

Centers for Disease Control and Prevention. (1999). Framework for

program evaluation in public health. MMWR Recommendations

and Reports, 48(RR-11). Retrieved October, 17, 2004, from

http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm

Centers for Disease Control and Prevention. (2002). Physical activity

evaluation handbook. Retrieved October 17, 2004, from

http://www.cdc.gov/nccdphp/dnpa/physical/handbook/index.htm

MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K.,

& Turner, K. (2001, November). Introduction to program

evaluation for comprehensive tobacco control programs. Atlanta, GA:

Centers for Disease Control and Prevention. Retrieved October

17, 2004, from http://www.cdc.gov/tobacco/evaluation_manual/

Evaluation.pdf

Taylor-Powell, E., Steele, S., & Doughlah, M. (1996, February).

Planning a program evaluation. Madison: University of WisconsinExtension, Division of Cooperative Extension. Retrieved October

17, 2004, from www.uwex.edu/ces/pdande/evaluation/

evaldocs.html 

184

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
ANSWER KEY FOR EXERCISE

185

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• Were press releases

written and sent to media

outlets?

• Were posters created and

distributed to MSM

business venues?

• Were educational

pamphlets developed and

distributed to MSM?

Press releases written

and sent to media

outlets

Posters created and

distributed to MSM

business venues

Educational

pamphlets developed

and distributed to

MSM

PROCESS EVALUATION

QUESTIONS

PROCESS OUTPUTS

# of press releases written

# of press releases sent to media outlets

# of media outlets receiving press releases

# of posters created 

# of posters distributed to MSM 

business venues

# of MSM business venues accepting 

posters

# of educational pamphlets designed

# of educational pamphlets distributed 

to MSM

# of MSM receiving educational pamphlets

PROCESS INDICATORS

Were the campaign activities implemented as planned?

As a result of the syphilis

media campaign, were MSM

reached?

As a result of the campaign,

did awareness of the syphilis

outbreak increase among

MSM?

As a result of the campaign,

did knowledge of syphilis

prevention increase among

MSM?

MSM reached with

prevention messages

Awareness of syphilis

outbreak among MSM

as a result of the

campaign

Knowledge of syphilis

prevention among

MSM as a result of

the campaign

SHORT-TERM OUTCOME

EVALUATION QUESTIONS

SHORT-TERM

OUTCOMES

% of MSM who recall seeing

information (i.e., via brochures, posters,

press releases developed for the

campaign) on syphilis prevention.

% of MSM who recall the main

message(s) of the campaign

corresponding to the syphilis outbreak

% of MSM who can describe at least

two methods of preventing syphilis

transmission.

SHORT-TERM INDICATORS
Please note that you should always use your logic model as a

template for developing the indicators. Remember that if you want

your evaluation questions to answer whether your program is

functioning as planned, you need to develop process or performance

indicators that are logically related to your logic model outputs. If

you want your evaluation to answer whether changes in the target

population occurred as a result of your program activities, you need

to develop outcome indicators linked to your logic model outcomes. 

Tool 4.2 will assist you in determining which data sources and

methods will be used to gather information on indicators.

186

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
TOOL 4.2: DETERMINE THE DATA SOURCES AND

METHODS TO MEASURE INDICATORS

_____________________________________

INTRODUCTION

In Step 3, you learned how to (1) focus your evaluation, (2) identify

the resources and personnel that are available for the evaluation, and

(3) develop and prioritize evaluation questions based on your

program’s and other stakeholders’ needs and resources. In Step 4, you

have begun to examine various processes for gathering credible

evidence. Tool 4.1 outlined strategies for choosing appropriate and

reliable indicators for your evaluation questions. Tool 4.2 will

provide you with strategies for determining data sources and

methods for measuring your indicators. The flowchart below

provides a description of where data sources and methods fit within

your evaluation activities.

187

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS

PROGRAM GOALS

SMART

OUTCOME OBJECTIVES

SMART

PROCESS OBJECTIVES

ENGAGE

STAKEHOLDERS

THROUGHOUT

EVALUATION

EVALUATION PURPOSE, USERS, USES

EVALUATION QUESTIONS

EVALUATION DESIGN

DATA SOURCES AND METHODS

PROCESS AND OUTCOME INDICATORS

LOGIC MODEL

INPUTS ACTIVITIES OUTPUTS

PROCESS

SHORT-TERM

OUTCOMES

INTERMEDIATE

OUTCOMES

LONG-TERM

OUTCOMES

OUTCOME
LEARNING OBJECTIVES

Upon completion of this tool, you will be able to:

• Determine data sources for each indicator.

• Identify data collection methods that are appropriate for

indicators and data sources.

WHAT DATA SOURCES COULD YOU USE? 

The first task in this tool is to decide from where/whom you will get

the data to measure each indicator and what you should consider

when selecting these data sources. When choosing data sources, make

sure the data from the chosen source will actually answer your

evaluation questions. For example, surveillance data can tell you how

many Ct cases you have, but not how many people you screened to

get that number of cases. Balance the need for data that can be

considered useful against data that is critical with budgetary

constraints. You should consider the following questions as you

consider your data sources:

• What do you need to know?

• When do you need the data?

• Are the data available from the program and from other sources?

• How often do you need to collect the data?

• What indicators require collection of new data?

• Will the data be compared with data from elsewhere? If so, will it

be similar to the other data?

• Are the data credible?

• How much money do you have to spend and how costly will it

be to collect the data from a prospective source?

Some of the data sources that you might consider using are presented

in Table 1.

188

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
189

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• Grant proposals

• Administrative records

• Registration/enrollment forms

• Surveillance reports

• Database records

• Web pages

• Minutes of meetings

• Brochures

• The data are

available and

accessible.

• You may know how

the data were

collected if gathered

by your program.

EXAMPLES ADVANTAGES

• The value of the data depends on how

accurately and consistently it was

recorded.

• Existing records may not have the data

collected in the format you need.

• Existing records may not contain all the

data that you need for your evaluation.

• Due to privacy considerations, the

program may not have permission from

clients to use the information in their

existing records for this purpose.

DISADVANTAGES

Table 1: Advantages and Disadvantages of Possible Data Sources

Data Source: Documents

• STD clients’ or former clients’

knowledge, attitudes, or skills

during and after an activity.

• Target population(s)

perception and/or acceptability

of a new modality (e.g.,

partners therapy)

• Changes in STD clients’

behavior

• These are data your

program directly

collects from the

target population(s)

(primary data).

• Reports from individuals and self-report

data may be unreliable. However, when a

combination of data sources is used (e.g.

individuals and data records), the

combined set of data can provide useful

information.

• When obtaining information directly from

individuals (e.g., face-to-face interviews)

be aware of the possibility of receiving

socially desirable responses. For example,

a client who has several sexual partners

may report in an interview that s/he only

has one sexual partner because s/he feels

that is the more acceptable response.

Data Source: Individuals

(continued)
WHAT DATA COLLECTION METHODS COULD YOU 

USE FOR YOUR INDICATORS ACCORDING TO YOUR

RESOURCES? 

Once you have decided where or from whom you want to obtain the

data for each of your indicators, you are ready to decide which data

collection methods to use. The data collection methods to be

discussed in this tool include: surveys, interviews, focus groups,

observation, and document review. Their advantages and

disadvantages are discussed in Table 2. You should consider the

following questions when choosing a data collection method. 

Purpose of the evaluation: What method seems most appropriate

for the purpose of your evaluation and the evaluation questions that

you want answered?

Users of the evaluation: Will the method allow you to gather

information that can be analyzed and presented in a way that will be

seen as credible by your stakeholders? 

190

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• Meetings

• Client encounters

• Data observed by you or

by a trained observer on

indicators related to

behavior, facilities, and

environments among

others.

• The data provide

information on

verbal/non-verbal

behavior and skills.

• The data can be

used to supplement

self-report

information.

EXAMPLES ADVANTAGES

• The value of the data depends

on the training and skills of the

observer, and the specificity of

the instrument used to rate the

observations.

• Ratings may vary if there is

more than one observer.

DISADVANTAGES

Table 1: Advantages and Disadvantages of Possible Data Sources (continued)

Data Source: Observations
Respondents from whom you will collect the data: Where and

how can respondents best be reached? What is the culturally and

linguistically appropriate method to use? Is conducting a personal

interview or a survey more appropriate for certain target populations? 

Do you need a representative sample, a convenient sample, or the

universe?

Resources available (time, money, volunteers, travel expenses,

supplies): Which method(s) can you afford and manage? What

resource allocation is feasible? Consider when results are needed,

your own abilities, costs of hiring a consultant, and other resource

issues.

Degree of intrusiveness: Will the method disrupt the program or

be seen as intrusive by the respondents? Also consider issues of

confidentiality if the information that you are seeking is sensitive.

Type of information: Do you want representative information that

applies to all participants (standardized information, such as from a

survey that will be comparable across locations)? Or, do you want to

examine the range and diversity of experiences, or tell a story about

your target population(s) or a program component (e.g., case study)?

Advantages and disadvantages of each method: What are the key

strengths and weaknesses of each? Consider issues such as time and

respondent burden, cost, necessary infrastructure, and access to

records. What is most appropriate for your evaluation needs?

Data collection methods you might consider using include surveys,

interviews (e.g. individual, focus groups), observations, and

document reviews. The advantages and disadvantages of these

methods are summarized in Table 2.

191

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
192

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Surveys

Interviews

(individual/

in-depth)

Focus Groups

Observation

Document

Review

• Anonymous completion

possible

• Can administer to groups of

people at the same time

• Can be efficient and cost

effective

• Can build rapport with

participant

• Can probe to get additional

information

• Can get breadth or depth of

information

• Can get common

impressions quickly

• Can be an efficient way to

get breadth and depth of

information in a short time

frame

• Can view program

operations as they occur

• Can document historical

information about your

program

• Does not interrupt program

or client routine

• Information already exists 

METHOD ADVANTAGES

• Forced choices may miss certain

responses from participants

• Wording may bias responses

• Impersonal

• Time consuming

• Expensive

• Interviewing styles and wording

may affect responses

• Need experienced facilitator 

• Can be difficult and costly to

schedule a group of 6-8 people

• Time consuming to analyze

responses

• Difficult to interpret observed

behaviors

• My influence behaviors of

program participants

• May be expensive and time

consuming to record each

individual event.

• May be time consuming

• Available information may be

incomplete

• Gathering information is

dependent on quality of records

kept

DISADVANTAGES

Table 2: Advantages and Disadvantages of Possible Data Sources
HOW DO YOU IMPLEMENT A MIXED APPROACH?

In reviewing which sources or method(s) to use, keep in mind that

you can use a mixed approach to answer evaluation questions and

measure indicators. This involves using more than one source (e.g.,

individuals and documents) and/or different methods (e.g., survey

and focus groups). With mixed methods, you can obtain different

sets of data that have the potential to produce similar results and

therefore add credibility to your findings. Mixed methods can also

allow you to examine different facets of the same phenomenon and

add breadth and depth to your findings. 

An example of a mixed method approach would be conducting both

a focus group and a survey with sexually active men and women aged

15-19 years to understand their sexual history. The survey would

provide numeric data on average number of sex partners, frequency

of condom use, and other information. The focus group could

provide in-depth information on these same variables, and detailed

data on “why” members of the target population have multiple sex

partners and their views on condom use. Generally, it is best to use

some combination of quantitative and qualitative methods to obtain

the most comprehensive information to measure your indicators. Be

aware that in some instances, the results you obtain from your mixed

methods instruments may seem to produce conflicting results. For

example, your survey of sexually active men and women aged 15-19

years, may indicate that they do use contraception while your focus

group indicates that condoms are not widely used. In this example,

the results from the survey would appear to contradict the results

from the focus group. In this situation (resources permitting)

conduct a focus group or interviews of surveyed participants. 

This may help you reconcile the original results of the focus group

and survey.

A valuable source of feedback for decisions on selecting data

collection methods is someone with experience using the various

methods you are considering. This person may be a staff member of

your program or from other programs at the Department of Health,

a consultant from a local college or university, your program

consultant or evaluation staff from CDC’s Division of STD

Prevention (DSTDP). Their input regarding the feasibility and cost

of these data collection methods can be invaluable to you and your

STD program as you choose the data collection method(s) for your

indicators.

193

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
WHAT IS THE RELATIONSHIPS AMONG EVALUATION

QUESTIONS, INDICATORS, AND, DATA SOURCES 

AND METHODS?

Following are two tables that examine the relationship among

evaluation questions, indicators, and data collection methods for an

evaluation of a professional development workshop on chlamydia

screening protocols. The information you obtained in previous

evaluation tools will complement the information presented in Tables

3 and 4. These will help you (1) review what process or outcome

information should be collected based on the evaluation questions

and (2) determine appropriate data sources and collection methods

accordingly. 

194

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
195

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• Out of the six

objectives listed,

how many were

covered by the

facilitator(s)?

• What barriers or

facilitators to

workshop

implementation

were evident?

• Was the facilitator

knowledgeable

about workshop

content?

• Was the information

presented in a

culturally appropriate

manner? 

• Did participants

understand the

material that was

presented?

• Did participants feel

that the material

that was presented

was culturally

appropriate?

• Number of professional

workshop objectives met

during the implementation

of the STD workshop 

• Barriers and supports for

conducting the workshop

or for staff to attend.

• Percent of participants

who thought the

facilitator was

knowledgeable about the

workshop content 

• Percent of participants

who felt the facilitator

presented information in a

culturally and linguistically

appropriate manner.

• Extent to which

participants felt the

materials provided were

understandable 

• Extent to which

participants felt the

materials were culturally

appropriate.

EVALUATION 

QUESTIONS

INDICATORS

Documents

(facilitators’ training

implementation log)

Individuals

(facilitator of the

workshop, STD

program staff who

organized the event)

Individuals

(workshop

participants)

DATA SOURCE

Table 3: Example of data collection sources and methods used to measure

indicators and to answer process evaluation questions for an evaluation of a

professional development workshop on Chlamydia screening protocols.

OVERARCHING PROCESS EVALUATION QUESTION:

What was the quality of the professional development workshop the STD clinic staff received?

DATA COLLECTION

METHODS

Document review.

Interview

(with facilitators

and program staff)

Survey 

(at the end of the

workshop)
196

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• What proportion

of the clinical staff

who attended the

workshop apply

prescribed

screening

protocols in all

clinical

encounters?

• Percent of STD

clinic staff who

attended the

workshop who

follow the STD

screening protocols

in patient

encounters.

EVALUATION QUESTIONS INDICATORS

• Documents

(patient records)

• Individuals

(patients,

clinicians)

• Observations

(on the job

performance

observations)

DATA SOURCE

Table 4: Example of data collection sources and methods used to

measure indicators and to answer outcome evaluation questions for

an evaluation of a professional development workshop on

Chlamydia Screening Protocols.

OVERARCHING OUTCOME EVALUATION QUESTION: How did the professional development workshop 

on STD screening protocols impact the STD clinic staff Ct screening practice?

DATA COLLECTION

METHODS

• Document review

• interview (with

patients and

clinicians)

• Observation

c Select potential data sources for each indicator.

c Consider the advantages and disadvantages of each data source.

c Select your data collection method(s) by considering the:

• purpose of the evaluation

• users of the evaluation

• characteristics of the respondents from whom you will collect the data

• resources available (time, money, volunteers, travel expenses, supplies)

• degree of intrusiveness

• type of information needed 

• advantages and disadvantages of each method

c Link your data sources and data collection methods to the indicators and

evaluation questions.

SUMMARY CHECKLIST: Determine the Data Sources and Methods for Indicators
197

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

CONCLUSION AND NEXT STEPS

In this tool you have learned the advantages and disadvantages of

data sources and data collection methods from which you can gather

information on your indicators. You have also learned about the

issues and factors to consider as part of the process of selecting this

information. 

Next, based on the data sources and data collection methods you

have chosen for your evaluation, you will learn how to effectively

collect your data. Tool 4.3 (Establish a Clear Procedure to Collect

the Information), will help you establish a clear procedure for the

data collection phase of your evaluation design, and address factors

that may affect the quality of the information you gather. 

ACRONYMS USED IN THIS TOOL

CDC – Centers for Disease Control

Ct – Chlamydia trachomatis

DSTDP – Division of STD Prevention

JDC – Juvenile Detention Center

SMART – Specific, Measurable, Achievable, Relevant, and Time-bound

STD – Sexually Transmitted Disease

KEY TERMS

Focus Group: A qualitative method used to collect data from a group

of people (about 6 - 11) who meet for 1-2 hours to discuss their

insights, ideas, and observations about a particular topic with a

trained moderator. Participants are selected because they share certain

characteristics (e.g., individuals who have been tested for syphilis,

women in detention facilities) relevant to the evaluation.

Indicator: A specific, observable, and measurable accomplishment or

change that shows whether progress has been made toward achieving

a specific program output or outcome.
Individual interview: A data collection method which involves

dialogue with individuals who are carefully selected for their personal

experience and knowledge with the issues at hand. Since these

interviews are conducted individually, they are useful when

anonymity is an issue or when asking about sensitive topics so

participants can feel free to express their ideas.

Mixed-method design: A methodological approach where you

collect data from more than one source and/or through different

methods. The advantages of using mixed methods include: increasing

the cross-checks on the evaluation findings, examining different

facets of the same phenomenon, and increasing stakeholders’

confidence in the overall evaluation results. An example of mixed

methods is using both a focus group and a survey to understand a

target population’s reluctance to use condoms.

Observation: A data collection method in which you take field notes

on the behavior and activities of individuals or describe the

environment while observing these in the field. For example, you

might take notes on the behavior of gay men in bath houses as part

of your data collection procedures, or take notes on how patients are

treated by clinic staff, and use such information to further develop or

improve your program. 

Primary data: Data directly obtained by your program (e.g.,

surveillance, number of sex partners of syphilis cases collected

through DIS interviews).

Qualitative methods: Data collection methods used to gather

narrative data to better understand the experiences of the target

population and how a program activity works.

Quantitative methods: Data collection methods that are used to

collect numerical data. An example is the use of a survey that queries

respondents about their sexual history using closed-ended questions

in which numbers can be assigned to responses (e.g., number of

sexual partners, frequency of condom use).

198

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
Secondary Data: Information your program can use that has been

collected by someone else (e.g., national data). This may include

epidemiological data, socio-demographics, health risk behaviors and

health policies.

Stakeholders: Individuals or organizations directly or indirectly

affected by your STD program and/or the evaluation results (e.g.,

STD program staff, family planning staff, representatives of target

populations). 

Surveillance data: Data collected in an ongoing, systematic way

regarding agent/hazard, risk factor, exposure, or health event.

Surveillance data are essential for the planning, implementation, and

evaluation of public health practice.

Survey: A method of collecting information that can be selfadministered, administered over a telephone, administered using a

computer or administered face-to-face. Surveys generally include

close-ended questions that are asked to individuals in a specific order

and provide multiple choice or discrete responses (e.g., “Have you

been tested for syphilis in the last 6 months?”). 

199

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
EXERCISE: 

SELECTING DATA SOURCES AND 

COLLECTION METHODS

Listed below in Tables 5 and 6 are the evaluation questions and

indicators that you and your stakeholders want to address in a

process and outcome evaluation of a Chlamydia screening program

in female juvenile detention centers (JDCs) in County Z. Based on

the information you have learned in this tool, as well as the

information listed in the tables, complete the tables by specifying the

data sources and collection methods you would use as well as the

rationale for source/method selection. 

Note: The completed tables are presented at the end of this tool.

200

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

What were the

barriers and

facilitators of

implementing Ct

screening in JDCs?

• Extent to which STD

program leadership

identifies project as a

priority.

• Percent of JDCs that

formally adopt the Ct

screening project.

• Percent of trained

medical providers

retained in project.

• Percent of trained

medical providers who

are committed to the

objectives of the project.

PROCESS EVALUATION

QUESTION

INDICATORS DATA SOURCE

Table 5: Provide data sources, data collection methods and rationale for their selection.

DATA COLLECTION

METHODS

RATIONALE FOR

SOURCE/METHOD

SELECTION
201

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

As a result of the

Ct screening

initiative, did

more JDCs screen

and treat

adolescent

females for Ct.

As a result of

participating in

the Ct screening

initiative, were

more adolescent

females in JDCs

screened and

treated for

Chlamydia?

Increased

number of JDCs

that provide Ct

screening,

counseling and

treatment

services for

adolescent

females.

Increased

number of

adolescent

females in JDCs

who are

screened and

treated for Ct.

OUTCOME 

EVALUATION 

QUESTION

OUTCOMES INDICATORS

Table 6: Provide data sources, data collection methods and rationale for their selection.

DATA 

SOURCES

RATIONALE FOR

SOURCE/METHOD

SELECTION

DATA 

COLLECTION

METHOD

Percent of JDCs

that provide Ct

screening,

counseling, and

treatment services

for adolescent

females 3 months

before and 3

months after the

initiative.

Increased number

of adolescent

females in JDCs

who are screened

and treated for Ct

Percent of females

in JDCs who were

screened for Ct 3

months before and

3 months after the

initiative.

Percent of

screened females

with positive Ct

results in JDCs who

were treated 3

months before and

3 months after the

initiative.
REFERENCES

Centers for Disease Control and Prevention, National Center for

Chronic Disease Prevention and Health Promotion, Division of

Adolescent and School Health. (2003). Handbook for conducting

youth risk behavior surveys. Atlanta, GA: Centers for Disease

Control and Prevention.

Creswell, J. W. (2003). Research design: Qualitative, quantitative, and

mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage.

MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowski, K.,

& Turner, K. (2001, November). Introduction to program

evaluation for comprehensive tobacco control programs. Atlanta, GA:

Centers for Disease Control and Prevention. Retrieved March 21,

2005, from http://www.cdc.gov/tobacco/evaluation_manual/

Evaluation.pdf 

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis:

An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.).

Thousand Oaks, CA: Sage. 

Porteous, N. L., Sheldrick, B. J., & Stewart, P. J. (1997). Program

evaluation tool kit: A blueprint for public health management.

Ottawa-Carleton, Ontario, Canada: Ottawa-Carleton Health

Department, Public Health Research, Education and

Development Program.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A

systematic approach (7th ed.). Thousand Oaks, CA: Sage.

Sanders, J. A. (1994). The program evaluation standards: How to assess

evaluations of educational programs. (2nd ed.). Thousand Oaks,

CA: Sage.

Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (1994).

Handbook of practical program evaluation. San Francisco: JosseyBass.

202

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
203

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

What were the

barriers and

facilitators of

implementing Ct

screening in JDCs?

• Extent to which

STD program

leadership

identifies project

as a priority.

• Percent of JDCs

that formally

adopt the Ct

screening

project.

• Percent of

trained medical

providers

retained in

project.

• Percent of

trained medical

providers who

are committed to

the objectives of

the project.

PROCESS EVALUATION

QUESTION

INDICATORS DATA SOURCE

Table 5: Process evaluation question, indicators, data source, data collection

method and rationale for source/method selection

DATA COLLECTION

METHODS

RATIONALE FOR

SOURCE/METHOD

SELECTION

• Documents

(Program

reports and

budget)

• Individuals

(Program

leadership & 

Ct screening

implementer)

• Documents

(Program

reports)

• Observation at

JDCs

• Documents

(program

reports)

• Individuals

(medical

providers)

• Document

review

• Interviews

• Document

review

• Observation

• Document

review

• Interviews

(with medical

providers)

These methods will

provide the most

comprehensive and

complete data in

order to answer the

evaluation question.
204

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

As a result of the

Ct screening

initiative, did

more JDCs screen

and treat

adolescent

females for Ct?

As a result of

participating in

the Ct screening

initiative, were

more adolescent

females in JDCs

screened and

treated for

Chlamydia?

Increased

number of JDCs

that provide Ct

screening,

counseling and

treatment

services for

adolescent

females.

Increased

number of

adolescent

females in JDCs

who are

screened and

treated for Ct.

OUTCOME 

EVALUATION 

QUESTION

OUTCOMES INDICATORS

Table 6: Outcome evaluation question, outcomes, indicators, data sources, data collection

method, and rationale for source/method selection.

DATA 

SOURCES

RATIONALE FOR

SOURCE/METHOD

SELECTION

DATA 

COLLECTION

METHOD

Percent of JDCs

that provide Ct

screening,

counseling, and

treatment services

for adolescent

females 3 months

before and 3

months after the

initiative.

Increased number

of adolescent

females in JDCs

who are screened

and treated for Ct

Percent of females

in JDCs who were

screened for Ct 3

months before and

3 months after the

initiative.

Percent of

screened females

with positive Ct

results in JDCs who

were treated 3

months before and

3 months after the

initiative.

• Documents

(detainees’

medical

records)

• Individuals

(medical

staff)

• Documents

(detainees’

medical

records,

surveillance

reports).

• Document 

review

• Interviews

(focus

groups)

• Document

review 

Source/method is

low-cost; is easy

to complete;

provides needed

evidence for

stakeholders

Source/method

are mediumcost; help

validate record

data; provides

in-depth

information on

process and

impacts

Source/method

are low- cost;

easy to abstract

data; highly

credible with

stakeholders;

provide required

evidence
205

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Tool 4.3: ESTABLISH A CLEAR PROCEDURE TO COLLECT

EVALUATION INFORMATION

________________________________

INTRODUCTION

You have identified the outputs and outcomes you want to measure, selected

indicators, identified data sources and the methods you will use to collect

data, and determined the resources you can devote to the evaluation. Now it

is time to collect the data to answer your evaluation questions. 

This evaluation tool will help you establish a clear procedure for the data

collection phase of your evaluation design and address factors that may affect

the quality of the information you gather. The completeness and quality of

the information gathered during data collection will determine whether your

evaluation generates practical, accurate, and useful information. The

flowchart below illustrates where establishing clear procedures to collect the

information fits in with your other evaluation activities.

UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS

PROGRAM GOALS

SMART

OUTCOME OBJECTIVES

SMART

PROCESS OBJECTIVES

ENGAGE

STAKEHOLDERS

THROUGHOUT

EVALUATION

EVALUATION PURPOSE, USERS, USES

EVALUATION OUTCOMES

EVALUATION DESIGN

PROCESS AND OUTCOME INDICATORS

DATA COLLECTION SOURCES

AND METHODS

DATA COLLECTION PROCEDURES

LOGIC MODEL

INPUTS ACTIVITIES OUTPUTS

PROCESS

SHORT-TERM

OUTCOMES

INTERMEDIATE

OUTCOMES

LONG-TERM

OUTCOMES

OUTCOME
206

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

LEARNING OBJECTIVES

After completing this tool, you will be able to:

• Develop data collection procedures. 

• Establish and implement quality control procedures for data

collection and management.

WHAT FACTORS DO YOU NEED TO CONSIDER WHEN

DEVELOPING DATA COLLECTION PROCEDURES?

Data collection involves administering instruments, and gathering

and organizing responses before analysis. A number of factors can

affect the quality, and thus the credibility, of the information you

assemble. Some of these factors include how you design data

collection instruments; forecasting when data should be collected;

determining who will be responsible for collecting and entering data;

and ensuring organizational requirements are met (e.g., informed

consent procedures, OMB approval, confidentiality concerns, or

departmental approvals).

Your data collection procedures should contribute to the reliability

and validity of the data collection methods and measures you use.

Methods are reliable if they obtain similar or identical results when

used repeatedly. For example, if you repeated a blood test three times

on the same blood sample, the test would be reliable if it generated

the same results each time. Valid methods/measures are those that

actually measure what they are supposed to measure. For example, a

question that asks sexually active adolescents how often they use a

condom is valid if it accurately measures their actual frequency of

condom use. It is not valid if instead respondents interpret that they

are being asked how frequently they should use condoms. 

Consider the following factors when developing the data collection

procedures.

1. Using existing data systems.

You have the option of using existing data systems (e.g., Disease

Intervention Specialist (DIS) interviews, surveillance systems,

Sexually Transmitted Disease [STD]/Management Information

Systems [MIS]). An advantage of this is that the instruments may

have been pre-tested to produce valid and reliable data. Be aware

that if you use another existing data base to inform aspects of

your evaluation, you may need to add specific questions to meet
207

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

your evaluation needs. Although existing data systems may

collect data routinely, they may have little flexibility with regard

to the questions asked in the instrument. If the existing data

systems in your project area can’t answer your evaluation

questions, you will need to design you own evaluation

instruments.

2. Designing Data Collection Instruments.

The design of your data collection instruments is an important

factor that influences the quality of your data. 

If you are designing your own data collection instruments: 1) you

may want to obtain advice from individuals who have experience

developing evaluation instruments (e.g., STD program or health

department staff, CDC evaluation staff); and 2) build in enough

time into your evaluation timeline to field-test the instruments and

modify them accordingly.

When developing questionnaires1

, keep them simple, short, focused,

and easy for respondents to complete. Use culturally appropriate

language, and very precise instructions. Questionnaires generally

include close-ended questions with multiple answers. Don’t combine

two questions in one. For instance, “What do you think about the

Chlamydia counseling you received today and the services you have

received at this clinic?” This question asks about multiple issues. The

first half has to do with opinions about the Chlamydia counseling,

and the other half has to do with opinions about the services received

at the clinic. The latter can still be broken-down into different

questions by asking about specific services. The problem with this

question is that you do not know which of the multiple questions

the respondent is answering, which may create confusion and affect

the validity of the responses. Also, consider whether the

questionnaire will be completed with the aid of a computer, selfadministered (paper-pencil), or administered by another person

either face-to-face or by telephone.

1 Fowler, F.F. (2002). Survey research methods (3rd ed.) California: Sage Publications. 
208

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

After developing a questionnaire or survey, you will want to pilot test

it. Pilot testing will assist you in determining: 

• How long it takes to complete the questionnaire. 

• Whether the order of the questions flow well. 

• Whether ample space is provided for responses. 

• Whether the directions, as well as the questions are

understood.

• Whether your questions are both reliable (i.e., obtains very

similar or identical results when used repeatedly) and valid

(actually measures what is supposed to measure). 

Another advantage to pilot testing is the opportunity to check the

responses you obtain against the major evaluation questions or issues

you’re exploring. Are patterns beginning to emerge? Are there

identical responses showing up in the “Other (Please Specify)”

category?

You will want to pilot test your instrument on people who resemble

your program participants. You also want to administer the

instrument under conditions similar to those you’ll be using in

gathering your data. Before pilot testing, ask some of your colleagues

to review the questionnaire, particularly those who have interest or

experience in evaluation or who may be familiar with the program or

audience you’re examining.

• Interview guides for use in one-on-one interviews and focus

groups should include open-ended questions.2 The guide should

allow some flexibility for the interviewer to probe and ask followup questions that can add depth to the information to be

collected. Develop an interview guide that is clear and easy to

follow by the interviewer. Field-test it with the interviewers and

among individuals with similar characteristics to the target

population(s) and modify accordingly. The effectiveness of

interviews depends on the skill of the interviewer. Thus, it is

critical to either have experienced interviewers or to train staff on

interviewing skills. DIS can be a great resource in this process. 

2 Morgan, D.L. and Krueger, R.A. (1998). The focus group kit. Thousand Oaks, CA: Sage

Publications.
209

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

• When you conduct interviews it is important to document what

the interviewees say, their body language, and the surrounding

environment (e.g., interview took place in a noisy area and this

interfered with the interview) since these factors may affect the

quality of the responses. It is recommended, with participants’

permission, to tape the interviews and to take detailed notes, in

case technology fails.

• Observation is a useful evaluation method to document

adherence to a protocol, but it can be very difficult to conduct

without using a structured instrument because the lack thereof

may affect the focus of the observer. Your instrument can be a

checklist that indicates the actions/behaviors that the observer

should record. Field-test it with your trained observers. The

parameters/criteria identified in your checklist should be clear

and unambiguous.

• When using Document Review, develop a checklist that includes

the information you need to abstract in a logical way. Also train

those who will conduct chart abstraction. Keep in mind that for

medical records, you may need individuals with clinical

background to abstract the information, and you will need to

become familiar and follow the confidentiality

procedures/protocols of the agency that you will be collecting the

data from.

• Coding is also an important part of the data collection process.

Coding can be initiated as part of the data collection design

when developing your evaluation questions (both close-ended

and open-ended), or once the instrument is developed. You

should note that if coding issues are not thought through early

on in the design process, data may not be appropriately coded.

Be aware that a code-book should be drafted early in the data

collection process. The data coding process is discussed in further

detail in Tool 5.1.
210

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

3. Determining when you need to collect data. 

You need to determine when and how frequently to collect the

data you need. Will you be collecting data before and/or after a

particular intervention (e.g., screening, health education, testing,

and media campaign)? Will you be collecting data at one

particular time in your program (e.g., quarterly)? Your evaluation

design should provide guidance as to when to collect certain

data. Let’s assume that you are evaluating a gonorrhea prevention

media campaign and one of the outcomes you are measuring

relates to increased awareness of the gonorrhea outbreak in your

target population. Since you have to compare changes in the

target population’s awareness as a result of the media campaign,

you may want to collect information from them before and

immediately after the intervention. 

Consider the following questions to determine when and at what

intervals you need to collect data: 

• When will the information be available? For example, if you

are using STD surveillance data from your state health

department, find out when the information is available and

plan accordingly.

• When collecting primary data (e.g., focus groups) try to

collect the information from your sources when it is the most

convenient and the least disruptive to them.

• When will you have appropriate resources to collect the data

(e.g., availability of staff at particular times, need for

technical assistance)? 

• How long will it take to collect the data? This is particularly

important in determining when your data collection efforts

will start and end. Be sure to allocate sufficient time to each

data collection task (e.g., completing 50 interviews with

inmates: arranging interview space to facilitate inmate input,

timing the interview during the field-test to determine the

range of completion times; allowing for transition of inmates

in and out of the interview space). When determining how

much time to allocate for interviews, estimate the amount of

time it will take a respondent to complete the interview, the

number of interviewers available, and the number of

interviews they can do in a day/week (given other

responsibilities, availability of respondents, etc.). 
211

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

4. Determining who will be responsible for collecting the

evaluation data you need. 

First, determine if your program staff have the specific

skills/qualities you need (e.g., experience/training in moderating

focus groups, speak the language of the target population). In

Appendix A you will see suggested skills and qualities of data

collectors. Please note that the data collection expertise may be

found in most state/local health departments in Assessment

Sections, DIS staff (interview skills), and Epidemiology Units. 

If you lack staff resources with the required skills, then you need

a plan for meeting those requirements. Having trained/skilled

individuals to collect your data affects the quality of your data.

In-service training can address some of the skills you need to

build/strengthen among the STD staff (see Appendix B for

advice on what information to cover when training data

collectors). In other cases, you will need to do outreach and

recruitment to identify individuals with the skills you need.

Consider contacting your DSTDP program consultant or

evaluation staff who can refer you to local resources (see Tool

3.2). Take steps to include training plans or recruitment efforts in

your data collection procedures. 

Along with the skills and qualities of your data collectors, you

should consider the types of evaluation data each is collecting. If

collecting information involves direct interviews with STD

program staff, then someone outside the program should

undertake this task to help assure frank responses. If a data

collection method involves record reviews (e.g., meeting minutes,

patient records), program staff may do this.

Keep in mind that if your evaluation design calls for aggregating

data across sites (e.g., STD clinics) or if more than one person is

collecting the data, it is crucial that data be collected in the same

way. This standardization of data collection will allow you to

aggregate the data and/or make comparisons. 
212

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

5. Dealing with administrative/logistical Issues. 

Your data collection procedures should also address certain

administrative or logistical issues that may affect your data

collection practices, and thus the quality of your data.

• Independent oversight and compliance with existing regulations.

Depending on the nature of the evaluation planned, the

methods of obtaining data, or the evaluation questions posed,

you may need to submit your evaluation proposal and data

collection instruments to an institutional review board (IRB)

to assure the protection of the rights of human subjects and

the use of informed consent, if needed. To do this you should

understand the approval process required by the IRB. Policies

for the approval of data collection activities vary across states

and local jurisdictions. It is important to understand the rules

and regulations that apply to your target population. 

Also, note that if you are planning a data collection that it is

federally sponsored and it involves 10 or more respondents,

you need to obtain OMB (Office of Management and

Budget) approval. For more information on what would

required OMB approval and the process please refer to

http://intranet.cdc.gov/od/ocso/osrs/ omb/OMBQ&A.pdf or

http://intranet.cdc.gov/od/ ocso/osrs/omb/OMB%

20Paperwork%20Reduction%20Act%20Fact%20Sheet.pdf.

• Security and confidentiality of the information being collected.

Ensuring the privacy of your program clients in an evaluation

should be an important element of your design. You can help

maintain confidentiality by stripping any identifiers from the

data gathered and making sure it is stored in a secure place. 

Your data security and confidentiality procedures should

extend to data entry and management. For example, decide

how the information will be transferred from the data

collectors to those responsible for entering data. How will the

information be computerized in a way that security and

confidentiality of the data are maintained? 

Your procedures should identify the physical location for

storing the data collected. In the case of focus groups, the

discussions are often tape recorded and transcribed. Make
213

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

sure that you identify how the tapes will be stored, who will

transcribe the tapes, and when will the tapes be destroyed. 

• Development of a codebook for questions included in the data

collection instruments. A codebook is a document detailing

instructions on how to code the data so that each data

element is coded in a standardized way. In quantitative data,

it specifies a brief name and description for each item or

question in a data collection instrument. In qualitative data,

coding is used to reduce the data by organizing the text

(data) into categories/themes. The codes are applied to text

segments that match the theme(s) associated with the code.

You do not have to wait until the data are collected to

develop a codebook. You can start working on this for

quantitative data once you finalize the instrument so that you

can design the program to be applied in analysis ahead of

time. For qualitative data, you can start this coding process

by determining main themes based on the questions in the

instrument. Then you will modify the code scheme during

and after data collection. 

• Completeness of data collection instruments. Before turning over

the completed data collection instruments to those who will

be entering, managing and analyzing the data, it is important

that you carefully check each instrument to assure its

completeness. If certain items or questions have not been

completed, your data collectors should try to retrieve this

information and/or clarify any responses that may not be

clearly written. 

6. Monitoring, Reviewing, and Revising Procedures.

During data collection, consider what is going well, what is not

going as expected, and what needs to be modified for the next

data collection. Revise your data collection design and methods

as needed, based on your resources (financial and human), how

the data collection process is received by clients, and how it is

perceived by your stakeholders, especially with respect to the

usefulness of the information you are collecting.
While data collection procedures are designed to maximize your

success with data collection, they do not guarantee that your

experience will be trouble-free. Despite all your planning, you

may discover that data collection is not going exactly according

to plan. This is not unusual, and you can learn from any

problems that you encounter. These problems may relate to

logistics (e.g. lack of child-care services for women of

childbearing age participating in a focus group); personnel (e.g.,

some staff are not correctly completing client files); or the

instruments themselves (e.g., question is being consistently

misinterpreted). It is important to identify issues early on and to

determine the actual source of the problem in order to correct it.

Pilot testing is a way if identifying data collection issues at an

early stage and we recommend that data collection instruments

be piloted and revisions made before instruments and data

collection procedures are used for the actual evaluation.

214

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
SUMMARY CHECKLIST: Establishing Clear Procedures for Data Collection

c Use data collection instruments that will contribute to the quality of data

collected.

• Try to use existing instruments that have been pre-tested to produce valid 

and reliable data.

– If possible, discuss the strengths and weaknesses of existing data instruments with

individuals who have used the instrument previously.

• When developing new data collection instruments:

– you may want to obtain advice from individuals who have experience developing

evaluation instruments;

– field-test the instruments and modify based on feedback.

c Determine the timing/frequency/schedule of your data collection activities.

Consider:

• When will the information you need be available?

• When will you have appropriate resources to collect the data?

• How long will it take to collect the data? 

c Establish who will be responsible for which data gathering activities 

(e.g., program staff, other staff, paid consultants, volunteers).

• What skills and qualities should the data collectors have?

• How will these requirements be met? What are the training and/or recruitment procedures?

• How will you assure that data collectors collect data in a standardized way?

c Determine what administrative or logistical issues need to be addressed.

• Do you need to obtain OMB approval?

• Do you need to obtain approval from any board or agency?

• Are informed consent procedures needed?

• What rules and regulations apply to your data collection activities?

• How will you address security and confidentiality related to data collection, entry, and

storage?

• How will you develop your codebook for the data collection instruments? 

• How will you assure the completeness of your data collection instruments? 

c Establish procedures for monitoring, reviewing, and revising your data collection

efforts to identify problems you encounter.

• How will you address problems related to personnel and other resources, logistics, and

reliability and validity of instruments?

• How will you address problems related to how the data collection process is perceived by

stakeholders?

215

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
216

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

CONCLUSION AND NEXT STEPS

Data collection procedures involve administering instruments,

gathering responses, and organizing responses before the data can be

analyzed. This tool has presented information on various factors and

issues to consider when developing your data collection procedures

to ensure the quality and accuracy of the information collected. 

Next, in Tool 4.4 (Complete an Evaluation Plan Based on Program

Description and Evaluation Design), you will learn how to develop an

evaluation plan that your program can easily follow. 

ACRONYMS USED IN THIS TOOL

Ct – Chlamydia trachomatis

DIS – Disease Intervention Specialist

MIS – Management Information System

STD – Sexually Transmitted Disease

OMB – Office of Management and Budget

KEY TERMS

Code book: A document detailing instructions on how the data for a

specific evaluation is coded. It describes each code so that codes are

applied to the data in a standardized way.

Coding: In quantitative analysis this is the process of arranging the

data so that the computer can “read” the code and perform an

analysis (e.g., if one of the variables is “sex” you might code this as 1

for “female” and 2 for “male”). In qualitative analysis, coding is used

to reduce the data by organizing the text (data) into categories/

themes. The codes are applied to text segments that match the

theme(s) associated with the code.

Data collection: The process of administering instruments and

gathering responses.

Reliability: The consistency of a measure or question in obtaining

very similar or identical results when used repeatedly. 

Validity: The extent to which a question actually measures what it is

supposed to measure. For example, a question that asks how often an

individual uses a condom is valid if it accurately measures the actual

level of condom use; it is not valid if instead it measures the extent to

which an individual realizes that s/he should wear a condom.
EXERCISE

County Z STD program staff and stakeholders have decided to

evaluate the Chlamydia (Ct) training that the health department

conducted for clinical staff in the juvenile detention center. They

have developed a logic model for the training program, specified

indicators, and identified the data collection methods they will use

based on available resources. They are now identifying the data

collection procedures. 

Based on the information below, complete the template with

information about the outcome, indicators, data source and method,

and corresponding data collection procedures. In summarizing data

collection procedures, specify the schedule for data collection, who is

responsible for data collection, the logistics/confidentiality, and data

quality control. [Note: The completed template is provided at the

end of this tool—Table 1.]

OUTCOME:

Increased adherence of clinical staff to Ct clinical guidelines.

INDICATORS:

(1) Percent of trained clinical staff who can correctly apply Ct

screening and treatment protocols with patients before the

training. 

(2) Percent of trained clinical staff who can correctly apply Ct

screening and treatment protocols with patients 30 days

following the training.

DATA SOURCE:

Trained clinical staff

METHOD:

Observation (i.e., Ct screening observation instrument)

217

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
218

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

DATA COLLECTION PROCEDURES:

Supervisors who are trained in the use of a Ct screening observation

instrument will observe each trained clinical staff person and

complete an observation instrument for each person within one

month of the training. To test the reliability and validity of the

observation instrument, it will be pre-tested with a sample of clinical

staff who are conducting Ct screening, but did not attend the

training that is being evaluated. To ensure confidentiality of the data,

the supervisor will create a unique code number for each completed

instrument. (Staff X) will collect all completed observation

instruments from supervisors, check for the completeness of the

instrument and place them in locked file drawers in the office of the

STD Director. Only evaluation staff will have key access to the file

drawers.

TEMPLATE FOR EXERCISE

OUTCOME INDICATORS DATA SOURCE

DATA

COLLECTION

METHOD

SCHEDULE RESPONSIBILITY LOGISTICS/

CONFIDENTIALITY

DATA QUALITY

CONTROL

DATA COLLECTION PROCEDURES
APPENDIX A

Suggested Skills and Qualities of Data Collectors

• Organizational skills. 

• Good memory and an eye for details. These skills are most

critical when gathering data through observation and reviewing

records. 

• Familiarity with program activities to be evaluated (e.g.,

knowledge on social marketing for a condom campaign).

• Experience with the data collection methods selected for the

evaluation. 

• Cultural competence and experience in data collection with the

populations the program works with. 

• Communication skills, including being a good listener and

communicating concepts clearly.

• People skills, such as being respectful (not condescending) and

the ability to start and maintain conversations with individuals

they do not know. These skills are particularly important when

interacting directly with the target population(s) (e.g., interviews

and focus groups).

• Ability to refrain from expressing one’s own opinions, attitudes,

and ideas because this may influence individuals’ responses. 

• Flexibility in adapting to changes in the way and the number of

questions to ask (e.g., you may need to revise your data

collection instruments by adding/deleting/rewording questions).

• Dependability.

219

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
APPENDIX B

Training Data Collectors

More than one person may be involved in your data collection effort.

The level and extent of training they need depends on the

complexity of the data collection method and the individuals’ skills. 

Your training should include:

• An orientation on the purpose of the evaluation and how the

information will be used.

• Logistical details involved in data collection, including where the

forms/instruments are obtained, what to do with them when

completed, any approvals that must be obtained, calendar and

timing considerations, and safeguarding confidentiality of

information and information sources. 

• Data collection methods and techniques (e.g., personal interviews,

focus groups, review of program /clinical records, and/or

observation techniques). For example, if you are conducting

interviews, your interviewers should fully understand the

questions being asked, when to probe for an expanded response

or for clarification, how to carry out the interview, and how to

record the data objectively, consistently, and carefully.

• Opportunities to practice administering the instrument (e.g.,

questionnaire, interview guide, observation log). Practice makes

perfect and increases the quality of the evaluation data.

220

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS
221

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

REFERENCES 

Centers for Disease Control and Prevention. (1999). Framework for

program evaluation in public health. MMWR Recommendations

and Reports, 48(RR-11). Retrieved October 17, 2004, from

http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm

Centers for Disease Control and Prevention. (n.d.). Oral health

infrastructure development tools, Step 2B: Logic models. Retrieved

October 17, 2004, from http://www.cdc.gov/oralhealth/

library/pdf/logic_models.pdf

Taylor-Powell, E., Steele, S., & Doughlah, M. (1996, February).

Planning a program evaluation. Madison: University of WisconsinExtension, Division of Cooperative Extension. Retrieved October

17, 2004, from www.uwex.edu/ces/pdande/

evaluation/evaldocs.html 

W.K. Kellogg Foundation. (1998, January). W.K. Kellogg Foundation

evaluation handbook. Retrieved October 17, 2004, from

http://www.wkkf.org/Programming/ResourceOverview.aspx?

CID=281&ID=770
222

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

TOOL 4.4: COMPLETE AN EVALUATION 

PLAN BASED ON PROGRAM DESCRIPTION 

AND EVALUATION DESIGN

________________________________

INTRODUCTION

You learned how to describe your program through well-written

goals and SMART objectives (Tool 2.2) and developed logic models

(Tool 2.4) — all with input from program staff and stakeholders.

You also learned how to select program evaluation questions (Tool

3.3) and identify corresponding indicators for each question (Tool

4.1). Further, you learned how to select appropriate and feasible data

collection methods suitable for each source of data, as well as data

collection procedures (Tools 4.2 and 4.3). 

UNDERSTANDING OF PROGRAM FOCUS AND PRIORITY AREAS

PROGRAM GOALS

SMART

OUTCOME OBJECTIVES

SMART

PROCESS OBJECTIVES

ENGAGE

STAKEHOLDERS

THROUGHOUT

EVALUATION

EVALUATION PURPOSE, USERS, USES

EVALUATION OUTCOMES

EVALUATION DESIGN

PROCESS AND OUTCOME INDICATORS

DATA COLLECTION SOURCES

AND METHODS

DATA COLLECTION PROCEDURES

EVALUATION PLAN

LOGIC MODEL

INPUTS ACTIVITIES OUTPUTS

PROCESS

SHORT-TERM

OUTCOMES

INTERMEDIATE

OUTCOMES

LONG-TERM

OUTCOMES

OUTCOME
223

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

In Tool 4.4, you will learn how to organize all this information

into a comprehensive evaluation plan. The flowchart on the previous

page provides a description of where an evaluation plan fits in with

your past evaluation planning activities. 

LEARNING OBJECTIVE

Upon completion of this tool, you will be able to:

• Develop an evaluation plan to guide the planning and

implementation of a program evaluation.

WHAT IS THE PURPOSE OF AN EVALUATION PLAN?

An evaluation plan gives you the opportunity to verify that you have

addressed all the elements required to implement and complete a

successful evaluation. An evaluation plan also helps program staff to

understand stakeholder perspectives on the evaluation process and to

clarify stakeholder expectations of program results.

WHAT ARE THE COMPONENTS

OF AN EVALUATION PLAN? 

An evaluation plan has two main components: (1) a narrative

component and (2) a matrix, both of which enable you to

operationalize the evaluation. 

The narrative component of an evaluation plan includes the

following: 

• Sexually Transmitted Disease (STD) program component or

activity to be evaluated

• Rationale for evaluating this STD program component or activity 

• Purpose of the evaluation

• Goal(s) and objectives that relate to the program

component/activity to be evaluated

• Logic model(s)

• Individuals and their roles in the evaluation team

• Users and uses of evaluation findings

• Approach to disseminating evaluation findings to appropriate

users

• Timeline for completing the evaluation

• Evaluation budget
224

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

The evaluation matrix of an evaluation plan includes the following:

• Evaluation questions

• Indicators

• Data sources and data collection methods

• Data collection procedures

– Person(s) responsible

– Schedule

• Data analysis (see Tool 5.1 for more details)

This evaluation plan reflects most of the steps in CDC’s Framework

for Program Evaluation in Public Health1

, and thus serves as a valuable

resource for planning and implementing your evaluation. As you

noticed, most of the evaluation plan components have already been

addressed and built in the previous tools (1.1, 2.2, 2.4, 3.1, 3.2, 3.3,

4.1, 4.2, 4.3). Tool 4.4 compiles the work done up to this point. 

HOW DO YOU CONSTRUCT AN EVALUATION PLAN? 

Following are some suggested steps to help you construct your

evaluation plan.

1. Gather all relevant materials. 

It is likely that you have had several meetings with program staff

and stakeholders to discuss elements of the evaluation, taken

minutes of these meetings, and developed several documents

essential to constructing your evaluation plan. 

Prior to developing the evaluation plan, make sure you have:

• reviewed your Comprehensive STD Prevention Systems

(CSPS) grant application

• gathered all the documents that describe which STD

program component or activity will be evaluated and why 

• determined the goals and objectives pertaining to the

program component or activity to be evaluated 

• determined who will be involved in the evaluation, who will

use the evaluation findings and how they will use them 

• the final version of the logic model(s) 

• the list of evaluation questions and corresponding indictors,

data sources, data collection methods, ideas on how you may

want to analyze the data 

• available resources and staffing for the evaluation 

1 Centers for Disease Control and Prevention. Framework for Program Evaluation in Public

Health. MMWR 1999; 48 (no. RR–11)
225

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

List the STD program activity to be

evaluated.

List stakeholders (agency) involved

in the evaluation.

List the rationale for evaluating this

STD program activity.

List the purpose of the evaluation.

2. Organize and sort the relevant materials. 

Divide the materials into two groups. The first group will include

all the information you will need to complete the narrative

component of the evaluation plan. The second will include all

the information you will need to complete the evaluation matrix.

Next, sort the materials in each group to match the sequence of

items in the narrative and matrix components.

3. Complete the narrative component of the evaluation plan. 

The guidelines presented in Table 1 will help you to complete

the narrative evaluation plan.

4. Complete the matrix component of the evaluation plan. 

Table 2 provides you with guidance for completing each column

in the evaluation plan matrix. The plan that is presented does not

include information on data analysis. Specifics on this can be

found in Tool 5.1 (Analyze the Evaluation Data). 

Table 1: Guidance for completing the evaluation plan narrative

OUTCOME GUIDELINES

Based on your CSPS grant application and per your discussion with the

stakeholders, indicate the program activity that you plan to evaluate as well as the

purpose and rationale for evaluation of the program activity (refer to Tools 1.1,

2.1, and 2.2).

For example:

Program activity to be evaluated: Chlamydia (Ct) screening training

workshops.

Stakeholders involved in the evaluation: Program staff, external evaluator,

County STD clinic directors and staff, State STD program staff, professional

development organizers and implementers.

Rationale for selecting the program activity: Based on the findings from the

needs assessment, the program staff and stakeholders decided that clinician

training on Ct screening would increase appropriate diagnosis and treatment

of Ct infections. It was agreed that screening, treatment, and diagnosis

protocols should be consistently implemented, and that since professional

development (PD) is the foundation for the implementation of a quality

screening initiative, it is important to assess its effectiveness.

Purpose of the evaluation: Evaluate the extent to which PD workshops led to

clinical staffs’ increased knowledge and skills for screening, counseling, and

treatment protocols.

continued
226

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

List the goal(s) and objectives that

relate to the program component/

activity to be evaluated.

Attach appropriate logic models.

List individuals and their roles in

the evaluation team.

List the users and uses of the

evaluation findings.

List the approach to disseminating

the evaluation findings to

appropriate users.

Attach the timeline for completing

the evaluation.

Attach the evaluation budget.

5. Update the evaluation plan, as needed.

Changes may need to be made in parts of your evaluation plan due to changes in the

program component or activity you are evaluating, field conditions of the evaluation or

other practical considerations. Update your logic model, staff information, the timeline,

or other aspects of the plan, as needed.

Table 1: Guidance for completing the evaluation plan narrative (continued)

OUTCOME GUIDELINES

Your CSPS grant application includes your program goals and objectives. In this

section, list only those goals and objectives that are related with the program

component or activity you want to evaluate. Make sure that your objectives are

“SMART” (i.e. Specific, Measurable, Achievable, Relevant, and Time-bound). Also

make sure that they are articulated as process or outcome (short-term,

intermediate, or long-term) objectives (refer to Tool 2.2).

Example: You may be implementing a Ct screening initiative throughout

your project area with one component being PD workshops on Ct

screening for clinical staff. Since you have decided to evaluate the

effectiveness of the PD workshops in increasing participants’ knowledge

and skills, and not other parts of the initiative, include only the

objectives related to the PD workshops.

Make sure you include a logic model of the program component or activity you

want to evaluate (refer to Tool 2.4). Keep in mind that the logic model is a work in

progress. Highlight the specific components of the logic model you intend to

evaluate.

Staff your evaluation with individuals who are well-qualified and available for the

job (refer to Tool 3.2). Identifying all members (including stakeholders) and their

roles will allow you to proceed smoothly. This will also allow you to assess the

need for requesting technical assistance from DSTDP and/or hiring external

evaluation consultants.

Identify all stakeholders (i.e., implementers, decision makers, participants, and

partners) interested in the evaluation findings and how they would intend to use

them (refer to Tools 1.1 and 3.1).

Describe how the evaluation findings will be presented to (method of delivery

such as full report, short report, manuscript, presentation, etc.) and shared (mailed

report, discussion, etc.) with appropriate users.

The timeline should reflect the entire evaluation timeframe; from discussion of

evaluation questions with stakeholders to dissemination of results (refer to Tool 4.3).

The evaluation budget should include the direct and indirect costs of your

evaluation. If you are working with an external contractor, the cost for the

contractor can be broken out by consultant’s salary, fringe benefits, and nonpersonnel costs (refer to Tool 4.3).
PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 227 • Develop and prioritize each evaluation question with input from stakeholders. • Link each evaluation question with the goals and objectives of your program and the purpose of your evaluation. • Verify that the questions reflect the key elements of your program logic model that your evaluation will address, and that they can be addressed using the resources available at hand, i.e. budget, staff, and staff time. • Verify that the findings generated from the questions will be useful EVALUATION QUESTION • For each evaluation question, create an indicator to reflect achievement (e.g. number of staff trained, percent of health clinics implementing a policy, etc.). • If needed, you may use more than one indicator per question. INDICATORS • For each evaluation question, indicate what source(s) has the information to answer the question (i.e., individuals, observations, or documents). • A single data source can provide information on more than one evaluation question. DATA SOURCE(S) • For each data source, identify and list the best method to gather data (e.g., use of telephone rather than mail survey for a certain target group). • More than one data collection method may be used to gather information from one data source.). DATA COLLECTION METHODS • For each data collection method, identify roles and responsibilities of ALL individuals involved. This will allow you to estimate the workload of each individual, maintain a suitable staffing level, and develop a reasonable timeline for data collection. PERSON RESPONSIBLE • For each data collection method, fill in the data collection schedule (e.g., baseline survey data will be collected by mm/dd/yy; follow-up survey data will be collected by mm/dd/yy). SCHEDULE PROCEDURE TIMELINE PERSON RESPONSIBLE

Table 2: Evaluation Plan Matrix6

DATA COLLECTION PROCEDURES DATA ANALYSIS1

6 This plan needs to incorporate the data analysis component which will be discussed in Tool 5.1.
228

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

CONCLUSION AND NEXT STEPS

A completed evaluation plan will address most steps from CDC’s

program evaluation framework. The plan will include narrative and

matrix components to provide a comprehensive picture of the rationale

of, and approach to, the evaluation. This Tool has shown you how to

consolidate the evaluation information into one plan. 

Now that you have an evaluation plan, it is time for you to implement

your evaluation. In Tool 5.1 (Analyze the Evaluation Data), you will learn

how to manage, analyze, and synthesize your evaluation data.

c Gather relevant documents from the various meetings you have had with

program staff and stakeholders.

• CSPS grant application

• Meeting minutes 

• Documents such as logic models, evaluation budget and evaluation timeline

c Sort and organize relevant documents for the narrative and matrix 

components of the evaluation plan, to reflect the flow of these components 

in the evaluation plan.

c Complete the narrative component of the evaluation plan. Include:

• STD program or activity to be evaluated

• Rationale for evaluating this STD program or activity 

• Purpose of the evaluation

• Program goal(s) and objectives to be addressed through the evaluation

• Logic model(s)

• Individuals and their roles in the evaluation team

• Users and uses of evaluation findings

• Approach to disseminating evaluation findings to appropriate users

• Timeline for completing the evaluation

• Evaluation budget

c Complete the matrix component of the evaluation plan. Include:

• Evaluation questions

• Indicators

• Data sources 

• Data collection procedures

– data collection methods 

– person(s) responsible

– schedule

• Data analysis

c Update the evaluation plan (e.g. logic model, staff, timeline, etc.) if needed.

SUMMARY CHECKLIST: Analyze the Evaluation Data
229

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

ACRONYMS USED IN THIS TOOL

CSPS – Comprehensive STD Prevention Systems

Ct – Chlamydia trachomatis

JDC – Juvenile Detention Centers

STD – Sexually Transmitted Disease

PD – Professional Development

PDSB – Program Development and Support Branch

SMART – Specific, Measurable, Achievable, Relevant, and Time-bound

KEY TERMS

Evaluation plan: A document that includes what an evaluation

consists of (i.e., purpose/uses/users of the evaluation, program goals

and objectives related with the evaluation, logic model, evaluations

questions and design, data collection sources and methods, and

dissemination plan) and the procedures that will help guide the

implementation of evaluation activities to be undertaken by your

program. 

Goal: A broad statement related to the purpose of your program that

states what your program will accomplish (the desired result).

Indicator: A specific, observable, and measurable accomplishment or

change that shows whether progress has been made toward achieving

a specific program output or outcome. 

Logic model: A picture of how a program/component/activity is

supposed to work.

Objectives: Measurable statements that describe the manner in

which your program goals will be achieved. 

Post-only design: A non-experimental design where measures (data

collection) are taken from the target population(s) after the

activity/intervention. Since this is a non-experimental design, it does

not involve comparison/control groups.
230

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Pre/post design: A non-experimental design where measures (data

collection) are taken from the target population(s) before and after

the activity/intervention.

Stakeholders: Individuals or organizations directly or indirectly

affected by your STD program and/or the evaluation results (e.g.,

STD program staff, family planning staff, representatives of target

populations). 

Survey: A method of collecting information that can be selfadministered, administered over a telephone, administered using a

computer or administered face-to-face. Surveys generally include

close-ended questions that are asked to individuals in a specific order

and provide multiple choice or discrete responses (e.g., “Have you

been tested for syphilis in the last 6 months?”).

CASE SCENARIO 

The following is an example of an evaluation plan for assessing a

professional development (PD) component of a larger Chlamydia

(Ct) screening initiative that the STD Program plans to conduct in

juvenile detention centers (JDCs) in County Z. The example

provides both the evaluation plan narrative and matrix based on all

the steps detailed in this evaluation tool. This evaluation plan is not

exhaustive and is provided only to illustrate a sample evaluation plan.
231

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Evaluation Plan Narrative

STEP APPLICATION

List the STD program activity to be

evaluated.

List stakeholders (agency) involved

in the evaluation.

List the rationale for the STD

program activity to be evaluated.

List the purpose of the evaluation.

List the goal(s) and objectives that

relate to the program component/

activity to be evaluated.

Attach appropriate logic models.

• Program activity to be evaluated: PD workshops in Ct screening offered to

JDC medical providers in County Z.

• Stakeholders involved in the evaluation: Program staff, County and State

agency assisting with PD workshops, STD clinical staff participating in the

workshop.

• Rationale for selecting the program activity: PD activities are the

foundation for providing the necessary knowledge and skills to implement

appropriate Ct screening, counseling, and treatment protocols. High-quality

PD workshops should lead to consistency in implementation of the

recommended screening, counseling, and treatment protocols

• Purpose of the evaluation: To evaluate the implementation and

effectiveness of PD workshops.

GOAL: Reduce STD rates among adolescent females in JDCs in County Z.

PROCESS OBJECTIVE:

• By (month/year), contracted organization (X) will conduct (#) professional

development events with (#) medical providers in JDCs in County Z on

appropriate Ct screening, counseling, and treatment for female adolescents.

SHORT-TERM OUTCOME OBJECTIVES:

• By (month/year), the medical staff from County X JDCs who participated in

the training will demonstrate, in a pre- and post-training questionnaire, an

increase in their knowledge of the Ct screening protocols from X% to Y%.

• By (month/year), the medical staff from County X JDCs who participated in

the training will demonstrate, in a pre- and post-training questionnaire, an

increase in their knowledge of the Ct counseling protocols from X% to Y%.

• By (month/year), the medical staff from County X JDCs who participated in

the training will demonstrate, in a pre- and post-training questionnaire, an

increase in their knowledge of the Ct treatment protocols from X% to Y%.

• By (month/year), the medical staff from County Z JDCs who participated in

the training will demonstrate, in a performance exercise, an increase in

their skills in conducting Ct screening from X% to Y%, as recorded in a

skills performance sheet.

• By (month/year), the medical staff from County Z JDCs who participated in

the training will demonstrate, in a performance exercise, an increase in

their skills in conducting Ct counseling from X% to Y%, as recorded in a

skills performance sheet.

• By (month/year), the medical staff from County Z JDCs who participated in

the training will demonstrate, in a performance exercise, an increase in

their skills in providing Ct treatment from X% to Y%, as recorded in a skills

performance sheet.

See Appendix A.

continued
232

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Evaluation Plan Narrative (continued)

STEP APPLICATION

List individuals and roles on the

evaluation team.

List the users and uses of the

evaluation findings.

List the approach to disseminating

the evaluation findings to

appropriate users.

Attach the timeline for completing

the evaluation.

Attach the evaluation budget.

• Project director: Oversees all evaluation activities.

• Project manager: Develops timeline; hires and trains; supervises data

collection and handling; reviews all components of evaluation and final

report; disseminates findings.

• College intern: Develops evaluation instruments in consultation with project

manager and trainers; trains program staff in data entry; analyzes data.

• Contractor (X) training staff: Develops, staffs, and implements training for

JDC medical staff; administers pre- and post-training questionnaire and

performance exercise; completes skills performance sheet for each training

participant; analyzes data; drafts final report.

• STD program administrative assistant: Maintains confidential files of

completed evaluation tools; conducts data entry of evaluation data.

• Implementers (JDC medical staff, JDC administrators, STD program trainers):

Determine the effectiveness of the workshop in changing knowledge and

skills of participants. Use evaluation findings to improve future workshops.

• Decision makers (STD program director and manager, Correction System

Director): Ensure the quality of the larger Ct screening initiative in JDCs;

demonstrate the value of PD to stakeholders.

• Partners (JDCs and community representatives): Encourage in-service PD for

JDC staff in the future; ensure that detainees are receiving quality care.

• Funders: Written report.

• Other STD programs: Presentation and Written Report for Program

Development and Support Branch (PDSB) Thursday Report, presentation at

National STD Prevention Conference 

• JDC staff and detainees: Report and presentation

• Advocacy group: Use of media (radio, newspaper)

• Scientific community: Manuscript publication.

See Appendix B.

See Appendix C.
PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS 233 1.How many PD workshops were offered by the contracted organization? 2.How many medical staff attended each workshop? 3.Was the workshop content delivered as planned? 4. Were participants satisfied with the training they received? EVALUATION QUESTION Number of PD workshops implemented by the contracted organization. Number of medical staff who attended each workshop. Number of content topics addressed during training. Percent of participants satisfied with the training. INDICATORS Documents (workshop planning log to verify if (#) workshops were delivered in a timely manner) Documents (workshop attendance roster) Documents (training implementation log maintained by workshop trainers) Individuals (training participants) DATA SOURCE(S) Document Review (program manager will complete workshop planning log) Document Review (trainers will provide workshop attendance roster and all completed implementation logs to the program manager) Observation (college intern will complete a checklist to record how well trainers follow the script) Survey (self- administered post- training survey) DATA COLLECTION METHODS Program manager and trainers. PERSON RESPONSIBLE All evaluation instruments will be completed at the time of individual trainings (mm/dd/yy). SCHEDULE Frequency (count) of number of workshops conducted. Count of workshop attendees. Code themes from program implementation log and observation checklist for each session. Compute percentages. PROCEDURE All data will be analyzed by mm/dd/yy. Tables and graphics will be developed by mm/dd/yy. TIMELINE All data will be analyzed by mm/dd/yy. Tables and graphics will be developed by mm/dd/yy. PERSON RESPONSIBLE Evaluation Plan Matrix DATA COLLECTION PROCEDURES DATA ANALYSIS1 Process Evaluation Objective: By (month/year), contracted organization (X) will conduct (#) professional development events with (#) medical providers in juvenile detention centers (JDCs) in County Z on appropriate Chlamydia (Ct) screening, counseling, and treatment with female adolescents. (continued)
234

GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

D

ATA

C

OLLECTIO

N PR

O

CED

U

RES

1.Did medical staff

who participated in

the training

increase their

knowledge of

related protocols?

2. Did medical staff

who participated in

the training

demonstrate an

increase in Ct

screening,

counseling, and

treatment skills?

EVALU

ATIO

N

Q

UESTIO

N

Percent of

participants who

can correctly

describe Ctprotocols for

screening,

counseling, and

treatment

Percent of

participants who

can demonstrate

appropriate skills in

conducting Ct

screening,

counseling, and

treatment.

IN

DIC

ATO

RS

Individuals (training

participants)

Observations (skills

used during

trainingperformanceexercise

D

)

ATA

SO

U

R

CE(S)

Survey (selfadministered

questionnaire to be

completed by

participants before

and right after the

training)Observation

(trainers will score

participants’ skillsduring exercise and

record them on

skills performance

sheets)

DATA

C

OLLECTIO

N

METH

O

DS

Trainer will oversee

participants’

completion of

surveys.

Trainers.

PERSO

N

RESPO

NSIBLE

Before (m

m/dd/yy)

and im

mediately

after PD workshops

(m

m/dd/yy).

By end of trainings

(m

m/dd/yy).

SC

HED

ULE

Compute changedifference in pre

and post-trainingscore for each

participant.Using postworkshop data

identify and selectthose with correctresponses for eachprotocol.

Compute

percentage/proportion ofparticipants who

demonstrate low,medium, and high

skills (will use a

tercile split wherein

the lowest onethird is categorized

as low skill, middleone-third as

medium skill, and

top one-third as

high skill level).

PR

O

CED

U

RE

All data will be

analyzed bym

m/dd/yy.

Tables and graphics

will be developedby m

m/dd/yy.

All data will be

analyzed bym

m/dd/yy.

Tables and graphics

will be developedby m

m/dd/yy.

TIMELINE

Program

manager

will work closelywith the collegeintern.

Program

manager

will work closelywith the collegeintern

P

.

ERSO

N

RESPO

NSIBLE

Ev

alu

atio

n

Pla

n

M

atrix (co

ntin

u

e

d)

D

ATA

A

N

ALYSIS

Outco

me

SHORT-TERM

OUTCO

ME EVALUATIO

N

OBJECTIVES:

• By (month/year), the medical staff from County

X JDCs who participated in the training will

demonstrate, in a pre- and post-training

questionnaire, an increase in their knowledge

of the Ct screening protocols from

X

% to Y

%.

• By (month/year), the medical staff from County

X JDCs who participated in the training will

demonstrate, in a pre- and post-training

questionnaire, an increase in their knowledge

of the Ct counseling protocols from

X

% to Y

%.

• By (month/year), the medical staff from County

X JDCs who participated in the training will

demonstrate, in a pre- and post-trainingquestionnaire, an increase in their knowledge

of the Ct treatment protocols from

X

% to Y

%.

• By (month/year), the medical staff from County

Z JDCs who participated in the training will

demonstrate, in a performance exercise, an

increase in their skills in conducting Ct

screening from

X

% to Y

%, as recorded in a

skills performance sheet.

• By (month/year), the medical staff from County

Z JDCs who participated in the training willdemonstrate, in a performance exercise, an

increase in their skills in conducting Ct

counseling from

X

% to Y

%, as recorded in a

skills performance sheet.

• By (month/year), the medical staff from County

Z JDCs who participated in the training willdemonstrate, in a performance exercise, an

increase in their skills in providing Ct treatment

from

X

% to Y

%, as recorded in a skills

performance sheet.
APPENDIX A1

Evaluation Components Highlighted 

in the Program Logic Model

235

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

Funds

CDC/DSTDP, other

federal sources,

state sources, local

sources, private

sources

JDC Staff 

JDC administrators,

health care provider

staff, administrative

staff

Technical Direction,

Assistance, and

Collaboration

Local/state health

department and

laboratories

Contracted partners

performing staff

training

Advisory committee

Materials

Protocols/guide-lines

for screening

Prevention

education material

Provide

professional

development in 

screening and

treatment.

INPUTS ACTIVITIES OUTPUTS SHORT-TERM 

OUTCOMES

INTERMEDIATE

OUTCOMES

Logic Model of Chlamydia (Ct) Screening Program for Adolescent Females in County Z 

Juvenile Detention Centers (JDCs)

LONG-TERM

OUTCOMES

Provide JDC

support for

developing and

implementing

screening program.

Conduct screening,

counseling, and

treatment for

sexually active

female adolescent

admittees.

JDC clinical staff

participated in

professional

development (PD)

events on screening

counseling, and

treatment.

JDC administration

authorized time

and staff for

development and

implementation of

screening program.

Female adolescents

counseled and

screened

JDC Providers

Improved

knowledge and

skills for screening,

counseling

treatment services.

JDCs

Increased number of

programs in JDCs

that provide

screening,

counseling and

treatment services.

Adolescent

Females

Increased

awareness of and

other STD

transmission and

prevention.

Increased intention

to use condoms.

JDC Providers

Increased clinicians’

adherence to

clinical practice

guidelines

(screening and

treatment).

Adolescent

Females

Decreased number

of partners.

Increased health

seeking behavior.

JDCs

Reduction of

prevalence among

adolescent females

in JDCs.

1 Highlighted boxes indicate program components to be evaluated.
APPENDIX B

Projected Timeline for Evaluation Activities

236

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

MONTHS EVALUATION ACTIVITIES

Plan evaluation with program staff

and stakeholders.

Develop evaluation data collection

instruments, and train data

collectors.

Monitor attendance and

participation levels of workshop

participants.

Train clinical staff and collect preand post-training survey from them.

Collect post-workshop performance

exercise data from participants.

Analyze evaluation data

Report findings.

Revise program, if needed.

1 2 3 4 5 6
APPENDIX C 

Six-Month Evaluation Budget

237

GATHER CREDIBLE EVIDENCE 4

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS

EVALUATION ACTIVITY COST

College intern stipend (instrument development, data analysis, staff training) $ 300.00 

Contractor $2,000.00

Communications (postage, telephone calls, etc.) $ 30.00 

Printing and Duplication $ 30.00 

Supplies and Equipment $ 70.00 

Indirect Costs $100.00 

TOTAL $2,530.00 
REFERENCES 

Centers for Disease Control and Prevention, National Center for

Chronic Disease Prevention and Health Promotion, Division of

Adolescent and School Health. (2004). Evaluation steps tools.

Unpublished document.

Stecher, B. M., & Davis, W. A. (1987). How to focus an evaluation.

Newbury Park, CA: Sage.

238

4 GATHER CREDIBLE EVIDENCE

PRACTICAL USE OF PROGRAM EVALUATION AMONG STD PROGRAMS