TB Elimination

TM

TB Program Evaluation 

Handbook
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

TB Program Evaluation Handbook: 

Introduction to Program Evaluation 

Tuberculosis Evaluation Work Group 

Division of Tuberculosis Elimination 

National Center for HIV, STD, and TB Prevention 

Centers for Disease Control and Prevention 

Department of Health and Human Services, 

Winter 2006

1 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

INTRODUCTION 

This handbook offers an introduction to program evaluation that has been tailored for 

staff of state and local tuberculosis (TB) programs in the United States. It is designed to 

take a person with little or no knowledge or skills in evaluation through the process with 

TB-specific examples. It assumes that the user, “you,” may be placed in a leadership 

role for an evaluation. This handbook is designed to complement the general CDC 

Introduction to Program Evaluation for Public Health Programs1

 manual (available at 

www.cdc.gov/eval). Users who desire to learn more about program evaluation are 

encouraged to refer to that manual which also includes a glossary and additional 

program evaluation resources. 

Background

The nation's TB control programs are on the frontline of controlling one of the world’s 

most deadly diseases. To date, TB programs have been very successful, as evidenced 

by a sustained downturn in TB incidence during the last decade. However, recent 

demographic and health system changes in the United States are challenging state and 

local control efforts. These include the concentration of TB in high-risk, hard-to-reach 

U.S.-born populations; the increasing immigration of persons from high-burden areas of 

the world; and changes in the organization, delivery, and financing of health care. In 

addition, all public health programs are being asked to increase services but without an 

increase in resources.2

 As a result, TB programs cannot continue to conduct business 

as usual and expect to achieve the overall goal of elimination. 

The 2000 Institute of Medicine (IOM) report, Ending Neglect: The Elimination of 

Tuberculosis in the United States,3 encourages all public health departments to 

evaluate their performance regularly. Likewise, the Division of Tuberculosis Elimination 

(DTBE) and the National TB Controllers Association (NTCA) also recognize the need for 

systematic program evaluation, as evidenced by its emerging prominence as a priority 

in policies and cooperative agreements.4

 Program evaluation offers public health 

programs a systematic, structured mechanism for improving and enhancing services 

and operations.5

 Recently, a collaborative TB Evaluation Work Group (EWG) was 

established to actively involve CDC, DTBE, state, and local partners in the promotion of 

program evaluations. The EWG actively works to build evaluation capacity within TB 

programs nationwide. To achieve this goal, EWG is developing evaluation tools and 

materials, such as this handbook, to guide TB programs in conducting self-evaluations. 

After reading this handbook and referring to the CDC manual, anyone seeking further 

evaluation guidance or technical assistance should contact EWG at TBEWG@cdc.gov. 

TB Goals and Objectives

In the broadest sense, the goal of TB programs is self-evident: to better control, and 

eventually, eliminate TB. The strategies to reach that goal are diverse. At a national 

level, the Healthy People 2010 objectives are 

• [objective 13-11] Conduct HIV testing in TB patients (aged 25–44 years). 

 2 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

• [objective 14-11] Reduce TB. 

• [objective 14-12 ] Increase the proportion of all TB patients who complete curative 

therapy within 12 months. 

• [objective 14-13] Increase the proportion of contacts and other high-risk persons 

with latent TB infection who complete a course of treatment. 

• [objective 14-14] Reduce the average time for a laboratory to confirm and report 

TB cases. 6

Also at a national level, the Government Performance Reporting Act (GPRA) goals are 

• Increase the percentage of TB patients who complete a course of curative TB 

treatment within 12 months of initiation of treatment (some patients require more 

than 12 months). 

• Increase the percentage of TB patients with initial positive cultures who also have 

drug-susceptibility results. 

• Increase the percentage of contacts of infectious (AFB smear-positive) cases 

who are placed on treatment for latent TB infection and complete a treatment 

regimen. 

At the state and local level, these broad goals can help in setting policy objectives, but 

given the wide differences in TB programs’ target populations, community risk factors, 

and organizational structures and capacities, each program must identify more tailored 

intermediate goals and objectives in order to address the unique concerns of its 

community. Often these goals are not formally articulated, but are reflected in program 

practice, resource allocation, and general program operations. For example, a TB 

program with a stated goal of increasing completion of therapy, may implicitly reach the 

goal by providing extensive services for persons in a high-risk immigrant community. 

Specific outreach, testing, and treatment strategies are implemented to achieve this 

goal. Another program may reach the same goal by focusing on specific populations. 

Part of the evaluation cycle, as you will see, necessitates that these implicit, unstated 

goals (e.g., provision of extensive services for persons in high risk immigrant 

community) be formally recognized and documented. 

Why evaluate TB prevention and control programs? 

All program resources and activities work together to accomplish a goal. Program 

evaluation is a valuable tool to help ensure that this is occurring. It is a structured 

process that helps TB staff monitor progress toward program goals, learn from both 

successes and mistakes, make modifications as needed, and judge the success of the 

program in achieving its short-term, intermediate, and long-term outcomes. Through 

program evaluation, you and others at the TB program can track changes and, with 

careful evaluation designs, assess the effectiveness and impact of a particular program, 

intervention, or strategy. Evaluating your TB program can help you: 

• Monitor progress toward national and program goals, 

• Demonstrate that a particular TB program activity is effective, 

 3 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

• Determine whether program components are producing the desired effects, 

• Permit comparisons among groups, particularly among populations with 

disproportionately high TB rates, 

• Justify the need for further funding and support, 

• Learn how to improve programs, and 

• Ensure that only effective programs are maintained and resources are not 

wasted on ineffective programs. 

It is important to assure participants that program evaluation is not a mechanism to 

evaluate individual staff members. The evaluation process focuses on operations and 

systems, not specific behaviors. 

Program evaluation and data collection for surveillance and research 

TB programs collect a large amount of data. For example, the Report of a Verified Case 

of Tuberculosis (RVCT) requires data for dozens of elements to be collected on each 

patient. The National Tuberculosis Surveillance System, for example, provides routine 

and continuous collection of individual data over time on predetermined TB factors such 

as incidence and prevalence. Surveillance data are largely standardized across

programs and, among their many uses, can play a helpful role in program evaluation, 

allowing monitoring/tracking of what a program does and how it is doing in a global 

sense. However, apart from surveillance, program evaluation aspires to answer not only 

“what” and “how,” but “why” a program is doing well or poorly. Hence, when using 

surveillance data for evaluation purposes, other quantitative and/or qualitative data are 

needed as well. Surveillance and other data sources will be discussed later. 

Most TB programs also collect and analyze data for quality assurance processes. For 

example, many programs conduct cohort reviews to assess and ensure each patient’s 

achievement of treatment objectives. Again, these data serve many purposes, and can 

be valuable sources of information that can be integrated into program evaluations. 

Finally, many TB programs participate in research projects that require data collection. 

Research and program evaluation both answer complex questions, and the line where 

research stops and evaluation starts is often blurry. Typically, research strives to 

produce generalizable knowledge and contribute to the overall science and evidence 

base. By contrast, program evaluation focuses on specific questions about specific 

programs and their beneficiaries. While the insights from evaluation may indeed be 

generalizable beyond the specific program, the primary intent is to examine one TB 

program given its specific context, opportunities, and limitations. 

 4 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Why use the CDC Evaluation Framework in TB Programs? 

As we continue to move into the future, CDC is expected to focus on achieving results 

in health improvement through performance reports and budget requests. The 

effectiveness of our programs is measured by evaluation information. To help programs 

provide credible information on program effectiveness, CDC developed a framework for 

program evaluation which describes a systemic way to collect, analyze, and 

evaluate public health actions. 

In general, most TB program managers and staff already know what works, and what 

does not, in their programs. Why should anyone take the time to use this framework? 

The answer is that all TB programs are complex - even small ones - and have many 

dimensions. This framework provides a systematic, disciplined way to ensure that you 

ask the right questions and consider the appropriate range of factors when identifying 

your problems – or your successes. It also ensures that your evaluation will provide you 

with sufficient information to enable you to use your findings to improve your program. 

The Framework is based on sound research. The process described works, and works 

well. It is also flexible and adaptable. Small programs and large programs alike can use 

it effectively. It also supports a participatory process. This type of approach is proven 

to be most effective in ensuring that evaluation results are implemented, and that the 

report does not just sit on a shelf. 

While some of these steps may seem time consuming, they need not be in every case. 

For small-scale evaluations, you may complete any given step in a few minutes or a few

hours. You may go back and forth between steps throughout the process. For example, 

you may identify correctional health staff as stakeholders in Step 1, but only develop a 

plan to engage them if you choose to focus on that component of your program. 

Applying the Standards for “Good” Evaluation

Strengths of the CDC Framework for 

Program Evaluation 

• Provides a systematic method for 

evaluation 

• Is based on sound research 

• Is flexible and adaptable

• Promotes a participatory approach 

• Focuses on using evaluation findings 

In addition to applying the CDC Evaluation Framework, program evaluators should be 

guided by a set of standards to ensure 

an effective and productive evaluation. 

The four standards guiding the 

evaluation are utility, feasibility, 

propriety, and accuracy. Examples of 

how these evaluation standards can 

be applied throughout a TB program 

evaluation will be provided at each 

step of the evaluation process. 

TB Evaluation Team

Before starting the evaluation, as with any organizational process, your program should 

designate one person to lead the effort. The role of the lead evaluator is to coordinate 

evaluation efforts, including planning, budgeting, and interfacing with partners. The lead 

evaluator should have an understanding of evaluation principles, but he or she does not 

need to be an evaluation expert. The information in this manual should be sufficient to 

lead anyone through the evaluation process. 

 5 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Good evaluations are not conducted by only one person. Program evaluations require a 

multidisciplinary team to ensure different perspectives are represented. In addition to 

the lead evaluator, other members of the evaluation team should be able to provide: 

• Knowledge of the TB program’s policies and procedures, 

• Knowledge of the program’s budget process and cycle, 

• Involvement in case management, 

• Representation of TB clients, 

• Representation of program management, and a 

• Commitment to evaluation objectives. 

The TB evaluation team should collectively represent most or all of the key stakeholders 

of the program. Team members should have dedicated time for conducting evaluation 

related activities if it is not their full-time assignment. After an evaluation team has been 

assembled, you should be ready to begin with the first step of the CDC Framework for 

Program Evaluation. 

 6 
Characteristics of a Good Evaluation Team

 Shared Vision 

Members of the team should share a common goal that drives the evaluation process. It is 

important that all team members come to consensus about what will be the focus of the 

evaluation, who will be the primary users of the evaluation, and what conduit should be 

used to report evaluation findings. 

 Diversity 

Your team should represent a cadre of disciplines and professionals at every level. A 

diverse group provides a multitude of different perspectives on how to drive the 

evaluation. This will result in a holistic evaluation that provides useful information to the 

multitude of people involved in TB control. 

 Expertise 

Team members must bring to the evaluation a certain level of knowledge about the TB 

control program that is being evaluated. Experts in program management, case 

management, finance, policy, health communications, and information technology are 

needed – in short, team members should represent every aspect of the program. 

 Defined Roles and Responsibilities 

Roles and responsibilities of each team member must be well thought out and defined. 

Moreover, each team member should be aware of other team member’s role in the 

evaluation process, actions that drive the evaluation, and processes that will ultimately be 

used to report evaluation findings. It is important that each member’s role be defined so 

that no confusion occurs within the group during a critical phase of the evaluation. 

 Communication 

Communication among team members is critical! Team members are parts of a functional 

unit. If team members do not communicate with each other about the evaluation, they can 

very well diverge from their common goal of conducting a holistic evaluation of a TB 

control program. Team members may duplicate each other’s work, may remain stagnant 

when a road block appears, or may lessen the level of their commitment if they do not 

communicate. The old adage “The left hand doesn’t know what the right hand is doing,” 

describes a team whose members fail to communicate with each other. 

 Commitment 

Members of the evaluation team should be committed to the project from beginning to 

end. At the very least they should be expected to carry out their individual tasks to 

completion. All too often, evaluation team members come aboard with high energy and 

high expectations, but as time goes by attention may wane, assignments are set aside, and 

ultimately valued members of the team may become less involved. This scenario doesn’t 

always occur, but special attention should be paid to the commitment level of your team. 

Team members should be ready to see the evaluation through completion. However, if an 

unforeseen event splits the team, plans should be in place on how to deal with the deficit 

in work power and expertise. 

 Willingness to Embrace Opportunities 

Group members should be open to learning new things and expanding their knowledge. 

Members of a good evaluation team are also willing to seek out and embrace any training 

opportunities that arise, as all evaluation is a learning process.
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 1: ENGAGE STAKEHOLDERS

Why Involve Stakeholders in TB Program Evaluation?

As noted, program evaluation is a value-laden process... but whose values? The CDC 

Framework for Program Evaluation tells you that a wide variety of stakeholders have an 

investment in TB programs and are potentially affected by an evaluation. Hence, there 

may be wide differences in their judgments about the program, what it does, what 

constitutes success, how data should be collected, and even how results should be 

distributed. To ensure that evaluations are useful and ethically sound, evaluators must 

respect the stakeholders’ values throughout the evaluation process. 

Identifying Stakeholders

So how do you find the TB program’s stakeholders? The lead evaluator of the TB 

program is usually charged with ensuring that stakeholders are involved. In this role, the 

evaluator must resist the temptation to limit the scope of stakeholders to those most 

visible, vocal, or optimistic about the program. The team of people involved in the 

evaluation should be as representative of all stakeholders’ perspectives as possible. 

Following are some examples of stakeholders who may be identified to participate in the 

evaluation process: 

Persons involved in program operations

• TB program nurses, outreach workers, clinicians, clerks, program supervisors 

• Other public health staff and managers 

• Service providers (both public health staff and in the community) 

• Staff and managers at partner/collaborating agencies 

• Coalition/advisory groups 

Persons served or affected by the program

• TB patients

• LTBI patients 

• Family, friends, co-workers of TB patients 

• Employers, unions, business leaders 

• Managers at schools, jails/prisons, nursing homes, other congregate settings 

• Hospital representatives 

• Insurance company representatives 

• Private physicians and health care providers 

• Legal system/law enforcement representatives 

• Advocacy group representatives 

• Representatives of populations disproportionately affected by TB 

• Program critics 

Intended users of evaluation findings

• TB program managers 

• Public health managers and administrators 

• Health commissioners 

• Funding agency representatives 

 8 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

• CDC representatives 

• Advocacy group representatives 

Most programs will have a large inventory of stakeholders. Once you have an initial 

listing, you may see the benefit of 

determining the most important 

stakeholders to keep engaged, 

especially if time or resources are 

scarce. The most important 

stakeholders will vary with the particular 

intervention or program. They are the 

stakeholders who can increase the 

credibility of the evaluation, implement 

the findings, or authorize or fund the 

program. 

“Priority” stakeholders: 

• Increase the credibility of the evaluation 

efforts 

• Are involved in the implementation of 

program activities 

• Will advocate for or authorize program 

changes 

• Will fund program improvements 

Stakeholder Perspectives

By definition, each of these groups has a different perspective on the TB program and 

how its resources should be channeled. For example, program administrators may hold 

a skeptical view of the use of funds for enablers and incentives while outreach staff may 

see such funds as insufficient. As an evaluator, you should be aware that these different 

perspectives will influence how the evaluation and its findings are ultimately viewed. 

Anticipating these differences will help you address them appropriately; however, you 

should be careful not to make too many assumptions about people’s perspectives. 

Talking to persons from each of these groups will help you understand their views and 

concerns about the program. For example, in one TB program, a manager assumed 

that staff would be overburdened by adding a new initiative. The staff members, on the 

other hand, recognized the value of the initiative to their patients, and thus were less 

concerned about extra work. Further, some staff persons felt the new tasks would help 

them develop professionally. 

It is especially important to address the concerns of the “critics.” While it is hard to 

imagine people being skeptical of the need to control TB, it is important to recognize 

that TB programs are competing for resources with other important programs and 

services. Even within a TB program, people may have different ideas about what 

priorities deserve the limited resources. The evaluation needs to justify the value of TB 

services to those critics who would use resources elsewhere. 

 9 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Engaging Stakeholders throughout the Evaluation 

Not all stakeholders need to be engaged throughout the evaluation or engaged in the 

same way. For many stakeholders, such as health department administrators, it may 

only be necessary to inform them that the evaluation is occurring. Other stakeholders 

such as program managers, supervisors, and client representatives will want to be 

involved in defining evaluation questions and using results to modify services. Nurses, 

outreach workers, and other staff may help collect data. Others can be involved in 

advocating 

for the 

evaluation 

and for the 

program 

itself. You 

must 

strategicall

y involve 

key 

stakeholde

rs 

throughout 

the 

process to 

ensure that 

the 

ultimate 

users will 

be satisfied 

with the 

results of 

the 

evaluation 

and will use the findings to enhance their program. 

How to Gain True Buy-in from Stakeholders

• Taking Ownership – Involvement of stakeholders in the evaluation increases the 

likelihood of buy-in, use of evaluation findings, and programmatic changes that result 

from evaluation findings.

• Group Persuading - Use group members who are already “on-board” with evaluation 

efforts to persuade dissenting members to participate in and support efforts. 

• Infusing - Determine what is most important to the stakeholders, and if it does not 

conflict with other stakeholder views dramatically, infuse elements of these views into 

the evaluation. To do this effectively, understand their position on program and 

evaluation efforts: are they over-the-top optimists, realists, pessimists, or antagonists? 

• Using inherent talents - Determine what contributions the stakeholders can make in

evaluation efforts, engage the stakeholders, and then make them aware of how greatly 

they can influence the evaluation by participating. 

• Determining your role as an evaluator: Leader or Facilitator- Know whether or not 

to assume the position of a leader or a facilitator. A leader manages and controls, 

while a facilitator provides services and aids the stakeholders in conducting the 

evaluation. You can determine what role to assume by the amount of evaluation 

capacity built into the program and the environment in which the program functions.

Application of Standards for “Good” Evaluation - Step 1

Utility: Who will use the results to improve or enhance the TB program?

Feasibility: How much time and effort should be devoted to stakeholder engagement? 

Propriety: To be ethical, which stakeholders need to be consulted (i.e., do we need to 

engage patient representatives, advocates, community liaisons)? 

Accuracy: How broadly do we need to engage stakeholders to paint an accurate picture 

of this TB program and its context?

 10 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 2: DESCRIBE THE PROGRAM 

Purpose and Elements of Program Description

The second step in the CDC evaluation framework is to describe the program. Everyone 

agrees that the mission of a TB program is to control and eventually eliminate 

tuberculosis. In TB, perhaps more than in other public health programs, it may appear 

that everyone knows what happens and why, since many TB control activities have 

been in place for decades. However, based on their role or position, different 

stakeholders have different perspectives on the program’s objectives, resources, 

activities, and timelines. As your evaluation team goes through the process of 

describing your program, you may discover unfounded assumptions or gaps in your 

planning that could, if unexamined, impede success. 

The scope of your program description should reflect the system’s current standards of 

care and public health practice. It is tempting to only describe the program’s “known” 

problem areas. The evaluation process specifically encourages you not to do this. First, 

because of the increasing complexity of the TB service provision environment and the 

increasing need to coordinate services with providers such as health maintenance 

organizations or other specialized service providers (e.g., HIV, methadone, or migrant 

health clinics), it is important to examine how the entire program is interconnected to 

fully understand internal and external factors. Second, describing program areas that 

are working may give insight into factors that need to be examined in an area that is not. 

For example, a program may discuss an appointment system in the description of its 

successful clinical scheduling system, but mention none for its under-performing fieldbased activities. 

Having the evaluation team involved in program description ensures that you all have a 

shared understanding of the program’s activities and intended effects and a recognition 

of how your program fits into the larger context within which you operate. By working 

through the process of describing your program, your evaluation team should uncover 

and resolve any discrepancies in order to reach a common understanding of your 

activities and their effects. A clear description of your program will also provide a 

common foundation for the evaluation. If a strong strategic plan already exists for a TB 

program, as is often the case in TB, the task of program description is made simpler, 

since the goals and objectives have already been articulated. At a minimum, the 

program description should address: 

• The specific needs for TB program services in the community, 

• The target audience of TB program services, 

• The context in which the program operates, 

• The objectives of the program, 

• The program’s stage of development, 

• The program’s resources/inputs, 

• All of its activities, and 

• The intended results (outputs and outcomes) of the program. 

Note that while the description needs to address the above areas, the level of detail 

does not need to be extensive. Progress reports and existing documents will provide 

 11 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

much of the needed information, supplemented by the information known by the 

members of the evaluation team. 

The Need

Your description of the need for your program should include an analysis of the 

magnitude of TB-related morbidity and mortality in various segments of the population in 

your program area. Many of the data needed for these calculations are already routinely 

collected by TB control programs through the Report of a Verified Case of Tuberculosis 

(RVCT) and Aggregated Report for Program Evaluation (ARPE) forms. The information 

gathered on these data collection instruments can be stratified to identify TB-related 

health disparities among specific population segments or communities, and also to 

determine trends in local epidemiology. In addition to the data garnered from the RVCT 

and ARPE, other national and state health data collected by the Census Bureau, 

national and state surveys, regional or community surveys, case studies, cohort 

reviews, expert panels, and similar sources may also be included in the description of 

need. However, these data will only provide a partial picture. Ideally, you should use 

state or regional data in combination with national data to describe the need for the TB 

program. 

Target audience

The target audience is the group(s) that your program tries to reach in addressing TB 

concerns. For example, for some communities where TB rates are high, your program 

may have chosen a subset of the population on which to focus prevention efforts. To 

reach these groups, you may need to work with other intermediaries or partners. For 

example, to reach homeless persons, you will need to work with shelter directors; to 

reach inmates, you will need to work with prison health staff. 

Context

The context for TB programs varies widely. As part of the evaluation process, you need 

to understand how “environmental” factors affect your program’s operations. These 

environmental factors include how the program is administered, how TB services are 

delivered, and how the TB program “fits” with other health and social services in the 

community determine what a program can achieve and how it can change. Your 

description needs to address all of these fundamental questions. 

Objectives

Objectives define what a program intends to do. They should clearly define what 

changes will occur after program activities have been implemented. Although they may 

not be formally written, all programs have underlying objectives. If objectives have not 

been written, you may find that different stakeholders have different assumptions about 

the program’s intentions. For an evaluation to be successful, it is essential to take the 

effort to make the objectives explicit and SMART (specific, measurable, achievable, 

relevant, and time-bound), so that they can be assessed. Objectives define the 

intended outcomes of your efforts. They will also help you in Step 4 when you are 

defining indicators and benchmarks of success. 

 12 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

A tool to help write SMART objectives is given below: Objective 

Increase percentage of adult patients with TB who complete therapy (within 12 months) 

from 80% to 90% by 2008.

VERB METRIC POPULATION OBJECT

BASELINE

MEASURE 

GOAL 

MEASURE TIMEFRAME 

Breakdown 

Increase Percent

Adult patients 

with nonresistant TB

Completion 

of therapy 

(w/in 12 mos)

80% 90% By 2008

SMART Objective

In order to be most effective, objectives should be clear and leave no room for interpretation. 

S-M-A-R-T is a helpful acronym for developing objectives that are specific, measurable, 

achievable, relevant, and time-bound. 

An example of a SMART objective for TB is as follows: 

In County X, increase the percentage of adult patients with TB who complete treatment in 

less than 12 months (as measured by cohort review) from 80% to 90% (the national goal) by 

2008. 

The objective is specific because it identifies a defined event: adult TB patients will 

complete treatment in less than 12 months. The objective is measurable because it 

specifies a baseline value and the quantity of change the intervention is designed to 

achieve: from 80% to 90%. As in the example, it is worthwhile to note whether there is an 

existing data source for the objective. The objective is achievable because it is realistic 

given the 10-year time frame. The objective is also relevant because it relates to the

elimination of exposure to TB. Finally, the objective is time-bound because it provides a 

specified time frame by which the objective will be achieved (from 2004 to 2008). 

Stage of Development

You will also need to assess the developmental stage of the different components of 

your program. Although the overall TB program may have a long history, specific 

services and activities are likely to be at different stages of program development. If the 

scope, duration, or frequency of an activity is going to change in the near future, you will 

want to note that in your program description, since the stage of development influences 

how ambitious you can be in expectations of program success. A changing or 

transitioning program development stage is especially important in TB, since any 

outcomes related to completion of treatment or prevention of relapse require sufficient 

time.

 13 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Resources or Inputs

The description of your TB program should also include the program inputs or resources 

that enable you to carry out activities. In describing the program’s resources, you need 

to clearly articulate all human and other resources invested in TB control. 

In TB, typical resources include: 

• TB program staff (including supervisors), 

• Training and continuing education for staff, 

• Funds (from multiple sources, including CDC, Medicaid, private insurance), 

• Resources associated with TB clinics (space, medicines, x-ray facilities, etc.), 

• Resources associated with TB laboratories, 

• Health department infrastructure resources (administrative resources, etc.), 

• Data systems, and 

• Program policies and procedures, and laws regarding TB control. 

In addition to the direct resources used by the TB program itself, inputs should also 

include the resources tapped from partnerships with other health care providers, such 

as hospitals and private clinics. For example, HIV, alcohol, and drug treatment 

programs often provide resources to TB programs. School, correctional, immigrant 

health, and homeless assistance programs may also have resources to include as 

inputs. Similarly, local advocacy groups such as the American Lung Association may be 

existing or potential resources for your program. In this way, describing how the TB 

program “fits” with other health departments and public and private services in the 

community is essential; these interconnections often enhance or limit a program’s ability 

to implement changes. 

Program Activities

Outlining a TB program’s activities is perhaps the most intensive stage of developing a 

program description. Some starting points for describing TB activities might include 

some of these core components:

• Conducting overall planning and development of policy; 

• Identifying persons who have clinically active TB; 

• Managing the assessment, problem identification, planning, implementation, and 

evaluation of the patient’s psychosocial factors related to TB diagnosis; 

• Identifying and managing persons infected with M. tuberculosis;

• Providing laboratory and diagnostic services; 

• Collecting and analyzing data ; or 

• Providing training and education to patients, providers, and the community 

Although you may find that your evaluation focuses only on some of these activities, 

 14 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

understanding the entirety of TB-related activities in your community is important. As 

noted above, these program pieces are interrelated, and changes to one activity will 

likely impact all. 

Outputs. As the examples in Table 1 show, outputs are tangible materials, services, 

and capacities that are a product of your program’s activities. However, they do not 

sufficiently provide indications of a program’s success in creating changes that affect 

the TB burden in a community. They are means to ends. 

 15 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Table 1 

Activities Outputs

Planning and policy development 

• Strategic plan for TB 

• Policy manual 

Finding TB cases: testing • Placed and read skin tests 

• X-rays performed 

• Sputa collected and tested 

Managing TB cases: assessment, treatment, 

and case management 

• Case managers assigned to cases 

• Assessments completed

• Treatment plan written 

• Case management contracts developed 

Finding and managing persons with LTBI: 

testing, treatment, management 

• Contacts/high-risk persons identified

• Placed and read skin tests 

• Treatment plans written

• Follow-up provided 

Providing laboratory and diagnostic services • Lab tests done 

• Reports sent

Collecting and analyzing data • Forms completed 

• Reports run 

Providing training and education • Provider training materials developed 

• Patients (cases and LTBI) education 

provided 

Outcomes. Outcomes are the changes in persons or the community that result from TB 

programs activities. For most programs, evaluators distinguish outcomes into 

categories: short term/intermediate outcomes, and long term outcomes. Program 

objectives define many of the intended outcomes. 

• Short term or intermediate outcomes program results that show that 

changes have occurred because of a program, but are not the long term 

results intended by the program. For example, changes in patient or provider 

knowledge, attitudes, beliefs, or behavior can all be intermediate outcomes. 

Like markers along the way, these outcomes can show if the program is 

headed in the right direction. For example, an increase in patient adherence 

to treatment is an intermediate outcome that shows progress toward 

completion of treatment. 

• Long-term outcomes are more distal changes in organizations, communities, or 

systems that occur as a result of your activities. Examples of long term 

 16 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

outcomes are:

 Decrease morbidity and mortality among foreign-born persons with multidrugs resistant TB, and 

 TB in the US-born African-American community in a given county will be 

eliminated. 

Graphic Depictions of Program Description: Logic Models

A logic model takes all the information in the program description above and organizes 

it into a graphic depiction—a picture. 

A logic model links the 

aforementioned inputs, activities, 

outputs, outcomes, and impacts of 

your program as a series of if-then 

statements. If certain inputs are 

provided, then specific activities can 

be performed. If those activities are 

performed, then outputs will result. If

outputs are produced, then shortterm outcomes will be achieved. If

short-term outcomes are achieved, 

then long-term outcomes will be

realized. 

Why draw a logic model? 

• Provides a sense of scope – what are the 

program’s components? How are they 

interconnected? 

• Serves as a “map” to help ensure that

systematic decisions are made about what is to 

be measured in the evaluation process and that 

gaps in information do not occur 

• Organizes indicators and ensures that none are 

overlooked 

• Visually communicates why indicators and tools 

matter in the overall scheme of TB programs' 

efforts to achieve outcomes 

It may help to keep the following in mind as you develop your logic model: 

• To plan a new program, start the logic model by listing the outcomes you want to 

achieve. 

• To evaluate an existing program, start with activities you are doing. 

• Use boxes or arrows as necessary to fully describe the program. 

• Remember, there is no right or wrong logic model. Focus on developing one that 

is clear to the stakeholders. 

• Be ready to do several versions and revisions in the process of developing a 

logic model that reflects the shared understanding of the program’s intended 

purpose. 

A key reason to use logic models is to show the interconnection among program 

resources, activities, and outcomes. For example, a program had incomplete data from 

sputa test results. Routine practice assumed the specimen collection process was 

problematic. However, in the process of developing a comprehensive logic model it 

became clear that laboratory activities also contributed to the incomplete data. Revising 

the contracting process with the lab provided an effective solution.

Since logic models can exist at any level of generality, you are encouraged to represent 

 17 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

a complex TB program by using a 

global logic model that provides an 

overview of the program and more 

specific logic models that highlight 

areas of focus. This is what was done 

by the Evaluation Working Group 

(EWG) based in the Division of 

Tuberculosis Elimination at CDC. The 

EWG developed a meta-logic model for 

TB elimination that represented the 

intended relationships between five areas of concentration and the goal, TB elimination. 

They then developed more specific logic models that “zoomed in” on each of the five 

focus areas – capacity and infrastructure, evaluation capacity, timely completion of 

therapy, contact investigation, and prevention of TB in high-risk populations. The EWG 

logic models are included in Appendix A. However, logic models don’t have to be 

complex graphics. Tables or hand-drawn sketches often serve the purpose well. 

How to use the examples of TB logic models

Logic models developed for high-priority TB 

program activities are available in the appendix. 

Review these models to see how the “if-then” 

sequence from inputs to activities to outcomes 

works in a theoretical program. Then decide if 

you can adapt these models to describe your 

own program. 

Application of Standards for “Good” Evaluation - Step 2

Utility: Is the level of detail with which the TB program is described appropriate for the use of 

this evaluation? Is the program description intelligible to those who need to use it to make 

evaluation planning decisions? 

Feasibility: Does the program description show a feasible link between resources, staff 

activities, and patient- /program-level outcomes? 

Propriety: Is the description complete and fair in assessing all aspects of the TB program? 

Does the program description include enough detail to examine both strengths and 

weaknesses, and unintended as well as intended outcomes? 

Accuracy: Is the program description comprehensive? Have you documented the context of 

the TB program to ensure that likely influences, both internal and external, can be identified?

 18 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 3: FOCUS THE EVALUATION DESIGN 

Getting Started

A major payoff for investing time in a complete description and logic model(s) of your TB 

program is that it makes it easier to identify what to address in the evaluation. First, the 

logic model will help you decide the right questions to ask about the program. Every box 

in the logic model can inspire questions: Was this resource available? Did this activity 

occur as planned? Did this outcome occur as intended? Likewise, the arrows may be a 

focus: Did the output produce the intended outcome? Did early outcomes progress to 

more long-term outcomes? The key is to apply a disciplined process for deciding which 

boxes and arrows should be part of a specific evaluation. 

There are different ways to focus an evaluation. Asking questions about inputs, 

activities, and outputs is often referred to as process evaluation, and asking questions 

about changes that have occurred or effects that have taken place as the result of the 

TB program – the short- and long-term outcomes – is referred to as outcome 

evaluation. 

RESOURCES ACTIVITIES OUTPUTS 

Types of Evaluation: Focus in TB Control

Process evaluation 

For established programs, process evaluations help program stakeholders understand 

why the programs are achieving the results they are, and serve to complement outcome 

evaluations. This information specifies what needs to be done in order to enhance the 

program: either improve implementation or change the program. For example, a TB 

program operates a DOT clinic in a community center. After performing a process 

evaluation, the evaluators find that many TB patients are not attending the clinic. Follow 

up interviews with a few patients and providers show that owing to changes in the public 

transportation system, the center is not easily accessible to most of the TB patients. 

Program managers rethink the plan and decide to relocate the clinic. 

For new programs, process evaluation helps staff find and correct problems before they 

can affect the program. For example, a TB program begins a targeted education 

campaign to increase the number of private providers who are aware of TB. The 

campaign utilized a brochure that provides a web address for more information. By 

evaluating the process, TB program managers found that only a few of the targeted 

Are there 

adequate 

resources and 

qualified staff? 

Did the activity

occur as 

planned? 

What parts of the 

program 

contribute to 

outcome 

achievement?

Did the 

activities 

produce the 

product? 

Affect the 

quality of the 

product? 

OUTCOMES

Did this outcome occur as 

intended? Did the output 

produce the 

intended

outcome?

Did our early outcomes 

progress to our more distal 

outcomes and impact? 

 19 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

providers were accessing the website. Interviews with a small number of providers 

determined that the instructions for the brochure distribution were unclear and the web 

address was difficult to access. A follow-up letter, clearly explaining the process and 

providing a simpler web address, was drafted and sent to the private providers. As a 

result, website usage increased. 

Even when outcomes are being achieved, process evaluations can be a valuable way to 

assess the efficiency of a program. Many process evaluations have shown that certain 

activities are duplications of other work, and that other once-useful activities are no 

longer productive. For example, a TB program may find that both the nurse and the 

DOT worker are recording the same patient information in different files. A simple 

system to update computer records reduces the redundancy and improves access to 

the most current information. 

Outcome evaluation 

Outcome evaluations show whether or not a program achieves the desired effects. You 

may ask, “Did our program activities produce the changes we wanted? “ 

 

While some evaluations may have only one focus, i.e., either processes or outcomes, 

program evaluation that is intended to yield the most insights for program improvement 

must consider both. For example, a program manager may find that the rate of 

completion of treatment increased by more than a third after the implementation of a 

universal DOT program. However, this statistic provides only a limited amount of 

information. Could there be another explanation for the increased rate (i.e., a change in 

reference population)? Is an increase of at least 33% sufficient, or could the rate be 

even better? Could the DOT program work better? Combining process and outcome 

evaluation can help answer these questions. 

Logic models help in this scenario also. The logic model is your best tool for “tracing 

back” the factors that contribute to good or poor performance on an outcome. For 

example, if the percentage of patients completing treatment is less than what is set in 

the objective, the logic model shows what parts of the program were supposed to 

produce that result, in this case, patient tracking and support efforts. Process evaluation 

of those components will help reveal whether the current level and quality of these 

activities are sufficient or whether something more effective will need to be 

implemented. 

Likewise, when outcomes are being met, knowing what works facilitates the education 

of new staff and the sharing of successes with other programs. For example, a TB 

program developed a procedure for conducting contact investigations in congregate 

settings. The evaluation served to document how the program worked and to share the 

process with new staff, thus ensuring that the successes were sustained. The 

evaluation provided a “best practice” that could be shared with other TB programs. 

Choosing an evaluation focus and defining evaluation questions 

Seldom are entire TB programs evaluated. For every program, there are thousands of 

 20 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

possible evaluation questions that can be asked. However, no evaluation team has the 

resources to answer all of them. Recognizing that you have limited resources for 

evaluation, you must prioritize your questions and decide which parts of the program to 

address. As the evaluator, your role should be to guide the narrowing process to ensure 

the information produced will be usable. Stakeholders can help prioritize areas of 

concern, and the logic model can serve as a map to help you understand how the 

program elements fit together. The logic model can also help you determine the time 

span (short-term, mid-term, and long-term) over which outcomes should be measured. 

For example, while we in TB may want all of our programs to aspire to the goal of 

elimination, we cannot expect any single program to achieve this alone. Hence, it is 

better to judge a single program on more short-term and localized outcomes. 

There is no single “right” evaluation focus. The purpose of the evaluation and the 

intended use of the findings should drive what is asked. For this reason, evaluations 

looking at the same general program area will focus on different questions. For 

example, two different programs may choose to evaluate their case management 

processes. One county program planned to hire and train new staff, so they needed to 

know what case managers do effectively. They chose a process focus that examined 

activities and outputs achieved. Another program needed to justify the funding for case 

managers to county administrators. Their evaluation focused on how case management 

activities lead to positive patient and cost outcomes such as reduction of lengths of 

hospital stays and timely completion of therapy. 

Evaluation Designs 

Although program evaluation questions are geared to answering specific questions for 

specific programs, the designs for answering them can vary. Some designs resemble 

research designs, while others are less formal. However, it is important to remember 

that the purpose of evaluation is to improve programs, not to publish generalizable 

findings, and therefore, you need only collect data sufficient to answer your evaluation 

questions. For example, if you wanted to know if an educational message or brochure 

influenced patient treatment adherence, you could easily measure this with a traditional 

research design, randomly selecting exposure and control groups, then tracking 

outcomes. Where random selection of participants is not possible, quasi-experimental 

designs, such as implementing an intervention in one clinic but not in another, can be 

used. However, since an evaluation is not performed to extrapolate findings to other 

programs, you should collect only the data that you need to answer the questions

at hand. 

The evaluation standards are helpful in selecting your design. You will want to select a 

design that provides accurate information, but is at a level of detail that is most useful 

for the intended users of the evaluation. The design must also be feasible given the 

program’s operations and the resources available for the evaluation. 

Traditional research designs are often hard to implement in field settings. Fortunately, 

other types of designs may be equally appropriate and more easily employed. The time 

series design is a popular strategy. For example, if you want to test the effectiveness of 

new contact investigation interview techniques, you can assess indicators such as 

 21 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

number of contacts identified before and after the program’s implementation. 

Alternatively, you may only be able to measure what happens after a program change 

has occurred. In such a situation, you can use the logic model to track short and 

intermediate outcomes and compare the actual results of the program to those 

expected when the intervention was planned. Although this evidence may not be 

irrefutable, such information is generally sufficient to know how well your program is 

working. 

Design considerations 

The audience. Utility of the evaluation finding is what drives the evaluation. How and by

whom will the information will be used should be the primary concerns when selecting 

an evaluation design. It is essential that the intended users believe that the information 

generated by the evaluation is credible. When selecting a design, be sure to explain the 

types of results that can be expected. 

The resources. Feasibility is another key concern for selecting an evaluation design, 

and often impacts choices. A design that would severely disrupt normal clinic operations

should not be selected. Further, it is often not feasible to implement a program that 

serves only a subset of eligible persons, and it may be unethical to randomly assign 

patients into control groups. 

Time. Time is a precious resource that must be considered when focusing an 

evaluation. Brilliant evaluation findings that come after the decisions have been made 

will be of little use. 

Application of Standards for “Good” Evaluation – Step 3

Utility: Does the focus match the intended use of the evaluation? What do the key 

stakeholders need from the evaluation to improve or enhance the TB program? 

Feasibility: What is the stage of development of the program? How intensive is it? What are 

the relevant resources and logistical considerations? 

Propriety: Are staff and patients’ rights protected by this design? 

Accuracy: Does the evaluation design provide a sufficient level of evidence that the users 

will expect and feel is credible? 

 22 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 4: GATHER CREDIBLE EVIDENCE 

Now that you have the evaluation questions and focus defined, you need to determine 

how best to collect the data. There are two related steps: expressing the general 

concepts in the evaluation questions as more tangible indicators, and then defining data 

sources for the indicators. 

What are indicators? 

Indicators are visible signs of whether or not a program is achieving the expected 

outcomes or progressing in the intended direction. They are usually measurable factors 

(e.g., numbers or percentages) that can be tracked to determine if there is an increase 

or decrease.7

 A good indicator is relevant, understandable, and useable. Indicators are 

commonly used to describe, monitor, set goals, advocate, and evaluate, and to provide 

information to agencies and organizations accountable for processes and outcomes. 

As noted earlier in Step 2, if the program staff has already developed objectives in the 

SMART format, then they have a jump start on indicator development. SMART 

objectives are by definition measurable, and generally the indicator is self-evident. 

Remember the SMART example cited in Step 2: 

Increase the percentage of adult patients with TB who complete treatment 

in less than 12 months (as measured by cohort review) from 80% to 90% 

(the national goal) by 2008. 

Since the objective was written to be measurable, the easily recognizable indicator is 

the percentage of adult patients with TB who complete treatment within 12 months. In 

this as in many cases, there may also be an existing data source (RVCT, ARPE, etc.) 

from which to collect this data. More will be covered about this in a later section. 

A tool to create indicators from SMART objectives follows. 

 23 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook Objective 

Increase percentage of adult patients with TB who completed therapy (within 12 months) 

from 80% to 90% by 2008.

VERB METRIC POPULATION OBJECT

BASELINE

MEASURE 

GOAL 

MEASURE TIMEFRAME 

Breakdown 

Increase Percent Adult patients 

with TB 

Completion 

of therapy 

(w/in 12 mos)

80% 90% By 2008

VERB METRIC POPULATION OBJECT

BASELINE

MEASURE 

GOAL 

MEASURE TIMEFRAME

Breakdown 

Increase Percent Adult patients 

with TB 

Completion 

of therapy 

(w/in 12 mos)

80% 90% By 2008

Indicator 

Percent of adult patients with TB who completed therapy (within 12 months) in 

2008 

Even when your evaluation questions do not draw on existing program objectives, you 

may use a similar process for developing objectives and indicators. The key is to 

determine an observable manifestation of the concept, or, if there is no directly 

observable indicator for a specific question, some proxy, or alternate, measure. For 

example, assessing the number of skin tests placed is easy and observable because 

you can count them. On the other hand, “patient trust in his/her health care provider” is 

important, but vague. You cannot “see” trust, but you can observe or measure its 

evidences, such as “patient reports personal information to the provider” or “patient 

states feeling of trust and confidence in the provider” and use it as a proxy measure. 

 

Below are examples of some indicators that have been used for evaluating key 

components of TB programs. It is unlikely that any single evaluation will require all of 

them at the same time, but over the life of your program all the indicators on this list 

might prove useful: 

 24 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Components Indicators 

Timely reporting • Proportion of verified TB cases reported to the local health 

jurisdiction within 1 working day from treatment start date. 

Program capacity • Demonstrated ability (i.e., organization, staffing, resources, 

and facilities) to carry out the core components of a TB 

control program. 

TB case rate • Number of TB cases identified per 100,000 people. 

Complete reporting • Proportion of cases with complete data on key variables 

(i.e., homelessness, injecting drug use, non-injecting drug 

use, excess alcohol use) 

Culture identification • Proportion of pulmonary or laryngeal TB patients > 12 years 

of age with sputum culture obtained. 

Recommended initial 

therapy 

• Proportion of TB patients started on the recommended 4-

drug regimen. 

Timely treatment • Proportion of sputum smear-positive pulmonary or laryngeal 

TB patients initiating treatment in < 7 days of specimen 

collection. 

Culture conversion • Proportion of sputum culture-positive TB patients with 

documented conversion to sputum culture-negative within 

90 days of initiation of treatment. 

Appropriate directly 

observed therapy (DOT) 

• Proportion of TB patients for whom DOT is recommended 

who receive DOT throughout the course of treatment. 

Inappropriate selfadministered therapy (SAT) 

• Proportion of TB patients for whom DOT is recommended 

who receive inappropriate SAT throughout the course of 

treatment. 

Timely completion of 

therapy 

• Proportion of TB patients who complete treatment in < 12 

months. 

Default from treatment • Proportion of TB patients who default prior to completing 

treatment. 

Contact identification • Proportion of sputum smear-positive cases with at least one 

contact identified.

Contact evaluation • Proportion of identified contacts to smear-positive cases 

who complete evaluation for TB infection or disease. 

Contact treatment initiation • Proportion of infected contacts to pulmonary cases who 

started treatment for LTBI. 

Contact treatment 

completion 

• Proportion of infected contacts to pulmonary cases who 

have started on treatment for LTBI complete treatment. 

Pediatric TB cases • Number of TB cases in children 0-4 years old.

TB deaths • Number of persons who die of TB. 

 25 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Data Collection

As noted earlier, while the list of indicators may appear daunting, TB programs routinely 

collect huge amounts of data, and many of these can provide information for key 

evaluation indicators. Data collected for the RVCT comprise a rich source of readily 

available information. Aggregated, these data can answer a wide variety of evaluation 

questions, especially regarding patient treatment outcomes. Similarly, the information 

collected for the ARPE is a great source for data related to contacts or targeted testing 

efforts. 

While existing data sources like the 

RVCT and ARPE provide answers to 

some questions, you may need to 

gather other sources of information to 

supplement them. First consider 

sources you already have such as 

charts, records, and policies. You 

also have people who can provide 

information. The typical way to gain 

information from documents is review 

and abstraction. Some typical data 

collection methods for obtaining 

information from people include 

surveys, interviews, focus groups, 

document abstraction, and 

observation. 

The following table provides an overview of the major methods used for collecting data 

during evaluations. 8

Method Overall Purpose Advantages Challenges

Questionnaires, 

Surveys, 

Checklists

Quickly and/or easily 

get lots of information 

from people in a non 

threatening way 

• Can complete 

anonymously

Inexpensive to 

administer 

• Easy to compare and 

analyze 

• Administer to many 

people 

• Can get lots of data 

• Many sample 

questionnaires 

already exist 

• Might not get careful 

feedback 

• Wording can bias 

client's responses 

• Are impersonal 

• In surveys, may

need sampling 

expert 

• Doesn't get full story 

Data Sources 

• Medical/clinical charts 

• Clinic policies/procedure documents 

• Clinic records (sign in/out sheets, supply 

inventories, personnel documents) 

• Contact investigation forms 

• DOT logs 

• Training records (attendance, agendas) and 

materials (curricula, pretests and posttests, 

evaluation forms) 

• Patient education materials 

• Staff members

• Patients 

• Other health care providers interacting with 

program 

 26 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Interviews Fully understand

someone's impressions 

or experiences, or 

learn more about their 

answers to 

questionnaires 

• Get full range and 

depth of information 

• Develops relationship 

with client 

• Can be flexible with 

client 

• Can take much time 

• Can be hard to 

analyze and 

compare 

• Can be costly 

• Interviewer can bias 

client's responses 

Documentation 

review 

Get an impression of 

how program operates 

without interrupting the 

program; is from 

review of applications, 

finances, memos, 

minutes, etc. 

• Get comprehensive 

and historical 

information 

• Doesn't interrupt 

program or client's 

routine in program 

• Information already 

exists 

• Few biases about 

information 

• Often takes much 

time 

• Info may be 

incomplete 

• Need to be quite 

clear about what 

looking for 

• Not flexible means 

to get data 

• Data restricted to 

what already exists 

Observation Gather accurate

information about how 

a program actually 

operates, particularly 

about processes 

• View operations of a 

program as they are 

actually occurring 

• Can adapt to events 

as they occur 

• Can be difficult to 

interpret seen 

behaviors 

• Can be complex to 

categorize 

observations 

• Can influence 

behaviors of 

program participants

• Can be expensive 

Focus Groups Explore a topic in 

depth through group 

discussion, e.g., about 

reactions to an 

experience or 

suggestion, 

understanding 

common complaints, 

etc.; useful in 

evaluation and 

marketing 

• Quickly and reliably 

get common

impressions 

• Can be efficient way 

to get much range 

and depth of

information in short 

time 

• Can convey key 

information about 

programs 

• Can be hard to 

analyze responses 

• Need good facilitator 

for safety and 

closure 

• Difficult to schedule 

6-8 people together 

Case Studies Fully understand or 

depict client's

experiences in a 

program, and conduct 

comprehensive 

examination through 

cross comparison of 

cases 

• Fully depicts client's 

experience in 

program input, 

process and results 

• Powerful means to 

portray program to 

outsiders 

• Usually quite time 

consuming to

collect, organize and 

describe 

• Represents depth of 

information, rather 

than breadth 

 27 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Choosing the “right” data source, as noted in the main text, is situation-specific. Like 

everything else in the utilization-focused evaluation approach, the choice of data 

collection method reflects the time, resources, intended users, and use of the data. 

Typically, the validity and reliability of the data source are important, attention to these 

factors is always time well spent. When the evaluation question is about an abstract 

concept or reliable and valid data sources do not exist, a useful technique you can apply 

is “triangulation” of data sources (i.e., using multiple data sources to assess the same 

concept). For example, to obtain data on HIV status data it is useful to collect 

information from both RVCT and from the patient’s medical record. Both sources of 

data are important to review because sometimes the results of recent tests are in the 

chart but not recorded on the RVCT. Any data source can be biased for any number of 

reasons. Using multiple sources reduces the influence of these biases on information. 

Combining data gathered through different methods - interviews, observations, and 

patient records - will provide a much more accurate information. To obtain more 

information on data collection methods and how to use them, please refer to 

http://www.cdc.gov/eval/resources.htm#tools. 

Application of Standards for “Good” Evaluation – Step 4

Utility: Will the data collection methods you have selected provide sufficient information to 

enable the TB program to be changed in a meaningful way? 

Feasibility: Are the methods selected practical, and can they be implemented without 

upsetting program routine in a major way? 

Propriety: Are the methods selected ethically sound, and are safeguards built in to protect 

confidentiality? 

Accuracy: Are the methods sufficient to provide accurate, unbiased information?

 28 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 5: JUSTIFY CONCLUSIONS 

At this point you have a wealth of raw data. In this step, you will analyze it, draw 

conclusions, and justify those conclusions. Many of your earlier efforts in engaging 

stakeholders and choosing the right evaluation questions will provide us with good 

returns in this step. 

 

Analyzing data 

Assess the data using methods appropriate for the type of data collected. For RVCT 

and ARPE data, there are formulas you can use to calculate information that can be 

compared with national statistics. For other quantitative data, there are statistics, 

frequencies, and simple counts you can use. For qualitative data, techniques like 

content analysis can be employed. Both quantitative and qualitative data are useful in 

program evaluation. Any basic text or web search on research and analysis can 

provide sufficient instruction on how to conduct data analysis. Several of these can be 

found at http://www.cdc.gov/eval/resources.htm#tools. 

Drawing Conclusions

Analyzing the data involves looking at what the data mean in addition to what they say. 

To draw conclusions, you need benchmarks or standards to which to compare the 

results in order to know whether the program was successful, unsuccessful, or both. 

The benchmarks are informed by the values and preferences brought by the 

stakeholders. For example, an objective of City X’s TB program was to reduce the TB 

case rate by 9% over a 3-year period. At the end of 3 years, the program found that the 

TB case rate had exceeded expectations and declined by 10%. However, further 

investigation showed that during that same period, the TB case rate in the AfricanAmerican community rose 15%. Was the program successful in meeting its objective? 

That depends on the standard for success being brought to bear. At this point the 

evaluation team needs to decide on the benchmark to which it will compare these 

findings in order to determine if their efforts were successful. Each of the stakeholders 

should help determine the benchmarks in order to be sure their needs for the evaluation 

are met. In the example above, the TB program management may want to declare the 

efforts successful, especially in front of an audience of funders. Alternatively, another 

stakeholder representing the African-American community may declare that the 

objective was not met because a 9% reduction was not achieved in all demographics. 

Justifying Conclusions

This example demonstrates the need for stakeholders to reach consensus on the 

values by which they will judge the evaluation findings. From Step 1, you have already 

determined the different values and preferences stakeholders will be bringing to their 

judgments of the program, instead of waiting until it is time to interpret the findings. 

Once consensus on the benchmarks has been reached, the evaluation team will not 

only be able to draw conclusions, but will also be able to justify them to others. 

In justifying the conclusions of the evaluation, you must ensure that the results are both 

accurate and useful for them to be of maximum value. The evaluation team should be 

able to justify the conclusions and the processes used to reach the conclusions in order 

to demonstrate accuracy. Similarly, the evaluation rationale, methods, and conclusions 

 29 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

should be communicated in a way that is accessible and useful to the intended 

audience. 

Application of Standards for “Good” Evaluation – Step 5

Utility: Do the stakeholders judge the conclusions useful and sufficient for them to take 

action regarding the TB program? 

Feasibility: Are the recommendations realistic for the program to implement? 

Propriety: Are the conclusions and recommendations reflective of and respectful of the 

options of all stakeholders, including those served by the program? 

Accuracy: Can you explicitly explain your conclusions? Are the conclusions supported by the 

evidence? 

 30 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

STEP 6: ENSURE USE AND SHARE LESSONS LEARNED 

All participants and stakeholders in the evaluation should receive information 

summarizing the evaluation’s conclusions and recommendations. Even though you may 

choose to make different recommendations for different users of the evaluation, all 

recommendations should outline actions steps that can be taken to improve the 

program. 

As in Step 5, the utility of the evaluation results and conclusions must be considered. 

The optimal method of dissemination will likely differ by audience, and by what you 

anticipate that the audience members will do with the information (which is where you 

began in Step 1). For example: 

• TB program managers may prefer a written report detailing every aspect of the 

evaluation. They can use the report as a reference as they plan new initiatives

and assess resource allocations. 

• Other managers may want an informal briefing to help them brainstorm ways to 

enhance their program. 

• Program staff may gain the most from an in-service training addressing how and 

why program modifications are to be made. 

• Community members may benefit most from a town hall meeting or newsletter 

article in plain language. 

Also, it is important to think about who is the best person to deliver the message. This 

may also vary by audience. For example, a community advocate may be much more 

effective and credible at communicating messages to a community group than a health 

department official would be. 

Lessons about TB program activities or evaluation techniques should be documented, 

distributed within your TB program, and shared with other TB programs. Presenting 

results via presentations, posters, or articles in publications such as TB Notes are all 

ways to share your knowledge. 

 31 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Application of Standards for “Good” Evaluation – Step 6

Utility: Do reports/presentations clearly describe the program, including its context, and the

evaluation's purposes, procedures, and findings? Have you shared significant mid-course findings

and reports with users so that the findings can be used in a timely fashion? Have you planned,

conducted, and reported the evaluation in ways that facilitate improvement to the TB Program?

Feasibility: Is format appropriate to your resources and to the time and resources of busy 

administrators, managers, nurses, advocates, and other stakeholders? 

Propriety: Have you ensured that the evaluation findings (including the limitations) are made

accessible to everyone who will be involved with or affected by program changes? 

Accuracy: Do evaluation reports and products impartially and fairly reflect evaluation findings?

 32 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

References 

1 Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. 

Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for 

Disease Control and Prevention, 2005. 

2

 Trust for America’s Health, Ready or Not, Protecting the Public’s Health in the Age of Bioterrorism. 

Washington, D.C.: Trust for America’s Health, 2004. 

3 Institute of Medicine. Ending Neglect: The Elimination of Tuberculosis in the United States. 

Washington, DC: National Academy Press, 2000. 

4 Department of Health and Human Services, Centers for Disease Control and Prevention Tuberculosis 

Elimination and Laboratory Cooperative Agreements. Program Announcement Number: 05003, Catalog 

of Federal Domestic Assistance Number: 93.116. 2005. 

5 Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. 

MMWR 1999;48(No. RR-11). 6 U.S. Department of Health and Human Services. Healthy People 2010. 2nd ed. With Understanding and 

Improving Health and Objectives for Improving Health. 2 vols. Washington, DC: U.S. Government Printing

Office, November 2000.

7 Robert Woods Johnson Foundation, Guide to Evaluation Primers 

http://www.rwjf.org/news/special/researchPrimerPDF

8

Carter McNamara, PhD; Basic Guide to Program Evaluation. 

http://www.managementhelp.org/evaluatn/fnl_eval.htm 

 33 
Tuberculosis Evaluation Toolkit: Program Evaluation Handbook 

Appendix A 

Draft February 4 2004

Meta-model for TB Elimination Goal: Eliminate TB in the United States.

Capacity and 

infrastructure

• Adequate funding

• Strong and wellstaffed programs

• Functional data

systems, including

case management

data system

• Adequate physical,

diagnostic, and 

treatment

resources

• Policies, 

procedures, and

guidelines

• Clinical, 

epidemiological,

and operational

research

• Patient on

appropriate

treatment

• Patient

adheres to 

treatment

• Timely completion of

appropriate

treatment

Monitoring and evaluation

• Data driven interventions to

improve program activities

defined & disseminated

Inputs Activities

Enhanced

accountability

Reduced TB

transmission

= Data and programs undergo continuous improvement 

 with ongoing monitoring and evaluation feedback

Coordinate and

communicate 

effectively with

providers outside 

HD 

Outcomes

Reduced LTBI

prevalence

Reduced TB

incidence

Improved

quality of

patient life 

Provide TB education and 

training to community

providers 

Find and treat active

TB

• Identify early

• Use effective drugs 

• Reduce treatment

barriers 

Investigate 

Contacts 

• Identify contacts quickly

• Locate and evaluate in

a timely manner 

Provide 

patient

education

and 

treatment

Prevent TB in highrisk populations 

• Engage community

partners

• Detect LTBI & TB

Reduced TB

morbidity and

mortality

Elimination

of TB

 34