**A Guide to Developing a TB Program**

**Evaluation Plan**

**Division of Tuberculosis Elimination**

**National Center for HIV, STD, and TB Prevention**

**Centers for Disease Control and Prevention**

**[How to Use This Guide]{.underline}**

A good evaluation requires detailed planning. Program evaluation is an
essential part of effective program development and implementation.
Detailed planning prior to conducting evaluations will ensure successful
execution of the evaluation that meets the standards of utility,
feasibility, propriety and accuracy -- ensuring that the time, energy
and resources invested in evaluation will be worthwhile.

Writing an evaluation plan is an essential part of the planning process.
An evaluation plan serves as a guide to conducting program evaluations.
The plan should:

-   Lay out a cohesive approach to conducting the evaluation and using
    the results

-   Guide the evaluation by explaining what the activities are, when the
    activities should occur, how they will be accomplished, and who is
    responsible for completing them

-   Document the process for all stakeholders

-   Ensure implementation of the evaluation

This document is intended to help TB program staff develop and write an
evaluation plan. It consists of a **template** for writing your
evaluation plan, followed by detailed instructions and guidance for
filling in each section, as well as a sample evaluation plan written for
a program component in Appendix A. The detailed guide provide a
framework for thinking about your own program. Following the suggestions
along with completing the sections and tables, will result in a complete
evaluation plan. Suggestions or examples that may help you complete each
section are included. You can use these suggestions as they are, or as
guidance to help identify information for your own plan.

![](media/image1.jpeg){width="3.125in" height="3.125in"}An evaluation
plan should contain each of the following sections. As you can see by
the diagram in figure 1, the sections correspond to the Framework for
Program Evaluation in Public Health (MMWR, 1999).

-   Introduction

-   Stakeholder Assessment

-   Background and Description of the TB Program and Program Logic Model

-   Focus of the Evaluation

-   Gathering Credible Evidence: Data Collection

-   Justifying Conclusions: Analysis and Interpretation

-   Ensuring Use and Sharing Lessons Learned: Reporting and
    Dissemination

-   Appendices

> **Figure 1: Framework for Program Evaluation**

For additional information about how to conduct evaluation using the
Framework please refer to [www.cdc.gov/eval](http://www.cdc.gov/eval)

**[Evaluation Plan Template]{.underline}**

![](media/image2.wmf){width="0.3388888888888889in"
height="0.5006944444444444in"}

***{State Program Name}***

**Evaluation Plan for *{Years covered}***

**Prepared by:**

***{Names}***

***{Affiliation}***

***{Date}***

**INTRODUCTION**

**Evaluation Goal**

*What does this evaluation strive to achieve?*

*What is the purpose and use of this evaluation?*

**Evaluation Team**

> Lead Evaluator
>
> Team Members
>
> Evaluation Advisory Group (optional)

  -------------------- ------------------- ------------------------------
  Table 1. Roles and                       
  Responsibilities of                      
  the Evaluation Team                      
  Members                                  

  Individual           Title or Role       Responsibilities

                                           

                                           

                                           
  -------------------- ------------------- ------------------------------

**I. STAKEHOLDER ASSESSMENT**

*Who are the stakeholders for the TB program?*

*What role do they have in the evaluation?*

> *How do you plan to engage the stakeholders (i.e., conduit for
> coalition, advisory board, project teams)?*

**II. BACKGROUND AND DESCRIPTION OF THE TB PROGRAM AND PROGRAM LOGIC
MODEL**

**Need**

> *Why is the program needed (i.e., magnitude, cause(s) and trends of
> the TB problem)?*

**Context**

> *What context is the program operating under (i.e., environmental
> factors that may affect the initiative)?*

**Target Population**

*Who is the target population of this TB program?*

**Objectives**

> *What are the program's objectives (SMART objectives)?*

**Stage of Program Development**

> *What stage of development is the TB program currently in (i.e.,
> planning, implementation, mature phase of the program)?*

**Resources/Inputs**

> *What resources are available to the program in terms of staff, money,
> space, time, partnerships, etc.?*

**Activities**

> *What activities are being undertaken (or planned) to achieve the
> outcomes?*

**Outputs**

> *What products (i.e., materials, units of services delivered) are
> produced by your staff from the activities?*

**Outcomes**

> *What are the program's intended outcomes (intended outcomes are
> short-term, mid-term, or long-term)?*

**Logic Model**

*Provide a logic model of your program.*

**III. FOCUS OF THE EVALUATION**

**Stakeholder Needs**

> *Who will use the evaluation findings?*

*How will the findings be used?*

> *What do they need to learn from the evaluation?*

**Evaluation Questions**

> *What are your evaluation questions (include process driven or outcome
> driven evaluation questions)?*
>
> *What do you want to learn from the evaluation?*

**Evaluation Design**

> *Longitudinal data, points of comparison, multiple data sources, mixed
> methods*

**Resource Considerations**

> *What resources are available to conduct the evaluation?*
>
> *What data are you already collecting?*

**Evaluation Standards**

> *How will you address the standards for effective evaluation -*
>
> *utility, feasibility, propriety, and accuracy?*

**IV. GATHERING CREDIBLE EVIDENCE: DATA COLLECTION**

**Indicators**

*How will you measure success?*

> *What are some of the measurable or observable elements that can tell
> you about your program and its effects?*

+-------------------------+---------------------+---------------------+
| Table 4. Indicators and |                     |                     |
| Program Benchmark for   |                     |                     |
| Evaluation Questions    |                     |                     |
+-------------------------+---------------------+---------------------+
| Evaluation              | Process and Outcome | Program Benchmark   |
|                         | Indicators          |                     |
| Question                |                     |                     |
+-------------------------+---------------------+---------------------+
| 1\.                     |                     |                     |
+-------------------------+---------------------+---------------------+
|                         |                     |                     |
+-------------------------+---------------------+---------------------+
| 2\.                     |                     |                     |
+-------------------------+---------------------+---------------------+
|                         |                     |                     |
+-------------------------+---------------------+---------------------+

**Data Collection**

> *What methods will you use to collect the data?*

*Where is the data?*

> *How often will it be collected?*
>
> *Who is responsible for collecting the data?*
>
> *How will you manage and store the data?*

**Plan Timeline**

> *When will evaluation activities occur?*

+-------------------------+----------+----------+----------+----------+
| Table 6. Illustrative   |          |          |          |          |
| Timeline for Evaluation |          |          |          |          |
| Activities              |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
| Evaluation              | Timing   |          |          |          |
|                         | of       |          |          |          |
| Activities              | Ac       |          |          |          |
|                         | tivities |          |          |          |
|                         | for      |          |          |          |
|                         | {Year}   |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         | 1^st^    | 2^nd^    | 3^rd^    | 4^th^    |
|                         | Qtr      | Qtr      | Qtr      | Qtr      |
+-------------------------+----------+----------+----------+----------+
|                         |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         |          |          |          |          |
+-------------------------+----------+----------+----------+----------+
|                         |          |          |          |          |
+-------------------------+----------+----------+----------+----------+

**V. JUSTIFYING CONCLUSIONS: ANALYSIS AND INTERPRETATION**

**Analysis**

> *What method will you use to analyze your data (quantitative or
> qualitative techniques)?*

  -------------------------------------------- --------------------------
  Table 7. Analysis Plan                       

  Data Analysis Technique                      Responsible Person

                                               

                                               
  -------------------------------------------- --------------------------

**Interpretation**

*Who will you involve in drawing, interpreting and justifying
conclusions?*

*What are you plans to involve them in this process?*

**VI. ENSURING USE AND SHARING LESSONS LEARNED: REPORT & DISSEMINATION**

**Dissemination**

*Who is your audience?*

> *What medium do you plan to use to disseminate the evaluation findings
> to your audience?*

**Use**

*What are your plans for using evaluation findings?*

*How, where, and when will the findings be used?*

*Who will implement these findings?*

*How will you monitor your implementation plan?*

**APPENDICES**

**Guide to Writing a TB Program**

**Evaluation Plan**

**[Introduction]{.underline}**

-   *The evaluation plan begins with an introduction that provides
    background information important to the evaluation such as the goal
    of the evaluation and those who will be involved in the evaluation
    process. This section helps to identify the purpose of the document,
    and provide a roadmap of the document to the reader.*

***Evaluation Goal***

A goal is an overarching statement of desire that explains why the
evaluation is taking place. Examples of evaluation goals include:

-   To determine the effectiveness of the program

-   To assess the achievement and progress toward reaching program
    objectives

-   To investigate portions of the program that are performing optimally
    so that they can be replicated

-   To help equitably redistribute resources

Your evaluation plan should include a statement of the goal(s) and the
purpose of the evaluation, addressing how the evaluation and its
findings will be used.

***Evaluation Team***

Sound evaluation requires teamwork. This section provides guidance on
how to select your team and specifies each team member's roles and
responsibilities. Each team should have a leader or coordinator who is
responsible for ensuring the evaluation is conducted as planned. In
addition, evaluation teams will often have members who are responsible
for carrying out the various activities of the evaluation. More
comprehensive evaluations may also have an advisory group that can act
as a guidance body, a sounding board, or provide technical experts for
portions of the evaluation.

In this section, identify your evaluation leader and specify his/her
roles and responsibilities that may include....

-   Oversight of all evaluation activities

-   Coordinator of meetings for the evaluation team or advisory group

-   Principal analyst of the evaluation data

-   Primary author of the evaluation plan or reports

-   Point person for the dissemination of evaluation reports and
    materials

Identify other members on your evaluation team and specify their
evaluation responsibilities. Evaluation team members can include...

-   Individuals responsible for some aspect of data collection or data
    analysis

-   Individuals responsible for dissemination and use of the findings

-   Members of the evaluation planning committee

Your evaluation team may include people who may or may not have prior
knowledge of evaluation. These people include your program manager,
nurses, epidemiologists, representatives from partner agencies, or
others. You may need to teach them some basic evaluation concepts.
Resources are available to assist you by contacting DTBE at
TBEWG@cdc.gov

Use table 1 to help you identify the members of your evaluation team and
what role they will play. Table 1 can be inserted into your evaluation
plan or can be rewritten as text.

***The Evaluation Advisory Group** (optional)*

The evaluation advisory group is an optional component of your
evaluation's operation but it can provide important functions such
as....

-   Technical expertise not otherwise found in your evaluation team or
    your program

-   An objective viewpoint in the event of disputes

-   Representation for funders or other stakeholders

-   Interpretation of findings and justifying evaluation conclusions

-   A means of disseminating your evaluation reports outside of your
    program

**[Stakeholder Assessment]{.underline}**

 *Stakeholders are people with vested interests in the program and are
potentially affected by evaluations. Engaging stakeholders in your
evaluation is beneficial for several reasons. Their involvement
increases the credibility of your evaluation and they may hold an
important viewpoint as someone: a) responsible for day-to-day
implementation of your program; b) able to advocate for your program; or
c) able to authorize funding or changes to your program. Stakeholders
can be divided into 3 major categories: 1. those involved in program
operations, 2. those served or affected by the program, and 3. the
primary users of evaluation. The level of involvement of stakeholders
will vary among different program evaluations, but priority stakeholders
include those who can increase the credibility of the evaluation
efforts, are involved in the implementation of the program activities,
will advocate or authorize changes to the program, or will fund or
authorize improvements to the program.*

The lists below will help you think about who the stakeholders for your
evaluation might include. Depending on the context of your program,
categories of stakeholders appropriate for engagement and their levels
of involvement may vary widely. While you may already know your program
stakeholders well, you will need to reconsider their perspectives in
regard to program evaluation. Use worksheet 1 to check which group of
stakeholders may have an interest in your evaluation. This list includes
suggestions, but you can also identify other stakeholders who are not on
this list.

For each stakeholder you have identified, consider:

-   What is their interest in or perspective on the program and the
    evaluation?

-   What is their role in the evaluation?

-   How and when will they be engaged in the evaluation?

The lists below offer some answers to each of these questions. These may
or may not apply to your stakeholders. However, by considering these
factors, you may be able to anticipate some issues that may arise during
the evaluation process or later, when trying to implement the findings.

+-----------------------+------------------------+--------------------+
| -   **What is their   | -   **What is their    | -   **How and when |
|     interest in or    |     role in the        |     will they be   |
|     perspective on    |     evaluation?**      |     engaged in the |
|     the TB program    |                        |     evaluation?**  |
|     and the           |                        |                    |
|     evaluation?**     |                        |                    |
+-----------------------+------------------------+--------------------+
| -   Defensive of      | -   Define health      | -   Formal         |
|     current system    |     problem            |     Interview      |
|                       |                        |                    |
| -   Fear program      | -   Define problem in  | -   Informal       |
|     alteration & job  |     program from staff |     Interview      |
|     loss              |     perspective        |                    |
|                       |                        | -   Focus group    |
| -   Fear increase in  | -   Define problem in  |                    |
|     workload          |     program from       | -   Meeting        |
|                       |     client perspective |                    |
| -   See program       |                        | -   Advisory       |
|     evaluation as a   | -   Define program     |     committee      |
|     personal judgment |     processes          |                    |
|                       |                        | -   External       |
| -   Fear or reject    | -   Develop or review  |     Reviewer       |
|     health system     |     data collection    |                    |
|                       |     tools              | -   Data collector |
| -   Need for services |                        |                    |
|                       | -   Provide community  | -   Participant    |
| -   Concerns about    |     context            |                    |
|     immigration       |                        | -   Information    |
|     status            | -   Provide            |     only           |
|                       |     operational        |                    |
| -   Partnership       |     context            |                    |
|     capacity          |                        |                    |
|                       | -   Providing          |                    |
| -   Community image   |     administrative and |                    |
|                       |     funding context    |                    |
| -   Proof of effects  |                        |                    |
|     for funding and   | -   Identify           |                    |
|     program expansion |     stakeholders       |                    |
|     purposes          |                        |                    |
|                       | -   Identify data      |                    |
| -   Community health  |     sources            |                    |
|     among underserved |                        |                    |
|     population        | -   Collect data       |                    |
|                       |                        |                    |
| -   Interest in       | -   Interpret findings |                    |
|     self-improvements |                        |                    |
|     and the           | -   Disseminate        |                    |
|     improvements of   |     findings within    |                    |
|     the program       |     program            |                    |
|                       |                        |                    |
|                       | -   Disseminate        |                    |
|                       |     findings to        |                    |
|                       |     community          |                    |
|                       |     audiences          |                    |
|                       |                        |                    |
|                       | -   Disseminate        |                    |
|                       |     findings to agency |                    |
|                       |     leaders and        |                    |
|                       |     funding agencies   |                    |
+-----------------------+------------------------+--------------------+

In table 2 below, list the stakeholders that you have identified and
fill in: their interest or perspective, their role in the evaluation,
and how and when you will engage them. After completing table 2, it can
be inserted into your evaluation plan with some explanatory text.

Often, the roles of stakeholders will change during an evaluation. You
may need to revise your plan several times as changes occur.

+-------------+---------------------+------------+-------------------+
| **Table 2.  |                     |            |                   |
| Stakeholder |                     |            |                   |
| Assessment  |                     |            |                   |
| and         |                     |            |                   |
| Engagement  |                     |            |                   |
| Plan**      |                     |            |                   |
+-------------+---------------------+------------+-------------------+
| Stakeholder | Interest or         | Role in    | # How a           |
| Category    | Perspective on      | the        | nd When to Engage |
|             | Evaluation          | Evaluation |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+
|             |                     |            |                   |
+-------------+---------------------+------------+-------------------+

**[Background and Description of the TB Program]{.underline}**

 *The purpose of this section is to describe your TB program. The
program description will ensure that all evaluation stakeholders have a
shared understanding of your TB program's objectives, activities, and
outputs. It will also identify any unfounded assumptions or logical gaps
in your evaluation planning. Once completed, the program description
will lead naturally into your program's logic model.*

***Need***

The need for your program can be addressed by considering a series of
questions. Your answers to these questions will help you develop the
plan for this section.

1.  What problem does or should your program address (e.g., TB incidence
    among immigrant population is rising, patients not adherance to
    treatment plan)?

2.  What causes the problem (e.g., persons from high TB incidence
    countries arriving in state, these persons are not being reached for
    TB and LTBI treatment; cultural/language barriers)?

3.  What are the consequences of the problem (e.g., treatment costs
    rise, morbidity, ill-health, backlash against the foreign-born
    population)?

4.  What is the magnitude of the problem (e.g., there exists a potential
    for a large problem, but acting now may reduce this potential)?

5.  What changes or trends are occurring (e.g., TB incidence is
    increasing, demographic patterns are changing, decrease in
    completion of therapy rate)?

***Context***

Context considers the environment that affects your TB program's
operations. These factors can include

-   How the TB program collaborates and coordinates with other health
    and social services in the community, such as hospitals, schools,
    HIV service programs, and programs that provide housing and food
    support.

-   How the program is funded -- does it compete for resources with
    other public health programs within the community?

-   Organization's structural factors (i.e., personnel, training,
    administrative regulations) that impact your program's operations

-   Policy and political environment surrounding your program

-   Community perceptions of TB as a disease

Consider and explain the contextual factors in your evaluation plan as
necessary.

***Target Population***

This section describes the group that your TB program targets in
addressing the concerns. For example,

-   Newly arrived immigrants

-   Homeless persons

-   People who are HIV infected

-   People who abuse substances

-   People living in congregate settings (e.g., low-income,
    institutions, prisons)

***Objectives***

Your statement of program objectives should refer back to your program's
goal(s). Objectives should be established that support this goal. Your
program's objectives are frequently national or state TB objectives,
however, your program may also have its own objectives. If your
objectives are not established, it is an important part of your plan to
do this now.

If you choose to write your own objectives, be sure that they are
*S-M-A-R-T*: *specific*, *measurable*, *achievable*, *relevant*, and
*time-bound*. Detailed explanations and examples of SMART objectives can
be found in the Appendix B.

***Stage of Program Development***

Assessing the developmental stage of your program will help you frame
your evaluation and write your evaluation questions. The stage of
development may also vary among the different program components that
carry out specific program activities. Consider carefully how long the
program has been in existence and what stage is each one of your program
components/activities in?

-   Needs assessment

-   Design and planning

-   Implementation: early (e.g., less than a year) or middle?

-   Well-established implementation (e.g., 13 years)

Stage of development will also help you to decide on process evaluation,
outcome evaluation, or both.

***Resources/Inputs***

This section describes all of the resources available to implement your
program. This can include (but is not limited to)

-   TB program staff

-   Training and continuing education staff

-   Funding

-   Physical or structural components (space, testing facilities)

-   Infrastructure resources that are part of the health department

-   Partner organizations

To complete this section, make a list of all the resources available to
your program in table 3 below.

***Activities***

Activities describe what your program is doing to accomplish objectives.
These can include

-   Hiring and training new staff

-   Policy development or revision

-   Providing targeted TB testing to specific high-risk populations

-   Identifying persons with latent TB infection or active TB disease

-   Administering DOT

-   Educating patients or community members

-   Providing laboratory and diagnostic services

-   Surveillance

-   Managing clinic operation

-   Providing technical assistance

To complete this section, make a list of all the activities undertaken
by your program in table 3 below. Note that activities can be grouped as
initial and subsequent activities. Although most activties in TB
programs are ongoing, some activities may need to be done before others.
For example, diagnosis of TB disease may come prior to prescribing a
treatment to treat the disease.

***Outputs***

These are the immediate materials, services, and capacities that are a
direct product of your programmatic activities and processes. Examples
include

-   A strategic plan for your program

-   TB Tests conducted

-   Providers educated about TB

-   Education sessions presented to community groups

Outputs are assessed to "show" that the program is being implemented as
planned. However, they are not indications of effectiveness. For
example, we can record that 10 educational sessions were conducted or
that 50 skin tests were administered, but there's no direct indication
whether people have learned or acted on the results of their tests. In
short, outputs show what work was done by our staff, but do not indicate
that changes have taken place or resulted due to these actions. To
identify your programs outputs, consider each activity you listed in
table 3 and list corresponding outputs in the column to the right of
your activities. For example, the output "TB test conducted" corresponds
to the activity, "providing TB testing." The activity, "administering
DOT" corresponds to the output, "DOT adminisered or patients received
DOT."

***Outcomes***

Outcomes are the intended effects of the program's activities. They may
or may not have been achieved by your program. They are the changes you
want to occur in patients, providers, or the community, because of the
work of your TB program. These are typically thought of as short-, mid-
or long-term outcomes and should be tied to your program objectives.
Some examples are...

-   Patients' knowledge is increased

-   Patients accept treatment for TB and LTBI

-   Patients identify contacts

-   Patients on appropraite treatment

-   Patients adhere to treatment

-   Timely completion of treatment

-   Reduced hospital admissions for TB

-   Reduced transmission of TB

Some of the important changes we want to occur, however, are less direct
or easy to measure. For example we may want:

-   Trust built within the community

-   Patients identify all of their close contacts

-   Stigma about TB reduced

-   Improved patient quality of life

-   Patients are healthier overall

For these outcomes, "proxy" or indirect measures can be used to assess
whether they have been achieved. Often, program staff may be able to
suggest ways to measure these effects. For example, an outreach worker
may "know" trust is built when a community member approaches him with a
question about TB. A nurse may "know" stigma is reduced when a patient
talks openly in front of a friend. These types of information can be
incorporated and used in the evaluation. See page 26 for more
information on using "proxy" measures in evaluation.

To identify your program outcomes, consider the activities and outputs
you listed in table 3 and list corresponding outcomes in the column to
the right. Once completed, table 3 can be included in your evaluation
plan as part of the program description.

+-------------+--------+--------+--------------+----------+---------+
| **Table 3.  |        |        |              |          |         |
| Program     |        |        |              |          |         |
| De          |        |        |              |          |         |
| scription** |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
| Resources   | Acti   |        | Outputs      | ##       |         |
|             | vities |        |              | Outcomes |         |
+-------------+--------+--------+--------------+----------+---------+
|             | I      | Subs   |              | Short-/  | Lo      |
|             | nitial | equent |              | Mid-term | ng-term |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+
|             |        |        |              |          |         |
+-------------+--------+--------+--------------+----------+---------+

**[Program Logic Model]{.underline}**

-   *A logic model is a graphic depiction of the program. The arrows
    describe the links between inputs/resources, activities, outputs and
    outcomes as illustrated by figure 2 below. Using the resources,
    activities, outputs, and outcomes identified in the project
    description (table 3), you can now develop a logic model for your
    program.*

Drawing a logic model during the evaluation planning process is
important to define the associations among program resources,
activities, and its results. For the evaluation, a logic model provides:

-   A sense of scope -- what are the program's components? How are they
    interconnected?

-   A "map" to help ensure that systematic decisions are made about what
    is to be measured in the evaluation process and to identify areas
    where clarification may be needed.

-   A framework for organizing indicators and for ensuring that none are
    overlooked

```{=html}
<!-- -->
```
-   *Logic models for high priority TB program activities have been
    developed and are available as examples. It may be useful to review
    these models prior to developing your own to see if one of these
    logic models can be used to describe your program with minor
    modifications. The six TB logic models and their corresponding goals
    are (See Appendix C):*

```{=html}
<!-- -->
```
-   **Meta-model for TB Elimination**

> Eliminate TB in the United States.

-   **Capacity and Infrastructure to Eliminate TB**

> Develop capacity and infrastructure TB elimination.

-   **Evaluation Capacity Building**

> Promote and enhance active program monitoring and self-evaluation
> among TB control partners at each administrative level.

-   **Completion of Therapy**

> Complete TB treatment among person with TB disease within 12 months of
> diagnosis.

-   **Prevent TB in High Risk Populations**

> Prevent TB among high-risk populations.
>
> (through targeted testing and treatment)

-   **Contact Investigation**

> Prevent TB among contacts to cases.
>
> (by finding and testing contacts for TB and LTBI, and then treat
> infected contacts to completion)

After creating or adapting the logic model that best depicts your
program, include it in your evaluation plan. There are no "right" or
"wrong" logic models, but the logic model must clearly show the complete
paths linking inputs and activities to outcomes. You will probably need
to review and revise your logic model many times throughout the
evaluation.

Note that you can use a table format or a graphic (hand drawn is
acceptable) for the logic model. Attach this to your evaluation plan.

**[Focus of the Evaluation]{.underline}**

-   *Although the logic model may inspire many evaluation questions, it
    may not be feasible or useful to evaluate every element or path in
    your logic model. Thus, focusing your evaluation and selecting your
    evaluation questions are important steps. Focus will also ensure
    that the evaluation meets the needs of stakeholders and that the
    findings will be used as intended.*

***Stakeholder Needs***

> Your answers to the following questions will help you focus your
> evaluation by identifying what your stakeholders need to learn from
> the evaluation -- the evaluation questions.

-   Who will use the evaluation findings?

-   How will the findings be used?

-   What do they need to learn from the evaluation?

The worksheet below will help you think about possible answers to these
questions. Check who will use the evaluation findings, how the findings
will be used and what they need to learn from the evaluation. The list
includes sample answers to the questions. You may come up with different
answers on your own.

***Evaluation Questions***

There are two basic types of evaluation questions:

Process questions, which focus on examining the implementation of the
program,

Are the activities we planned being implemented?

Are we producing the outcomes we expect?

Are we using our inputs and resources effectively?

Outcome evaluation focuses on showing whether or not a program achieves
the desired changes in patients, providers, or the community.

Using your assessment of who will use the findings and how the findings
will be used, identify key evaluation questions based on your logic
model.

The sample evaluation questions are shown below. These questions are
based on the logic model shown in figure 3.

![](media/image3.jpeg){width="7.979166666666667in"
height="4.739583333333333in"}

Figure 3

From the long list of questions that can be generated, you will need to
select your 3-5 highest priority evaluation questions. Questions should
reflect both process and outcomes. However, these questions are still
preliminary pending the considerations of the data needs of your
evaluation questions. Not only should the evaluation serve the needs of
the stakeholders, but it should be feasible and produce accurate
findings. Depending on the resources and time available for your
evaluation, only a few evaluation questions can be assessed.

After selecting a few evaluation questions from your highest priority
questions, these questions will dictate your evaluation design and data
collection methods. In this section of your evaluation plan, state what
questions will be addressed with this evaluation.

***Evaluation Design***

Although program evaluation questions are geared to answering specific
questions for

specific programs, the designs for answering them often resemble
research designs.

However, it is important to remember that the purpose of evaluation is
to improve programs, not to publish generalizable findings, and
therefore you need only collect data sufficient to answer your
evaluation questions.

Key issues you need to decide about evaluation design are:

-   Will you have a control or comparison group?

-   Will you measure before and after or only after?

-   Will you collect data prospectively or retrospectively?

-   Do you need in-depth, detailed information about the question(s)
    (qualitative information), or specific, targeted information
    (quantitative information)?

***Resource Considerations***

-   What resources are available to conduct the evaluation (e.g., staff
    time for members of the evaluation team, data collection, funds to
    hire evaluators, space to house evaluators or evaluation data)?

-   What data do you already have or are already collecting for another
    purpose that may be useful for the evaluation? What data do you
    need?

[]{.mark}

Additional considerations that are helpful in selecting your design:

-   **Standards of "good" evaluation.** You will want to select a design
    that provides useful information to improve the TB program, and is
    feasible such that it is not overly disruptive of daily operations.
    In addition, the design that is selected should produce accurate
    findings given the program's operations and the resources available
    for the evaluation.

-   **Timeliness.** When do decisions need to be made about the program?

-   **Stage of program development.** For a newly developed
    component/activity, process evaluation helps us understand what we
    need to do in order to enhance the program. In a well established
    program, the addition of outcome evaluation may serve to help
    identify program's performance and effectiveness.

Consider strengthening your design by

-   Mix methods when appropriate such as combining interviews and chart
    reviews.

-   Use repeated measures

-   Triangulate (use multiple and varied data sources to draw
    conclusions)

**[Gathering Credible Evidence: Data Collection]{.underline}**

-   *This section provides information about how you will conduct the
    evaluation. It includes the indicators you will use to determine
    program success, program targets that will serve as benchmarks to
    evaluate performance against, data collection methods and tools, and
    a timeline for data collection activities.*

***Indicators***

Indicators are measurable signs of a program's performance. Good
indicators are relevant, understandable, and useful. Indicators are tied
to the objectives identified in the program description, the logic model
and/or the evaluation questions. A library of indicators that can be
used by TB programs has been developed*.* A sample list of indicators is
provided in Appendix D. Many of these will be useful in your evaluation
design. Note that these are a mix of process indicators that measure
implementation activities as well as outcome indicators. However, if you
choose to write your own indicators, tie your indicators to a S-M-A-R-T
objective as described below in worksheet 3. Use Table 4 to record your
indicators and tie them to evaluation questions.

**Worksheet 3. Creating Indicators from S-M-A-R-T Objectives**

+----+------+-------+----------+----------+---+-------+-------+---------+
| >  | Incr |       |          |          |   |       |       |         |
|  * | ease |       |          |          |   |       |       |         |
| *O | pe   |       |          |          |   |       |       |         |
| bj | rcen |       |          |          |   |       |       |         |
| ec | tage |       |          |          |   |       |       |         |
| ti | of   |       |          |          |   |       |       |         |
| ve | a    |       |          |          |   |       |       |         |
| ** | dult |       |          |          |   |       |       |         |
|    | pati |       |          |          |   |       |       |         |
|    | ents |       |          |          |   |       |       |         |
|    | with |       |          |          |   |       |       |         |
|    | n    |       |          |          |   |       |       |         |
|    | on-r |       |          |          |   |       |       |         |
|    | esis |       |          |          |   |       |       |         |
|    | tant |       |          |          |   |       |       |         |
|    | TB   |       |          |          |   |       |       |         |
|    | who  |       |          |          |   |       |       |         |
|    | c    |       |          |          |   |       |       |         |
|    | ompl |       |          |          |   |       |       |         |
|    | eted |       |          |          |   |       |       |         |
|    | the  |       |          |          |   |       |       |         |
|    | rapy |       |          |          |   |       |       |         |
|    | (wi  |       |          |          |   |       |       |         |
|    | thin |       |          |          |   |       |       |         |
|    | 12   |       |          |          |   |       |       |         |
|    | m    |       |          |          |   |       |       |         |
|    | os.) |       |          |          |   |       |       |         |
|    | from |       |          |          |   |       |       |         |
|    | 80%  |       |          |          |   |       |       |         |
|    | to   |       |          |          |   |       |       |         |
|    | 90%  |       |          |          |   |       |       |         |
|    | by   |       |          |          |   |       |       |         |
|    | 2    |       |          |          |   |       |       |         |
|    | 006. |       |          |          |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
| >  | **VE | **MET | **POPU   | **       | * |       | **G   | **TIME  |
|  * | RB** | RIC** | LATION** | OBJECT** | * |       | OAL** | FRAME** |
| *B |      |       |          |          | B |       |       |         |
| re |      |       |          |          | A |       | *     |         |
| ak |      |       |          |          | S |       | *MEAS |         |
| do |      |       |          |          | E |       | URE** |         |
| wn |      |       |          |          | L |       |       |         |
| ** |      |       |          |          | I |       |       |         |
|    |      |       |          |          | N |       |       |         |
|    |      |       |          |          | E |       |       |         |
|    |      |       |          |          | * |       |       |         |
|    |      |       |          |          | * |       |       |         |
|    |      |       |          |          |   |       |       |         |
|    |      |       |          |          | * |       |       |         |
|    |      |       |          |          | * |       |       |         |
|    |      |       |          |          | M |       |       |         |
|    |      |       |          |          | E |       |       |         |
|    |      |       |          |          | A |       |       |         |
|    |      |       |          |          | S |       |       |         |
|    |      |       |          |          | U |       |       |         |
|    |      |       |          |          | R |       |       |         |
|    |      |       |          |          | E |       |       |         |
|    |      |       |          |          | * |       |       |         |
|    |      |       |          |          | * |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
|    | Incr | Pe    | Adult    | Co       | 8 |       | 90%   | By 2006 |
|    | ease | rcent | patients | mpletion | 0 |       |       |         |
|    |      |       |          |          | % |       |       |         |
|    |      |       | with     | of       |   |       |       |         |
|    |      |       |          | therapy  |   |       |       |         |
|    |      |       | non-r    |          |   |       |       |         |
|    |      |       | esistant | (w/in 12 |   |       |       |         |
|    |      |       | TB       | mos.)    |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
|    |      |       |          |          |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
| >  | **VE | **MET | **POPU   | **       |   | **    | **G   | **TIME  |
|  * | RB** | RIC** | LATION** | OBJECT** |   | BASEL | OAL** | FRAME** |
| *B |      |       |          |          |   | INE** |       |         |
| re |      |       |          |          |   |       | *     |         |
| ak |      |       |          |          |   | *     | *MEAS |         |
| do |      |       |          |          |   | *MEAS | URE** |         |
| wn |      |       |          |          |   | URE** |       |         |
| ** |      |       |          |          |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
|    | Incr | Pe    | Adult    | Co       |   | 80%   | 90%   | By 2006 |
|    | ease | rcent | patients | mpletion |   |       |       |         |
|    |      |       | with     |          |   |       |       |         |
|    |      |       |          | of       |   |       |       |         |
|    |      |       | non-r    | therapy  |   |       |       |         |
|    |      |       | esistant |          |   |       |       |         |
|    |      |       | TB       | (w/in 12 |   |       |       |         |
|    |      |       |          | mos.)    |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+
| >  | Per  |       |          |          |   |       |       |         |
|  * | cent |       |          |          |   |       |       |         |
| *I | of   |       |          |          |   |       |       |         |
| nd | a    |       |          |          |   |       |       |         |
| ic | dult |       |          |          |   |       |       |         |
| at | pati |       |          |          |   |       |       |         |
| or | ents |       |          |          |   |       |       |         |
| ** | with |       |          |          |   |       |       |         |
|    | n    |       |          |          |   |       |       |         |
|    | on-r |       |          |          |   |       |       |         |
|    | esis |       |          |          |   |       |       |         |
|    | tant |       |          |          |   |       |       |         |
|    | TB   |       |          |          |   |       |       |         |
|    | who  |       |          |          |   |       |       |         |
|    | c    |       |          |          |   |       |       |         |
|    | ompl |       |          |          |   |       |       |         |
|    | eted |       |          |          |   |       |       |         |
|    | the  |       |          |          |   |       |       |         |
|    | rapy |       |          |          |   |       |       |         |
|    | (wi  |       |          |          |   |       |       |         |
|    | thin |       |          |          |   |       |       |         |
|    | 12   |       |          |          |   |       |       |         |
|    | m    |       |          |          |   |       |       |         |
|    | os.) |       |          |          |   |       |       |         |
|    | in   |       |          |          |   |       |       |         |
|    | 2    |       |          |          |   |       |       |         |
|    | 006. |       |          |          |   |       |       |         |
+----+------+-------+----------+----------+---+-------+-------+---------+

When your evaluation questions do not draw on existing program
objectives, a similar strategy can be used to develop indicators.
Keeping in mind that indicators are visible and measurable signs of
change, identify some observable manifestations of the concept, using
proxy measures as appropriate, and state them in the proposed fashion
(including terms for metric, population, object and timeframe as
suggested in worksheet 3).

***Program Benchmarks or Targets***

Program Benchmarks or targets are what the stakeholders of the TB
programs consider to be "reasonable expectations" for the program. In
thinking about the program benchmarks, it is important to think about
what "success" means. How do you measure success? What does it mean if
the program is successful or effective? These standards that programs
have set for themselves will be used as the benchmark against which you
will evaluate your program's performance. The overarching program
benchmarks or targets are usually established when the management team
formalizes the program objectives and sets the goal(s) for the program.
Standards or targets may not exist that relate to all of your evaluation
questions, but many standards are implicit in a program's strategic
plans, TB National Guidelines, and treatment protocols. Stakeholders can
also help you set program standards. The example below illustrates how
evaluation questions, indicators, and program benchmarks relate to one
another. Benchmarks or targets that you identify can be also recorded in
table 4.

[]{.mark}

  ------------------ ----------------------------- -----------------------
  **Evaluation       **Process and Outcome         **Program Benchmarks**
  Question**         Indicators**                  

  Have               Number of new                 Increase in the number
  Spanish-speaking   Spanish-speaking persons      of Spanish-speaking
  persons been       treated by clinic for TB &    patients
  treated            LTBI between Jan -- June.     
  appropriately for                                
  LTBI or TB?                                      

                     Number of times clinical      Clinical standards are
                     treatment standards (i.e.,    met 100% of time
                     proper referrals made and     
                     test conducted) are met for   
                     Spanish-speaking patients     

                     Percent of time that signs    Patient education signs
                     and forms are available in    and forms in Spanish
                     Spanish and written for       are available 100% of
                     persons with low-literacy     time; literacy level of
                     skills                        materials is at a 6^rd^
                                                   grade reading level

                     Percent of time that          80% of time
                     Spanish-speaking staff are    Spanish-speaking staff
                     present in clinic             are available

                     Percent of clerks/staff who   All staff know how to
                     know how to access            access and use
                     translators or certified      translators or
                     interpreters                  certified interpreters

                     Percent of patients who       90% of Spanish-speaking
                     report being treated          patients report that
                     respectfully                  they were treated
                                                   respectfully
  ------------------ ----------------------------- -----------------------

After completing table 4, it can be included in your evaluation plan
with explanatory text. Note that it is likely that you will have
multiple indicators tied to each evaluation question.

  ---------------- ------------------------------- -----------------------
  **Table 4.                                       
  Indicators and                                   
  Program                                          
  Benchmarks for                                   
  Evaluation                                       
  Questions**                                      

  Evaluation       Process and Outcome Indicators  Program Benchmarks or
  Question                                         Targets

  1\.                                              

                                                   

                                                   

                                                   

                                                   

  2\.                                              

                                                   

                                                   

                                                   

                                                   

  3\.                                              

                                                   

                                                   

                                                   

                                                   

  4\.                                              

                                                   

                                                   
  ---------------- ------------------------------- -----------------------

***Data Collection***

Your plan should include details about collecting the necessary data for
each indicator. Use table 5 to describe your data collection plan. To
complete this table, consider the following questions for each
indicator:

-   What methods will you use to collect the data?

-   Where are the data?

-   How often will it be collected?

-   Who is responsible for collecting the data?

-   How will you handle and store the data?

Note that more than one data source may provide information for each
indicator.

Examples of data sources include:

-   Records or charts

-   TB registry and databases

-   Interviews or focus groups

-   Participant observations

-   Other documents

  ------------------- ----------------------- ------------ ---------- ---------
  **Table 5. Data                                                     
  Collection Plan**                                                   

  Indicator           Data Sources            Collection              

                                              Who          When       How

                                                                      

                                                                      

                                                                      

                                                                      

                                                                      

                                                                      

                                                                      
  ------------------- ----------------------- ------------ ---------- ---------

***Tools***

Tools are the documents or strategies that you will use to collect the
data you need. When choosing tools, consider whether the questions you
are asking or data elements you are collecting are tied to your
indicators.

-   Collect the information you need in the most straightforward way
    possible

-   Collect only the information you need

-   Use tools that are easy to understand, administer and use, and do
    not place undue burdens on staff or patients

-   Pilot test tools before using them to collect the evaluation data so
    that you know that users can successfully use the tool for its
    intended purpose. Make changes based on your pilot test

In your evaluation plan, describe the tools you are using and their
purpose. The tools you will use to collect the evaluation data can be
attached to the plan as an appendix if you choose.

***Evaluation Timeline***

A timeline is not required in an evaluation plan but it can be helpful
in guiding evaluation activities. Developing a timeline will ensure that
all stakeholders are aware of what activities are occurring at any time.
It may also help you determine if your evaluation resources will be
strained by too many activities happening at any given time. Table 6 is
an illustrative timeline that may be useful in your evaluation plan if
you choose to use one.

+-----------------+------------+------------+------------+------------+
| **Table 6.      |            |            |            |            |
| Illustrative    |            |            |            |            |
| Timeline for    |            |            |            |            |
| Evaluation      |            |            |            |            |
| Activities**    |            |            |            |            |
+-----------------+------------+------------+------------+------------+
| Evaluation      | Timing of  |            |            |            |
|                 | Activities |            |            |            |
| Activities      | for {Year} |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 | 1^st^ Qtr  | 2^nd^ Qtr  | 3^rd^ Qtr  | 4^th^ Qtr  |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+
|                 |            |            |            |            |
+-----------------+------------+------------+------------+------------+

**[Justifying Conclusions: Analysis and Interpretation]{.underline}**

-   *This section describes your plan for analyzing and interpreting the
    evaluation data. Specific components of this section will be
    determined by your design and methods. However, general guidance for
    this section is below.*

***Analysis***

Describe what techniques you will use to analyze your evaluation data.
Although your initial analysis plan may be general, you may want to
address issues such as:

-   What data aggregation systems or software you plan to use

-   What statistical methods (if any) you plan to use

-   What stratifications (if any) you plan to examine among the data

-   What types of tables or figures you may use

Complete Table 7 to describe your analysis plan linking data collection
methods to analysis techniques and identifying who is responsible for
this analysis. If you need assistance with selecting methods or
techniques, [www.cdc.gov/eval](http://www.cdc.gov/eval) has many
resources available to assist in the process.

  --------------------------------------------------- -------------------
  **Table 7. Analysis Plan**                          

  Data Analysis Technique                             Responsible Person

                                                      

                                                      

                                                      

                                                      

                                                      
  --------------------------------------------------- -------------------

***Interpretation***

In this phase you will judge your findings against the program
standards. In drawing conclusions from the evaluation findings, it is
important to consider the context in which the program is operating. It
is also important that conclusions be sound, reasonable and objective.
Involving the stakeholders in this process will bring insights and
explanation to the evaluation findings, thus ensuring the validity of
the interpretation and that recommendations based on the findings are
relevant. Developing a draft report and sharing it with stakeholders is
one method of involving stakeholders in the interpretation process and
may be sufficient and appropriate in some cases.

Use worksheet 4 to help you plan what type of stakeholder should be
invited to your meeting to interpret the findings. This list is intended
to give you ideas and suggestions as to whom you might like to involve
in this important process. However, other people should be considered as
appropriate.

In your evaluation plan, explain who will be involved in interpreting
the findings and describe the procedures and guidelines you will use to
help you interpret the evaluation findings.

**[Ensuring Use and Sharing Lessons Learned: Reporting and
Dissemination]{.underline}**

-   *A reporting and dissemination plan will ensure that evaluation
    findings will be distributed to those who will make use of the
    lessons learned from the evaluation. The plan should describe what
    medium you will use to disseminate the evaluation findings, who is
    responsible for disseminating the findings, how the findings will be
    used and who will act on the findings. The purpose of program
    evaluation is to use the information from the evaluation to improve
    program operation and enhance its performance. An evaluation does
    not achieve its purpose if not used. It is helpful to review your
    stakeholder assessment in developing the reporting and dissemination
    plan. Use table 8 to indicate how the findings will be distributed
    and by whom.*

In writing this section of your plan, check the reporting and
dissemination plan against the stakeholder list you developed earlier
and refer to worksheet 3 to ensure that your reports will address
stakeholder needs and that the reports will reach them.

***Ensuring Use***

Throughout the evaluation process you will want to develop mechanisms to
ensure that findings are used and changes implemented. Your plan should
address how you plan to use your findings, in at least a general way.
You will add to the list of uses as your evaluation progresses.

Although not typically something attached to an evaluation plan, the
following action planning worksheet can help you anticipate how your
findings will be used to improve your program.

+----------------------+----------------+------------+----------------+
| **Worksheet 5.       |                |            |                |
| Action Planning**    |                |            |                |
+----------------------+----------------+------------+----------------+
| Evaluation           |                |            |                |
| recommendation:      |                |            |                |
+----------------------+----------------+------------+----------------+
| Action step(s)       | Person(s)      | Timeline   | Plans for      |
|                      |                |            | monitoring     |
|                      | Responsible    |            | results        |
+----------------------+----------------+------------+----------------+
|                      |                |            |                |
+----------------------+----------------+------------+----------------+
|                      |                |            |                |
+----------------------+----------------+------------+----------------+
|                      |                |            |                |
+----------------------+----------------+------------+----------------+

**[Appendices]{.underline}**

-   *Appendices to your evaluation plan can include many different items
    that are too detailed or long to include in the body of the plan.
    These might include...*

    -   *References*

    -   *Copies of instruments or tools*

    -   *Analysis programs*

    -   *Reporting formats*

**Appendix A: A Sample Evaluation
Plan**![](media/image2.wmf){width="0.32916666666666666in"
height="0.49444444444444446in"}

**TB Support Program**

**Evaluation Plan for January -- March 2004**

**Prepared by:**

**Ana Garcia**

***TB Nurse***

**Department of Health and Human Service**

**Division of TB Control & Prevention**

**Lull County**

**November 15, 2003**

**INTRODUCTION**

Tuberculosis was not a major concern in Lull County until in the recent
years. According to the county report, an average of 3 cases was
reported per year from 1995 to 2001. In 2002, six new cases of TB were
identified within three month period. Unlike previous years, the
infected suspects were new immigrant Salvadoran men rather than the
predominant white population. In response to this epidemic, "TB Support
Program" was implemented by the county health department director, Henry
Evans, to control and prevent further transmission of TB in Lull County.

**Evaluation Goal**

The goal of this evaluation is to determine the effectiveness of "TB
Support Program" in preventing transmission of TB among Salvadoran
immigrants. This evaluation will investigate components of the TB
Support Program that are performing optimally and should be expanded and
replicated in future initiatives. In addition, this evaluation will help
determine the funding needs of TB Support Program for the following
year.

**Evaluation Team**

Our team consists of TB Support program staff and a nurse from the TB
clinic who works within the system and can access clinic/program data
easily. A member of the county commission was asked to participate on
the team to review plans and ensure needs for information are met.
Carlos, a leader from the local community who has a background in the
social sciences, was also asked to participate to ensure the evaluation
is culturally appropriate to the target community.

+------------------------+------------------+-------------------------+
| **Table 1. Roles and   |                  |                         |
| Responsibilities of    |                  |                         |
| the Evaluation Team    |                  |                         |
| Members**              |                  |                         |
+------------------------+------------------+-------------------------+
| **Individual**         | **Title or       | **Responsibilities**    |
|                        | Role**           |                         |
+------------------------+------------------+-------------------------+
| Ana (TB nurse)         | -   Lead         | -   Oversight of all    |
|                        |     Evaluator    |     evaluation          |
|                        |                  |     activities to       |
|                        |                  |     ensure the          |
|                        |                  |     evaluation is       |
|                        |                  |     conducted as        |
|                        |                  |     planned             |
|                        |                  |                         |
|                        |                  | -   Coordinate meetings |
|                        |                  |     for the team        |
+------------------------+------------------+-------------------------+
| Libby (Program         | -   Data         | -   Analyze             |
| manager)               |     Analysis     |     quantitative data   |
|                        |                  |                         |
|                        |                  | -   Coordinate the      |
|                        |                  |     analysis of         |
|                        |                  |     qualitative data    |
|                        |                  |                         |
|                        |                  | -   Ensure              |
|                        |                  |     implementation of   |
|                        |                  |     findings            |
+------------------------+------------------+-------------------------+
| Don (County health     | -   Sta          | -   Provide support and |
| commissioner)          | keholder/advisor |     guidance and        |
|                        |                  |     dissemination of    |
|                        |                  |     results             |
+------------------------+------------------+-------------------------+
| John (Outreach worker) | -   Data         | -   Gather and review   |
|                        |     collection   |     data (QA?)          |
+------------------------+------------------+-------------------------+
| Carlos (Health         | -   Community    | -   Coordinate data     |
| outreach projects      |     liaison      |     collection with     |
| director, Latin        |                  |     community members   |
| American Association,  |                  |                         |
| (CBO))                 |                  |                         |
+------------------------+------------------+-------------------------+

**I. STAKEHOLDER ASSESSMENT**

![](media/image2.wmf){width="0.32916666666666666in"
height="0.49444444444444446in"}

At the monthly TB program staff meeting, the evaluation of TB Support
Program was discussed. The following stakeholders were identified, along
with their interests and perspectives and how each stakeholder should be
involved in the process. After the meeting, Carlos was also contacted to
give perspective on how patients and the community should be engaged.
The following table summarizes the plan for stakeholder engagement.

+------------------+--------------------+------------------+-----------+
| **Table 2.       |                    |                  |           |
| Stakeholder      |                    |                  |           |
| Assessment and   |                    |                  |           |
| Engagement       |                    |                  |           |
| Plan**           |                    |                  |           |
+------------------+--------------------+------------------+-----------+
| **Stakeholder    | **Intere           | **Role in the    | **How to  |
| categories**     | sts/perspectives** | evaluation**     | engage**  |
+------------------+--------------------+------------------+-----------+
| **Persons        |                    |                  |           |
| involved in      |                    |                  |           |
| program          |                    |                  |           |
| operations**     |                    |                  |           |
+------------------+--------------------+------------------+-----------+
| -   Libby, John, | -   Fear program   | -   Defining     | -         |
|     Ana and      |     (and jobs) may |     program and  |  Meetings |
|     other staff  |     be negatively  |     context      |           |
|                  |     changed        |                  | -         |
|                  |                    | -   Identifying  |    Direct |
|                  | -   See program    |     data sources |     roles |
|                  |     evaluation as  |                  |     in    |
|                  |     a personal     | -   Collecting   |     c     |
|                  |     judgment       |     data         | onducting |
|                  |                    |                  |     e     |
|                  |                    | -   Interpreting | valuation |
|                  |                    |     findings     |           |
|                  |                    |                  |           |
|                  |                    | -                |           |
|                  |                    |    Disseminating |           |
|                  |                    |     and          |           |
|                  |                    |     implementing |           |
|                  |                    |     findings     |           |
+------------------+--------------------+------------------+-----------+
| **Persons served |                    |                  |           |
| or affected by   |                    |                  |           |
| the program**    |                    |                  |           |
+------------------+--------------------+------------------+-----------+
| -   Patients of  | -   May fear or    | -   Providing    | -         |
|     the TB       |     reject         |     customer     |    Survey |
|     clinic       |     clinic/health  |     perspective  |           |
|                  |     system         |                  |           |
|                  |                    | -   Interpreting |           |
|                  | -   Want better    |     findings     |           |
|                  |     and accessible |                  |           |
|                  |     services       |                  |           |
+------------------+--------------------+------------------+-----------+
| -   The foreign  | -   May be         | -   Providing    | -         |
|     born         |     suspicious of  |     community    |    Inform |
|     community in |     outsiders to   |     context      |     of    |
|     general      |     neighborhoods, |                  |           |
|                  |     especially if  |                  |  findings |
|                  |     illegal        |                  |           |
|                  |     immigration is |                  |           |
|                  |     an issue       |                  |           |
+------------------+--------------------+------------------+-----------+
| -   Other        | -   May or may not | -                | -         |
|     service      |     be coping with |    Disseminating |    Inform |
|     programs in  |     similar        |     findings to  |     of    |
|     the area     |     challenges     |     community    |           |
|     (employers,  |                    |     audiences    |  findings |
|     hospitals)   |                    |                  |           |
+------------------+--------------------+------------------+-----------+
| **Intended users |                    |                  |           |
| of evaluation    |                    |                  |           |
| findings**       |                    |                  |           |
+------------------+--------------------+------------------+-----------+
| -   Libby        | -   To show        | -   Defining     | -         |
|                  |     effectiveness  |     information  |    Direct |
|                  |                    |     needed from  |     role  |
|                  | -   Use findings   |     the          |     in    |
|                  |     to enhance the |     evaluation   |     c     |
|                  |     program        |                  | onducting |
|                  |                    | -   Developing   |     e     |
|                  |                    |     and          | valuation |
|                  |                    |     implementing |           |
|                  |                    |                  |           |
|                  |                    |  recommendations |           |
+------------------+--------------------+------------------+-----------+
| -   County       | -   Know if the    | -   Providing    | -   Don,  |
|     Health       |     program is     |                  |           |
|                  |     effective:     |  administrative/ |    direct |
|    Commissioners |     best use       |     funding      |     role; |
|                  |     county funds   |     context      |           |
| -   Clinical     |                    |                  |    others |
|     staff        | -   Provide        | -   Interpreting |           |
|                  |     effective and  |     findings     |   through |
|                  |     acceptable     |                  |     a     |
|                  |     treatment and  | -   Interpreting |           |
|                  |     care           |     findings     |   meeting |
|                  |     interventions  |                  |           |
|                  |     for TB         | -   Modifying    | -         |
|                  |                    |     practice (if |   Meeting |
|                  |                    |     needed)      |           |
+------------------+--------------------+------------------+-----------+
| -   CBOs and     | -   Improve        | -                | -         |
|     community    |     community      |    Disseminating |    Inform |
|     planning     |     health , well  |     findings     |     of    |
|     board        |     being          |                  |           |
|                  |                    |                  |  findings |
| > \- Carlos      |                    |                  |           |
+------------------+--------------------+------------------+-----------+

**II. BACKGROUND AND DESCRIPTION OF THE TB PROGRAM AND PROGRAM LOGIC
MODEL**

**Need**

A culturally competent TB prevention program such as TB Support Program
is needed to combat the rise of TB incidence among the Salvadoran
immigrant community of Lull County. Newly arrived persons from a high TB
incidence country to Lull County are a major concern. Persons with TB
disease and infection are not being reached for timely treatment.
Consequently, TB cases are found on hospital admission after extensive
health damage has occurred and community transmission demonstrated. Some
newly discharged TB patients are lost to follow-up. Treatment cost
increases along with the rising morbidity, ill-health, and suffering of
the community members. The need for immediate attention to TB is great.
If the problem is left unattended, backlash against foreign born
population is possible.
![](media/image2.wmf){width="0.3402777777777778in"
height="0.5041666666666667in"}

**Context**

Historically, Lull County has been a low TB incidence area. Recent surge
in the number of TB cases has raised concerns in the community. TB
prevalence among the large number of newly arrived immigrants was fairly
high. Although the need for TB elimination and prevention in this
community is great, the capacity and resources needed to address the
magnitude of these surging TB issues in Lull County is limited. TB
Support program, an outreach program using "lay health advisors" (LHAs)
to target new immigrants in the community, is implemented in response to
this need. Under TB Support program, Spanish-speaking lay health
advisors are to provide directly observed therapy (DOT) and social
support in the immigrant community to increase targeted testing and
treatment adherence and promote early case finding in the community.

**Target Population**

The target population for TB Support Program is newly arrived Salvadoran
immigrants who cannot speak English, have limited resources, a small
pool of family and friends if any at all, and may possibly have legal
problems with their immigration status.

**Objectives**

The goal of TB Support Program is to eliminate TB transmission in the
Salvadoran community, improve quality of life, and ultimately, to
eliminate TB in Lull County. In order to achieve this overarching goal,
program objectives were identified for the first year of the program.

1.  Increase percentage of newly arrived, foreign born Salvadoran TB
    patients who adhere to treatment to 100% by December 2003.

2.  Increase the number of newly arrived, foreign born Salvadoran
    patients being treated for TB or LTBI by December 2003.

3.  Increase percentage of Salvadoran community members who understand
    what to do if someone has a productive cough for 3 weeks or more by
    March of 2004.

Because the program and problem are new, specific baselines, i.e., how
much of an increase can be expected, is unknown.

**Stage of Development**

The program is relatively new; in operation for less than a year.

**Resources/Inputs**

Lay heath advisors (LHAs), TB staff, funding, community based
organizations and community health centers serving the Salvadoran
community are key inputs of TB Support program.

**Activities**

Lay health advisors (LHAs) training, community outreach/education, TB
screening/testing, prescribing treatment, and DOT visits make up the
major activity categories.

**Outputs**

As a result of the efforts done in TB Support Program, LHAs were hired
and trained, counseling provided, TB outreach/education conducted, test
conducted, referrals made, a treatment plan developed, and DOT was
administered.

**Outcomes**

A list of short-term outcomes include: patient's knowledge increased,
trust is built, patients accept treatment for TB or LTBI, patients
identify contacts and possible source of cases, and patients use TB and
other services. Intermediate outcomes: patients adhere to treatment,
complete treatment, reduce hospital admissions for TB among Salvadorans,
and reduce transmission of TB. Long-term outcomes: Salvadoran patients
are healthier; eliminate TB in Salvadoran community, and healthier
community.![](media/image2.wmf){width="0.32916666666666666in"
height="0.49444444444444446in"}

+-----+------+------+--------+-----------+---------------+------------+
| *   |      |      |        |           |               |            |
| *Ta |      |      |        |           |               |            |
| ble |      |      |        |           |               |            |
| 3.  |      |      |        |           |               |            |
| P   |      |      |        |           |               |            |
| roj |      |      |        |           |               |            |
| ect |      |      |        |           |               |            |
| De  |      |      |        |           |               |            |
| scr |      |      |        |           |               |            |
| ipt |      |      |        |           |               |            |
| ion |      |      |        |           |               |            |
| of  |      |      |        |           |               |            |
| TB  |      |      |        |           |               |            |
| S   |      |      |        |           |               |            |
| upp |      |      |        |           |               |            |
| ort |      |      |        |           |               |            |
| Pro |      |      |        |           |               |            |
| gra |      |      |        |           |               |            |
| m** |      |      |        |           |               |            |
+-----+------+------+--------+-----------+---------------+------------+
| *   | **   |      | **Out  | ##        |               |            |
| *Re | Acti |      | puts** |           |               |            |
| sou | viti |      |        | ##        |               |            |
| rce | es** |      |        |  Outcomes |               |            |
| s** |      |      |        |           |               |            |
+-----+------+------+--------+-----------+---------------+------------+
|     | **I  | **   |        | **Sho     | **Mid-term**  | **L        |
|     | niti | Subs |        | rt-term** |               | ong-term** |
|     | al** | eque |        |           |               |            |
|     |      | nt** |        |           |               |            |
+-----+------+------+--------+-----------+---------------+------------+
| L   | Hi   | LHAs | LHAs   | Provide   | Increase      | Eliminate  |
| HAs | ring | trai | hired  | Spanish   | utilization   | TB in      |
| of  | LHAs | ning | &      | speaking  | of TB         | Salvadoran |
|     |      |      | t      | and       | services by   | community  |
| S   |      |      | rained | c         | Salvadoran    |            |
| alv |      |      |        | ulturally | Community     |            |
| ado |      |      |        | competent |               |            |
| ran |      |      |        | services  |               |            |
| com |      |      |        | for       |               |            |
| mun |      |      |        | S         |               |            |
| ity |      |      |        | alvadoran |               |            |
|     |      |      |        | community |               |            |
| TB  |      |      |        |           |               |            |
| St  |      |      |        |           |               |            |
| aff |      |      |        |           |               |            |
|     |      |      |        |           |               |            |
| F   |      |      |        |           |               |            |
| und |      |      |        |           |               |            |
| ing |      |      |        |           |               |            |
|     |      |      |        |           |               |            |
| Com |      |      |        |           |               |            |
| mun |      |      |        |           |               |            |
| ity |      |      |        |           |               |            |
| ba  |      |      |        |           |               |            |
| sed |      |      |        |           |               |            |
| o   |      |      |        |           |               |            |
| rga |      |      |        |           |               |            |
| niz |      |      |        |           |               |            |
| ati |      |      |        |           |               |            |
| ons |      |      |        |           |               |            |
| s   |      |      |        |           |               |            |
| erv |      |      |        |           |               |            |
| ing |      |      |        |           |               |            |
| S   |      |      |        |           |               |            |
| alv |      |      |        |           |               |            |
| ado |      |      |        |           |               |            |
| ran |      |      |        |           |               |            |
| com |      |      |        |           |               |            |
| mun |      |      |        |           |               |            |
| ity |      |      |        |           |               |            |
+-----+------+------+--------+-----------+---------------+------------+
|     | C    | E    | TB     | Increase  |               |            |
|     | ommu | duca | ou     | TB        |               |            |
|     | nity | tion | treach | knowledge |               |            |
|     | outr |      | and    | in        |               |            |
|     | each |      | edu    | S         |               |            |
|     |      |      | cation | alvadoran |               |            |
|     |      |      | con    | community |               |            |
|     |      |      | ducted |           |               |            |
+-----+------+------+--------+-----------+---------------+------------+
|     |      |      | Coun   | Trust     |               |            |
|     |      |      | seling | built     |               |            |
|     |      |      | &      | between   |               |            |
|     |      |      | s      | health    |               |            |
|     |      |      | upport | care      |               |            |
|     |      |      | pr     | providers |               |            |
|     |      |      | ovided | and the   |               |            |
|     |      |      |        | S         |               |            |
|     |      |      |        | alvadoran |               |            |
|     |      |      |        | community |               |            |
+-----+------+------+--------+-----------+---------------+------------+
|     | TB   | R    | T      | Early TB  | Reduce TB     |            |
|     | sc   | efer | esting | and LTBI  | transmission  |            |
|     | reen | rals | done   | detection |               |            |
|     | ing/ |      | and    | and       | Increase      |            |
|     |      |      | ref    | inte      | completion of |            |
|     | tes  |      | errals | rventions | therapy rate  |            |
|     | ting |      | made   |           |               |            |
|     |      |      |        |           | Patient       |            |
|     |      |      |        |           | complete      |            |
|     |      |      |        |           | treatment     |            |
|     |      |      |        |           |               |            |
|     |      |      |        |           | (reduce       |            |
|     |      |      |        |           | hospital      |            |
|     |      |      |        |           | admissions    |            |
|     |      |      |        |           | for TB among  |            |
|     |      |      |        |           | Salvadorans)  |            |
+-----+------+------+--------+-----------+---------------+------------+
|     |      | Pre  | Tre    | Patients  |               |            |
|     |      | scri | atment | identify  |               |            |
|     |      | bing | plan   | contacts  |               |            |
|     |      | t    |        | and       |               |            |
|     |      | reat | dev    | possible  |               |            |
|     |      | ment | eloped | source    |               |            |
|     |      |      |        | cases     |               |            |
|     |      |      |        |           |               |            |
|     |      |      |        | Patients  |               |            |
|     |      |      |        | accept    |               |            |
|     |      |      |        | treatment |               |            |
|     |      |      |        | for TB    |               |            |
|     |      |      |        | and LTBI; |               |            |
+-----+------+------+--------+-----------+---------------+------------+
|     |      | DOT  | DOT    | Patients  |               |            |
|     |      | vi   | admini | adhere to |               |            |
|     |      | sits | stered | treatment |               |            |
+-----+------+------+--------+-----------+---------------+------------+

**Logic Model**

![](media/image4.wmf)

**III. FOCUS OF THE EVALUATION**

**Stakeholder Needs**

From the beginning, four groups of people were identified as the core
users of the evaluation findings. They will be using the findings in
different ways and for different purposes. The table below summarizes
the users of this evaluation, what information they need or would like
to get from the evaluation, and how they intend to use that information
to achieve what they need or set out to do.
![](media/image2.wmf){width="0.3402777777777778in"
height="0.5041666666666667in"}

+------------------+---------------------------+-----------------------+
| **Users**        | **Need/Want to Know**     | **Uses**              |
+------------------+---------------------------+-----------------------+
| County           | -   Whether the program   | -   Determine whether |
| administrators   |     is working or not     |     the program       |
|                  |                           |     should be funded  |
|                  |                           |     to continue or    |
|                  |                           |     expand its        |
|                  |                           |     services          |
+------------------+---------------------------+-----------------------+
| Program Manager  | -   How to enhance or     | -   Implement change  |
|                  |     refine the program    |     to increase       |
|                  |                           |     effectiveness of  |
|                  |                           |     the program       |
+------------------+---------------------------+-----------------------+
| Clinical staff   | -   Clinical outcomes     | -   Adjust clinical   |
|                  |                           |     practice if       |
|                  |                           |     needed            |
+------------------+---------------------------+-----------------------+
| CBO              | -   Social health         | -   Advocacy to       |
| representatives  |     outcomes              |     community         |
+------------------+---------------------------+-----------------------+

**Evaluation Questions**

Although the evaluation team generated many possible questions in a
brainstorming session, the evaluation team prioritized the following as
representing the most important aspects of the program that could be
examined at this
time.![](media/image2.wmf){width="0.32916666666666666in"
height="0.49444444444444446in"}

To determine if the program has been implemented as planned:

-   Has appropriate (Spanish-speaking with a good understanding of
    Salvadoran culture) staff been recruited?

-   Has the staff (all TB program employees) been trained appropriately
    (in TB practice and cultural competency)?

To determine if the program is meeting its objectives:

-   Have more Salvadoran persons been tested and treated appropriately
    (clinically sound, in Spanish, in a culturally sensitive manner) for
    LTBI or TB?

-   Are Salvadoran patients adhering to treatment (not lost to care and
    follow-up)?

**Evaluation Design**

The evaluation team decided to use multiple data sources because the
program was small (few number of people served), the need to obtain
information to improve the program was high, and in-depth information
was needed. No control or comparison group seemed appropriate. The team
discussed reviewing records for Salvadoran patients in the past and
comparing these data to current program information, but realized that
it would be difficult since data in the files were incomplete. The
decision was made to assess the data against the benchmarks the program
has set. Further, it was decided that the data collected for this
evaluation would serve as a baseline for later evaluations.

**Resource Consideration**

Resource available for evaluation is limited. Manpower consists of 1
full time staff member who is only able to devote a few hours a week to
the evaluation. The other evaluation team members are able to devote
only a limited amount of time to the evaluation. However, several of the
evaluation team members work together and most stakeholder groups meet
regularly. Thus, some evaluation work can be done as part of existing
routines. Personnel records, training curriculum and clinical records
are available as existing data
sources.![](media/image2.wmf){width="0.32916666666666666in"
height="0.49444444444444446in"}

Additional data collection measuring patients' perception will be needed
to understand whether patients feel trust is built with their health
care providers and whether their TB knowledge has increased.

**Evaluation Standards**

The evaluation team agreed that the evaluation data will be useful, and
should be feasible to collect. Propriety will be addressed through
consent form and all data will be kept confidential. While the group
agreed that there may be some issues regarding record keeping or patient
willingness to answer honestly, the accuracy of the strategy was judged
to be acceptable.

**IV. GATHERING CREDIBLE EVIDENCE: DATA COLLECTION**

**Indicators**

The evaluation team recognized that the terms used the evaluation
questions, e.g., "appropriately," needed to be operationalized, and the
following list of indicators show the team's decisions. Since several
program objectives were vague, the evaluation team had to determine
benchmarks that seemed reasonable, given the limited information
available. Each member of the evaluation team informally talked with
other stakeholders to ensure consensus.

# 

+-------+---------------------------+----------------------------------+
|       | **Indicators**            | **Program Benchmarks**           |
+-------+---------------------------+----------------------------------+
| **Has |                           |                                  |
| a     |                           |                                  |
| pprop |                           |                                  |
| riate |                           |                                  |
| staff |                           |                                  |
| been  |                           |                                  |
| re    |                           |                                  |
| cruit |                           |                                  |
| ed?** |                           |                                  |
| (     |                           |                                  |
| Evalu |                           |                                  |
| ation |                           |                                  |
| Ques  |                           |                                  |
| tion) |                           |                                  |
+-------+---------------------------+----------------------------------+
|       | -   Number of qualified   | -   Two LHAs of different gender |
|       |     (bilingual/           |     are on staff, both speak     |
|       |                           |     Spanish and are familiar     |
|       | bi-cultural) staff        |     with the Salvadoran culture  |
+-------+---------------------------+----------------------------------+
| **Has |                           |                                  |
| the   |                           |                                  |
| staff |                           |                                  |
| been  |                           |                                  |
| tr    |                           |                                  |
| ained |                           |                                  |
| a     |                           |                                  |
| pprop |                           |                                  |
| riate |                           |                                  |
| ly?** |                           |                                  |
+-------+---------------------------+----------------------------------+
|       | -   LHAs received         | -   All staff attended training  |
|       |     appropriate/adequate  |                                  |
|       |     training in TB,       | -   Training covered essential   |
|       |     interviewing and DOT  |     topics                       |
|       |                           |                                  |
|       | -   Other clinic staff    |                                  |
|       |     were trained in       |                                  |
|       |     working with LHAs and |                                  |
|       |     TB competency         |                                  |
|       |     strategies for        |                                  |
|       |     Salvadorian culture   |                                  |
+-------+---------------------------+----------------------------------+
| *     |                           |                                  |
| *Have |                           |                                  |
| more  |                           |                                  |
| Salva |                           |                                  |
| doran |                           |                                  |
| pe    |                           |                                  |
| rsons |                           |                                  |
| been  |                           |                                  |
| tr    |                           |                                  |
| eated |                           |                                  |
| app   |                           |                                  |
| ropri |                           |                                  |
| ately |                           |                                  |
| for   |                           |                                  |
| LTBI  |                           |                                  |
| or    |                           |                                  |
| TB?** |                           |                                  |
+-------+---------------------------+----------------------------------+
|       | -   \# Salvadoran persons | -   Each month of program        |
|       |     treated by clinic for |     operations has shown an      |
|       |     TB or LTBI            |     increase in the number of    |
|       |                           |     Salvadoran patients          |
|       | -   Clinical treatment    |                                  |
|       |     standards             | -   Clinical standards met       |
|       |                           |                                  |
|       | -   Signs, forms          | -   Patient education signs and  |
|       |     available in Spanish  |     forms in Spanish             |
|       |                           |                                  |
|       | -   Clerks/staff know to  | -   Staff knows how to access    |
|       |     access                |     and use translators          |
|       |     translators/LHA       |                                  |
|       |                           | -   90% of Salvadoran patients   |
|       | -   Patient's attitude of |     report they believe what     |
|       |     trust for TB program  |     their provider tells them    |
|       |     staff recommendations |     about TB and trust their     |
|       |                           |     provider's recommendations.  |
+-------+---------------------------+----------------------------------+
| **Are |                           |                                  |
| Salva |                           |                                  |
| doran |                           |                                  |
| pat   |                           |                                  |
| ients |                           |                                  |
| adh   |                           |                                  |
| ering |                           |                                  |
| to    |                           |                                  |
| tr    |                           |                                  |
| eatme |                           |                                  |
| nt?** |                           |                                  |
+-------+---------------------------+----------------------------------+
|       | -   Attendance at DOT and | -   100% Salvadoran TB and LTBI  |
|       |     clinic visits         |     patients who miss an         |
|       |                           |     appointment are immediately  |
|       |                           |     rescheduled and keep         |
|       |                           |     appointment; no one is lost  |
|       |                           |     to care                      |
+-------+---------------------------+----------------------------------+

**Data Collection**

The evaluation utilizes several methods of data collection in various
ways: interview, survey, observation, and record review. Staff interview
will be conducted to assess knowledge and behavior in accessing
translators and LHAs. Written survey (in Spanish) will be administered
to the patients in the clinic waiting room to assess patients'
perception of the clinic (access to care, hours, acceptable
environment/staff) and their comfort level (attitudes and beliefs) about
the staff (a compromise due to limited funding). Observation will be
conducted to assess staff behaviors and language used in TB awareness
signs. Lastly, review of records, charts and training materials will be
conducted. Table 5 summarizes the data source and data collection
methodology that will be used to gather evidence for evaluation.

+---------------------+------------------+----------------------------+
| **Table 5. Data     |                  |                            |
| Collection Plan**   |                  |                            |
+---------------------+------------------+----------------------------+
| **Indicators**      | **Data source**  | **Data Collection Method** |
+---------------------+------------------+----------------------------+
| Number of qualified | -   LHA staff    | -   Observe presence of    |
| (bil                |                  |     staff                  |
| ingual/bi-cultural) | -   Personnel    |                            |
| staff               |     records      | -   Review records if      |
|                     |                  |     staff not present      |
|                     |                  |                            |
|                     |                  |  Carlos                   |
|                     |                  |                            |
|                     |                  |  Feb.4-10                 |
+---------------------+------------------+----------------------------+
| LHAs received       | -   Training     | -   Review records of      |
| training in TB,     |     attendance   |     training sessions and  |
| interviewing and    |     records      |     attendance, materials  |
| DOT                 |                  |     presented              |
|                     | -   Curriculum   |                            |
| Other clinic staff  |                  |  John                     |
| were trained in     |                  |                            |
| working with LHAs   |                  |  Feb.6 -15                |
| and cultural        |                  |                            |
| competency          |                  |                            |
+---------------------+------------------+----------------------------+
| \# Salvadorian      | -   Clinic       | -   Review records for     |
| immigrants treated  |     records,     |     country of origin and  |
| by clinic for TB or |     interviews   |     date of arrival in US, |
| LTBI                |     with staff   |     notes in file, ask     |
|                     |                  |     staff to identify from |
| Clinical treatment  | ```{=html}       |     client list            |
| standards (calendar | <!-- -->         |                            |
| of standard         | ```              |  Libby                    |
| appointment dates   | -   Clinic       |                            |
| based on initiation |     charts       |  Feb.8 -15                |
| of TB drugs)        |                  |                            |
|                     | -   Signs and    | -   Chart review           |
| Signs, forms        |     forms        |                            |
| available in        |                  |  TBA                      |
| Spanish             | -   Staff        |                            |
|                     |     knowledge    | -   Observation            |
| Clerks/staff know   |     and          |                            |
| to access           |     behaviors    |  Staff                    |
| translators/LHA     |                  |                            |
|                     | -   Patient      |  Feb.3 -25                |
| Patients feel they  |     perceptions  |                            |
| can trust the TB    |                  | -                          |
| program             |                  |    Interviews/observations |
|                     |                  |                            |
|                     |                  |  TBA                      |
|                     |                  |                            |
|                     |                  | -   Survey sample of       |
|                     |                  |     patients (forms and    |
|                     |                  |     drop box in clinic     |
|                     |                  |     waiting room, in       |
|                     |                  |     Spanish, no            |
|                     |                  |     identifiers asked)     |
|                     |                  |                            |
|                     |                  |  Feb. 1-28                |
+---------------------+------------------+----------------------------+
| Attendance at DOT   | -   Program      | -   Review records, noting |
| and clinic visits   |     visit        |     if any patients are    |
| (appointment system |     records      |     lost of care           |
| and missed          |                  |                            |
| appointment         |                  |  TBA                      |
| follow-up)          |                  |                            |
+---------------------+------------------+----------------------------+

**Plan Timeline**

**V. JUSTIFY CONCLUSIONS: ANALYSIS AND INTERPRETATION**

**Analysis**

Both quantitative and qualitative methods will be used to analyze the
data. Simple counts of frequency will be use for quantitative data
analysis. Qualitative methods such as content analysis will be used to
review training curriculum, patient charts and records for themes and
patterns.

  ------------------------------------------------ ----------------------
  **Table 7. Analysis Plan**                       

  **Data Analysis Technique**                      **Responsible Person**

  Quantitative -- frequency/counts                 Libby

  Qualitative -- content analysis                  Libby, Ana and staff
  ------------------------------------------------ ----------------------

**Interpretation**

Stakeholders including the county administrator, program manager/staff,
LHAs, medical staff and CBO representatives will be included in a
scheduled meeting to interpret the findings. The data from the
evaluation will be compared to the established program benchmarks.
Stakeholders and those involved in program operations will be given an
opportunity to justify the findings and make recommendations
accordingly.

**VI. ENSURE USE AND SHARE LESSONS LEARNED: REPORT & DISSEMINATION**

**Dissemination**

Evaluation finding will be disseminated via various channels.
Presentations will be given at the program staff meeting and to the
health care providers at regular staff meetings. A short report will be
drafted and a presentation offered for the county health commissioners.
An article will also be added to the newsletter the health department
periodically compiles. Carlos will also write an article to go in a
local Spanish newspaper and offer to present at community meetings.

**Use**

Libby, the program manager, and staff will use the findings to refine
program strategies for TB Support Program. The findings will help guide
the program to focus on areas that are most crucial for effective
service delivery. Health care professional will use the finding to
improve their medical practices in serving Salvadoran community. County
health department administrators, led by Don, will use the findings to
determine the future funding distribution for the program. The community
liaison, Carlos, will use the findings in advocacy efforts. Finally,
findings from this evaluation will be used for future evaluations.

**Appendix B**

**Writing SMART Objectives**

In order to be most effective, objectives should be clear and leave no
room for interpretation. S-M-A-R-T is a helpful acronym for developing
objectives that are *specific*, *measurable*, *achievable*, *relevant*,
and *time-bound*.

An example of an S-M-A-R-T objective for TB is:

*In County X, increase the percentage of adult patients with
non-rifampin -resistant TB who complete treatment in less than 12 months
(as measured by cohort review) from 80% to 90% (the national goal) by
2014.*

The objective is *specific* because it identifies a defined event: adult
TB patients will complete treatment in less than 12 months. The
objective is *measurable* because it specifies a baseline value and the
quantity of change the intervention is designed to achieve: from 80% to
90%. As in the example, it is worthwhile to note whether there is an
existing data source for the objective. The objective is *achievable*
because it is realistic given the 10-year time frame. The objective is
also *relevant* because it relates to the elimination of exposure to
non-resistant TB. Finally, the objective is *time-bound* because it
provides a specified time frame by which the objective will be achieved
(from 2004 to 2006).

A Tool to help write SMART objective

+----+------+-------+----------+---------+--------+--------+--------+
| >  | Incr |       |          |         |        |        |        |
|  * | ease |       |          |         |        |        |        |
| *O | pe   |       |          |         |        |        |        |
| bj | rcen |       |          |         |        |        |        |
| ec | tage |       |          |         |        |        |        |
| ti | of   |       |          |         |        |        |        |
| ve | a    |       |          |         |        |        |        |
| ** | dult |       |          |         |        |        |        |
|    | pati |       |          |         |        |        |        |
|    | ents |       |          |         |        |        |        |
|    | with |       |          |         |        |        |        |
|    | n    |       |          |         |        |        |        |
|    | on-r |       |          |         |        |        |        |
|    | esis |       |          |         |        |        |        |
|    | tant |       |          |         |        |        |        |
|    | TB   |       |          |         |        |        |        |
|    | who  |       |          |         |        |        |        |
|    | c    |       |          |         |        |        |        |
|    | ompl |       |          |         |        |        |        |
|    | eted |       |          |         |        |        |        |
|    | the  |       |          |         |        |        |        |
|    | rapy |       |          |         |        |        |        |
|    | (wi  |       |          |         |        |        |        |
|    | thin |       |          |         |        |        |        |
|    | 12   |       |          |         |        |        |        |
|    | m    |       |          |         |        |        |        |
|    | os.) |       |          |         |        |        |        |
|    | from |       |          |         |        |        |        |
|    | 80%  |       |          |         |        |        |        |
|    | to   |       |          |         |        |        |        |
|    | 90%  |       |          |         |        |        |        |
|    | by   |       |          |         |        |        |        |
|    | 2    |       |          |         |        |        |        |
|    | 006. |       |          |         |        |        |        |
+----+------+-------+----------+---------+--------+--------+--------+
| >  | **VE | **MET | **POPU   | **O     | **BASE | **     | *      |
|  * | RB** | RIC** | LATION** | BJECT** | LINE** | GOAL** | *TIMEF |
| *B |      |       |          |         |        |        | RAME** |
| re |      |       |          |         | **MEA  | **MEA  |        |
| ak |      |       |          |         | SURE** | SURE** |        |
| do |      |       |          |         |        |        |        |
| wn |      |       |          |         |        |        |        |
| ** |      |       |          |         |        |        |        |
+----+------+-------+----------+---------+--------+--------+--------+
|    | Incr | Pe    | Adult    | Com     | 80%    | 90%    | By     |
|    | ease | rcent | patients | pletion |        |        | 2006   |
|    |      |       | with     | of      |        |        |        |
|    |      |       |          | therapy |        |        |        |
|    |      |       | non-r    |         |        |        |        |
|    |      |       | esistant | (w/in   |        |        |        |
|    |      |       | TB       | 12 mos) |        |        |        |
+----+------+-------+----------+---------+--------+--------+--------+

**Appendix C: Tuberculosis Program Logic Models**

![](media/image5.png){width="8.871527777777779in"
height="6.2965277777777775in"}

![](media/image6.png){width="8.227777777777778in"
height="6.4944444444444445in"}

![](media/image7.png){width="8.35625in" height="6.492361111111111in"}

![](media/image8.png){width="8.3625in" height="6.492361111111111in"}

![](media/image9.png){width="8.127083333333333in"
height="6.491666666666666in"}

![](media/image10.png){width="8.904861111111112in"
height="6.498611111111111in"}

**Appendix D**

**Sample indicators**

• Proportion of verified TB cases reported to the local health
jurisdiction within 1 working day from treatment start date.

• Demonstrated ability (organization, staffing, resources, and
facilities) to carry out the core components of a TB control program.

• Number of TB cases identified per 100,000 people.

• Proportion of cases with complete data on key variables (homelessness,
injecting drug use, non-injecting drug use, excess alcohol use)

• Proportion of pulmonary or laryngeal TB cases \> 12 years of age with
sputum culture obtained.

• Proportion of TB cases started on the recommended 4-drug regimen.

• Proportion of sputum smear-positive pulmonary or laryngeal TB cases
initiating treatment in \< 7 days of specimen collection.

• Proportion of sputum culture-positive TB cases with documented
conversion to sputum culture-negative within 90 days of initiation of
treatment.

• Proportion of TB cases for whom DOT is recommended who receive DOT
throughout the course of treatment.

• Proportion of TB cases for whom DOT is recommended who receive
inappropriate SAT throughout the course of treatment.

• Proportion of TB cases who complete treatment in \< 12 months.

• Proportion of TB cases who default prior to completing treatment.

• Proportion of sputum smear-positive cases with at least one contact
identified.

• Proportion of identified contacts to smear-positive cases who complete
evaluation for TB infection or disease.

• Proportion of infected contacts to pulmonary cases who started
treatment for LTBI.

• Proportion of infected contacts to pulmonary cases started on
treatment for LTBI who complete treatment.

• Number of TB cases in children 0-4 years old.

• Number of persons who die with TB.

\*As developed by California's Tuberculosis Indicators Project