**Final Report**

**Programs That Work (PTW)**

**Focus on Kids and Safer Choices**

**Contract No. 200-96-0599**

**Task 20, Phases II and III**

**Evaluating National Dissemination Strategies**

**for Effective HIV Prevention Programs for Youth**

**to**

**Leah Robin, PhD, Technical Monitor**

**Stephen Banspach, PhD, Technical Monitor**

**Susan Cleveland, Project Officer**

**Cheryl Stauss, Contracting Officer**

**Centers for Disease Control and Prevention**

**1600 Clifton Road**

**Atlanta, Georgia 30333**

**November 27, 2002**

**from**

**Yuqiu Cheng, ABD**

**Leilani Francisco, MA**

**Martha L. Hare, PhD**

**Mary Odell Butler, PhD**

**Battelle**

**Centers for Public Health Research and Evaluation**

**2101 Wilson Boulevard, Suite 800**

**Arlington, Virginia 22201**

> *This report is a work prepared for the United States Government by
> Battelle. In no event shall either the United States Government or
> Battelle have any responsibility or liability for any consequences of
> any use, misuse, inability to use, or reliance on the information
> contained herein, nor does either warrant or otherwise represent in
> any way the accuracy, adequacy, efficacy, or applicability of the
> contents hereof.*

*ACKNOWLEDGMENTS*

*We would like to thank all those who supported this project through
their time and knowledge. Specifically, we would like to express our
appreciation to Leah Robin of the Division of Adolescent and School
Health (DASH), National Centers for Chronic Disease Prevention and
Health Promotion at CDC, Technical Monitor for the project, and her
co-Technical Monitor, Stephen Banspach. We would also like to thank
Louise Glezen, who played an important role in the data collection
process. Also, many thanks go to ETR Associates for their cooperation
and technical assistance provision. Finally, we would like to thank Mary
Odell Butler and Joanne Abed for reviewing draft reports.*

# Table of Contents {#table-of-contents .unnumbered}

[Table of Contents i](#table-of-contents)

Executive Summary iv

[1.0 Overview 1](#overview)

***Part I: Focus on Kids***

[2.0 Introduction 3](#introduction)

[3.0 Methods 4](#methods)

[3.1 Master Trainer Methods 4](#master-trainer-methods)

[Telephone Survey 4](#telephone-survey)

[Tracking 5](#tracking)

[Data Entry, Cleaning and Analyses 5](#data-entry-cleaning-and-analyses)

[Focus Group 5](#focus-group)

[3.2 Local Health Educator Methods 6](#local-health-educator-methods)

[Mail Survey 6](#mail-survey)

[Data Entry, Cleaning, and Analyses
8](#data-entry-cleaning-and-analyses-1)

[4.0 Results 9](#results)

[4.1 Master Trainer Results 9](#master-trainer-results)

[Telephone Survey 9](#telephone-survey-1)

[Focus Group 12](#focus-group-1)

[4.2 Local Health Educator Results 13](#local-health-educator-results)

[Mail Survey 13](#mail-survey-1)

Part II: Safer Choices

[5.0 Introduction 16](#introduction-1)

[6.0 Methods 17](#methods-1)

[6.1 Master Trainer Methods 17](#master-trainer-methods-1)

[Telephone Survey 17](#telephone-survey-2)

[Tracking 18](#tracking-1)

[Data Entry, Cleaning and Analyses
18](#data-entry-cleaning-and-analyses-2)

[Focus Group 19](#focus-group-2)

[6.2 Local Health Educator Methods 20](#local-health-educator-methods-1)

[Mail Survey 20](#mail-survey-2)

[Data Entry, Cleaning, and Analyses
21](#data-entry-cleaning-and-analyses-3)

[7.0 Results 22](#results-1)

[7.1 Master Trainer Results 22](#master-trainer-results-1)

[Telephone Survey 22](#telephone-survey-3)

[Focus Group 24](#focus-group-3)

[7.2 Local Health Educator Results 26](#local-health-educator-results-1)

[Mail Survey 26](#mail-survey-3)

[8.0 Conclusions](#results-1) 29

***List of
Appendices*...........................................................................................*31***

***Appendix 1 The Master Trainer Focus Group Report***

***Appendix 2 Focus on Kids Master Trainer (MT) Survey Documentation***

> *Appendix 2.1 The codebook (Word file)*
>
> *Appendix 2.2 The coding decision rules (Word file)*
>
> *Appendix 2.3 The attention items (Word file)*
>
> *Appendix 2.4 The MT survey one data (SAS file)*
>
> *Appendix 2.5 The MT survey two data (SAS file)*
>
> *Appendix 2.6 The concatenated MT survey one and two data (SAS file)*
>
> *Appendix 2.7 The open-ended response data (Excel file)*
>
> *Appendix 2.8 The frequency program (SAS file)*
>
> *Appendix 2.9 The frequency output (SAS file)*
>
> *Appendix 2.10 The ad hoc analysis program (SAS file)*
>
> *Appendix 2.11 The ad hoc analysis output (SAS file)*
>
> *Appendix 2.12 The value label program (SAS file)*

***Appendix 3 Focus on Kids Local Health Educator (LHE) SAQ
Documentation***

> *Appendix 3.1 The codebook (Word file)*
>
> *Appendix 3.2 The coding decision rules (Word file)*
>
> *Appendix 3.3 The attention items (Word file)*
>
> *Appendix 3.4 The data entry database (Access file)*
>
> *Appendix 3.5 The LHE data (SAS file)*
>
> *Appendix 3.6 The frequency program (SAS file)*
>
> *Appendix 3.7 The frequency output (SAS file)*
>
> *Appendix 3.8 The ad hoc analysis program (SAS file)*
>
> *Appendix 3.9 The ad hoc analysis output (SAS file)*
>
> *Appendix 3.10 The value label program (SAS file)*

***Appendix 4 Safer Choices Master Trainer (MT) Survey Documentation***

  *Appendix 4.1 The codebook (Word file)*

> *Appendix 4.2 The coding decision rules (Word file)*
>
> *Appendix 4.3 The prominent coding decisions (Word file)*
>
> *Appendix 4.4 The attention items (Word file)*
>
> *Appendix 4.5 The MT survey one data entry database (Access file)*
>
> *Appendix 4.6 The MT survey two data entry database (Access file)*
>
> *Appendix 4.7 The MT survey one data (SAS file)*
>
> *Appendix 4.8 The MT survey two data (SAS file)*
>
> *Appendix 4.9 The concatenated MT survey one and two data (SAS file)*
>
> *Appendix 4.10 The frequency program (SAS file)*
>
> *Appendix 4.11 The frequency output (SAS file)*
>
> *Appendix 4.12 The ad hoc analysis program (SAS file)*

*Appendix 4.13 The ad hoc analysis output (SAS file)*

*Appendix 4.14 The value label program (SAS file)*

***Appendix 5 Safer Choices Local Health Educator (LHE) SAQ
Documentation***

*** ** Appendix 5.1 The codebook (Word file)*

> *Appendix 5.2 The coding decision rules (Word file)*
>
> *Appendix 5.3 The attention items (Word file)*
>
> *Appendix 5.4 The data entry database (Access files)*
>
> *Appendix 5.5 The LHE data (SAS file)*
>
> *Appendix 5.6 The frequency program (SAS file)*
>
> *Appendix 5.7 The frequency output (SAS file)*
>
> *Appendix 5.8 The ad hoc analysis program (SAS file)*
>
> *Appendix 5.9 The ad hoc analysis output (SAS file)*
>
> *Appendix 5.10 The value label program (SAS file*)

***Appendix 6 Instruments: Surveys and Questionnaires***

> *Appendix 6.1 The Focus on Kids Master Trainer Survey One Instrument
> (original)*
>
> *Appendix 6.2 The Focus on Kids Master Trainer Survey One Instrument
> (final)*
>
> *Appendix 6.3 The Focus on Kids Master Trainer Survey Two Instrument
> (original and final)*
>
> *Appendix 6.4 The Focus on Kids Local Health Educator
> Self-Administered Questionnaire (original)*
>
> *Appendix 6.5 The Focus on Kids Local Health Educator
> Self-Administered Questionnaire (final)*
>
> *Appendix 6.6 The Safer Choices Master Trainer Survey One Instrument
> (original)*
>
> *Appendix 6.7 The Safer Choices Master Trainer Survey One Instrument
> (final)*
>
> *Appendix 6.8 The Safer Choices Master Trainer Survey Two Instrument
> (original)*
>
> *Appendix 6.9 The Safer Choices Master Trainer Survey Two Instrument
> (final)*
>
> *Appendix 6.10 The Safer Choices Local Health Educator
> Self-Administered Questionnaire (original)*
>
> *Appendix 6.11 The Safer Choices Local Health Educator
> Self-Administered Questionnaire (final)*

# Executive Summary {#executive-summary .unnumbered}

**Title:** **Evaluation National Dissemination Strategies for Effective
HIV Prevention Programs for Youth**

**CDC Contract Number: 200-96-0599, Task 20**

**Sponsor: Centers for Disease Control and Prevention (CDC)**

**2920 Brandywine Road**

**Atlanta, Georgia 30341**

**Contractor: Battelle Memorial Institute**

**Centers for Public Health Research and Evaluation**

**2101 Wilson Blvd., Suite 800**

**Arlington, VA 22201**

#### Statement of the Problem

> The purpose of this project was to assist the Division of Adolescent
> and School Health (DASH), National Centers for Chronic Disease
> Prevention and Health Promotion (NCCDPHP) to conduct an evaluation of
> dissemination strategies to increase the adoption and implementation
> of HIV prevention programs for youth in the United States. This
> Program Operations evaluation will focus on the ongoing Centers for
> Disease Control and Prevention (CDC) *Programs That Work* project.
> Specifically, the dissemination of two curricula, *Focus on Kids* and
> *Safer Choices*, were evaluated.
>
> **Background**
>
> CDC developed the *Programs That Work* project in response to requests
> from educators for effective programs that reduce health risk
> behaviors among youth. The purposes of the project are to: (1)
> identify interventions with credible evidence of effectiveness in
> reducing risk behaviors, (2) disseminate *Programs That Work* from the
> national level to local community and school levels, and (3) encourage
> state and local education and health agencies, community-based
> organizations, and national non-governmental organizations to adopt
> research-based prevention programs and teaching methods. Currently,
> *Programs That Work* are disseminated through a "training of trainers"
> model. State teams of master trainers, who are training experts and
> administrators from state and local health and education agencies,
> attend a two-and-a-half-day national training on a "Program that
> Works". This national training provides master trainers with the
> skills to train a cadre of local health educators on a "Program that
> Works"; local health educators in turn implement the program with
> young people in their communities. Besides the national training, the
> master trainers from some states also received a one-and-a-half-day
> site visit, where they were provided additional training using a
> dissemination package. The master trainers who received additional
> site visits were therefore in an enhanced condition, whereas the
> master trainers who did not receive additional site visits were in the
> standard condition.

#### Evaluation Objectives

This evaluation has three objectives:

-   Determine if the dissemination of effective prevention programs for
    youth can be enhanced by providing enhanced technical assistance
    involving the use of a dissemination planning guide (Phase II only).

-   Identify the determinants of successful and unsuccessful site
    implementations of the *Programs That Work* curricula (HIV
    prevention).

-   Determine the cost-effectiveness of providing dissemination
    trainings to promote *Programs That Work* (Battelle is responsible
    for data collection, but not analysis).

#### III. Methodology {#iii.-methodology .unnumbered}

> This evaluation adopted both quantitative and qualitative research
> methods. For each of the curriculum, data were collected from two
> sources: master trainers and local health educators. Telephone surveys
> were conducted with master trainers at two points of time. In
> addition, the master trainers were also invited to participate in a
> focus group to provide more qualitative data. Mail surveys were sent
> to the local health educators as the major means of data collection
> with this group. To increase response rate, Battelle used a variety of
> methods including tracking, incentives, reminder, multiple mailing,
> and so forth.

##### Methodology for the Master Trainer Telephone Survey  {#methodology-for-the-master-trainer-telephone-survey .unnumbered}

> The telephone survey was administered by project staff who were
> trained in telephone survey administration procedures. A variety of
> procedures were implemented to assure high completion rates,
> confidentiality and valid answers. For example, Battelle contacted
> each master trainer before the telephone survey to arrange for the
> best time to administer the survey, and followed up with each master
> trainer to assure its completion. The survey questionnaire includes
> questions that provide contact information that will assist project
> staff in locating them for later survey and focus group data
> collection. Information requested is limited to name, place of
> employment, and job title. Each participant was informed about
> voluntary participation and confidentiality. Written or verbal consent
> was obtained from each participant prior to the telephone survey. Two
> waves of telephone survey were conducted. Typically, one was conducted
> approximately 6-months after the national training, and the other one
> was conducted 12-months after the training. The telephone surveys were
> administered using Computer-assisted Telephone Interviewing (CATI)
> methods. Battelle also adopted a series of rigorous methods to ensure
> that the data were entered and cleaned appropriately. Any coding
> decisions were noted in the coding decision log. For data analyses,
> SAS programs were used to conduct frequency analyses, create summary
> variables, and manage datasets.

##### Methodology for the Master Trainer Focus Group  {#methodology-for-the-master-trainer-focus-group .unnumbered}

> The focus groups were held in order to better understand the context
> in which the dissemination of curricula and training occur. Each focus
> group was homogenous with respect to the condition (enhanced or
> standard, Phase II only). Each participant was asked to join only one
> of the teleconference focus groups. The telephone focus groups were
> held shortly after the completion of the second wave of surveys for
> master trainers. The participants were also informed about voluntary
> participation and confidentiality. Written or verbal consent was
> obtained from each participant prior to the telephone focus group.
> Upon completion of each group discussion, a taped record was sent to a
> transcriptionist who provided Battelle with a verbatim account of the
> focus group. At this point, the focus group moderator analyzed each
> tape using NUD\*IST software by searching for each of the themes and
> organizing the information according to those themes.

##### Methodology for Local Health Educator Mail Survey  {#methodology-for-local-health-educator-mail-survey .unnumbered}

> Further data were collected among local personnel (local health
> educators) who attended regional or local trainings provided by master
> trainers during the one-year follow-up period. Data were collected to
> measure local adoption and implementation of the programs. In the
> mailing of the survey, Battelle provided the LHEs with an 800
> telephone number for a project staff member who had been trained on
> the survey and could answer respondent questions and concerns about
> it. As an incentive to complete and return the SAQ, each local health
> educator received an abstinence promotion poster produced by ETR
> Associates. The local health educators were also informed about
> voluntary participation and confidentiality. A signed and completed
> written consent was obtained from each respondent. The respondents
> sent back the consent form and the survey in one mail. Battelle staff
> separated the consent form and the survey so that they cannot be
> associated in any way.

#### Major Findings and Recommendations

##### Focus on Kids

##### Findings from the Master Trainer Telephone Survey  {#findings-from-the-master-trainer-telephone-survey .unnumbered}

> A total of 14 master trainers participated in the first round of
> survey and 15 master trainers participated in the second round.
> According to the master trainers who participated in the survey,
> during 12 months after the national training, four local trainings had
> been held which involved a total of 205 local health educators. No
> local training was held between 12 months and 18 months after the
> national training. Most master trainers claimed that they used more
> than one way to get the word out about the program. The most
> frequently used methods were presentations/briefings and networking.
> Most master trainers also said that they informed multiple agencies of
> the curriculum. The most frequently mentioned agencies/groups that had
> been informed were community organizations and school teachers/health
> educators. In general, fewer dissemination activities had been
> reported in the second round of the survey than in the first round.
> According to master trainers, multiple agencies agreed to support the
> implementation of the curriculum. The most frequently mentioned
> agencies/groups that agreed to support the implementation of the
> curriculum were state education agency and community agencies.
> Finally, it was found that not many young people had been taught by
> master trainers directly. Only two master trainers had taught the
> program to young people and a total of 63 young people were taught.

##### 

##### Findings from the Master Trainer Focus Group {#findings-from-the-master-trainer-focus-group .unnumbered}

> Of the fifteen master trainers that participated in the 12- and
> 18-month surveys, a total of 7 participated in the focus groups. The
> focus group with master trainers in the enhanced condition consisted
> of 5 participants, but only 2 master trainers from the standard
> condition participated in a focus group. In general, respondents had a
> positive regard for Focus on Kids and for the Programs That Work in
> general, but expressed many concerns about implementing the programs,
> particularly in school settings, because it had been designed for a
> community setting. The intervention in the form of an enhancement -- a
> 1 ½ day site visit from ETR Associates -- did not seem to have an
> impact on dissemination and training activities. This is likely
> because the majority of the sites' dissemination and training
> activities had been completed prior to the site visit enhancement.

#####  {#section-1 .unnumbered}

##### Findings from the Local Health Educator Mail Survey {#findings-from-the-local-health-educator-mail-survey .unnumbered}

> One hundred and twenty-one local health educators participated in the
> survey. Most local health educators (LHEs) reported that they used
> more than one way to provide information to others about the
> curriculum. The most frequently used methods were one-to-one meetings,
> presentations or briefings and networking. In addition, most LHEs
> informed more than one group/agency. The most frequently mentioned
> agencies/groups that had been informed of the program were school
> teachers or health educators and school principals. According to LHEs,
> most of them were required to obtain permissions from more than one
> agency/group. The top 3 agencies/groups mentioned by LHEs from which
> they were required to obtain permission were school principals,
> students' parents, and local education agencies. It was also found
> that an average of 3.1 agencies agreed to support the implementation
> of the curriculum. The top three agencies/groups that agreed to
> support the program were school teachers/health educators, school
> principals, and students' parents. Finally, 33% of the LHEs said that
> they had taught the curriculum to young people directly. Among these
> LHEs, the number of young people taught by each LHE ranged from 5 to
> 438, with a total of 1,486 young people taught and an average of 59
> young people taught by each LHE.

##### Safer Choices

##### Findings from the Master Trainer Telephone Survey {#findings-from-the-master-trainer-telephone-survey-1 .unnumbered}

#####   {#section-2 .unnumbered}

> A total of 26 master trainers participated in the first round of
> survey and 25 master trainers participated in the second round.
> According to the master trainers who participated in the survey,
> during 6 months after the national training, four local trainings had
> been held which involved a total of 65 local health educators. Between
> 6 months and 12 months after the national training, one local training
> with 8 local health educators was held. Most master trainers claimed
> that they used more than one way to get the word out about the
> program. The most frequently used methods were
> presentations/briefings, networking, and one-to-one meeting. Most
> master trainers also said that they informed multiple agencies of the
> curriculum. The most frequently mentioned agencies/groups that had
> been informed were school teachers/health educators and local educator
> agency. In general, fewer dissemination activities had been reported
> in the second round of the survey than in the first round. According
> to master trainers, multiple agencies agreed to support the
> implementation of the curriculum. The most frequently mentioned
> agencies/groups that agreed to support the implementation of the
> curriculum were state education agency and local education agencies.
> Finally, it was found that very few young people had been taught by
> master trainers directly. Only 1 master trainer implemented all 5
> components, 1 implemented 3 components, and 4 implemented 1 component
> with young people.

##### 

##### Findings from the Master Trainer Focus Group {#findings-from-the-master-trainer-focus-group-1 .unnumbered}

> Of the 26 master trainers that participated in the 6- and 12-month
> surveys, a total of 12 participated in focus groups. In general, the
> master trainers had a positive regard for Safer Choices and for the
> Programs That Work in general, but expressed many concerns about
> implementing Safer Choices in particular, due its large scope and the
> resources that would be required to implement it with fidelity. While
> they mentioned that it is difficult to implement such a large program
> with numerous components, they also felt that this was one of the
> strengths of the program. The various components enable the program to
> offer a comprehensive package. In general, they felt that the training
> could be shortened, that a local assessment piece could prove to be
> fruitful, and that further research is needed, particularly on the
> outcomes related to youth attitude and behavior change.

#####  {#section-4 .unnumbered}

##### Findings from the Local Health Educator Mail Survey {#findings-from-the-local-health-educator-mail-survey-1 .unnumbered}

> Forty-four local health educators participated in the survey. Most
> local health educators reported that they used more than one way to
> provide information to others about the curriculum. The most
> frequently used methods were one-to-one meetings, presentations or
> briefings and networking. In addition, most LHEs informed more than
> one group/agency. The most frequently mentioned agencies/groups that
> had been informed of the program were school teachers or health
> educators, school principals, and students' parents. According to
> LHEs, most of them were required to obtain permissions from more than
> one agency/group. The top 3 agencies/groups mentioned by LHEs from
> which they were required to obtain permission were school principals,
> students' parents, and local education agencies. It was also found
> that an average of 3.8 agencies agreed to support the implementation
> of the curriculum. The top three agencies/groups that agreed to
> support the program were school principals, students' parents, and
> school board. Finally, 76% of the LHEs said that they had implemented
> one or more components of the curriculum with young people directly. A
> total of 8,132 young people were taught with an average of 301 young
> people by each LHE who answered the question.

#### Conclusions

> To conclude, the dissemination of *Focus on Kids* and *Safer Choices*
> had been a success: multiple local trainings had been held;
> information of the curricula had been delivered to a variety local
> agencies/groups; and a large number of young people had been taught.
> However, there are still some barriers to the implementation of the
> program that can be addressed. The curricula may need to be tailored
> in order to better suit the needs of varying audiences. In addition, a
> cost-effectiveness analysis is needed to further determine the success
> of the programs.

# 1.0 Overview {#overview .unnumbered}

The Division of Adolescent and School Health (DASH), National Centers
for Chronic Disease Prevention and Health Promotion (NCCDPHP), is
conducting an evaluation of dissemination strategies to increase the
adoption and implementation of HIV prevention programs for youth in the
United States. This Program Operations evaluation will focus on the
ongoing Centers for Disease Control and Prevention (CDC) *Programs That
Work* project.

CDC developed the *Programs That Work* project in response to requests
from educators for effective programs that reduce health risk behaviors
among youth. CDC has been directed by Congress to make *Programs That
Work* a prevention program priority (*Congressional Record* 10/19/98,
H11392). Since 1992, the project has identified eight *Programs That
Work* to reduce HIV-related risk behaviors among youth. The purposes of
the project are to: (1) identify interventions with credible evidence of
effectiveness in reducing risk behaviors, (2) disseminate *Programs That
Work* from the national level to local community and school levels, and
(3) encourage state and local education and health agencies,
community-based organizations, and national non-governmental
organizations to adopt research-based prevention programs and teaching
methods. The project aims to enhance the adoption and implementation of
effective HIV prevention curricula in our nation's schools. As such, it
is directly related to the CDC FY 2000 performance plan to (a) reduce
the incidence of HIV/AIDS through the dissemination of HIV prevention
education programs, (b) reduce the percentage of high school students
who have ever engaged in sexual intercourse by 10% from a 1995 baseline
of 53%, and (c) reduce the percentage of currently sexually active high
school students who engage in sexual intercourse without a condom by 10%
from a 1995 baseline of 46%.

Currently, *Programs That Work* are disseminated through a "training of
trainers" model. State teams of master trainers, who are training
experts and administrators from state and local health and education
agencies, attend a two-and-a-half-day national training on a "Program
that Works". This national training provides master trainers with the
skills to train a cadre of local health educators on a "Program that
Works"; local health educators in turn implement the program with young
people in their communities.

In August 2000, CDC contracted with Battelle to conduct an evaluation of
the dissemination of *Programs That Work*. It follows participants from
two national *Programs That Work* trainings for HIV prevention in fiscal
year 1999 (Focus on Kids) and fiscal year 2001 (Safer Choices). The goal
of this evaluation is to improve national, state, and local training
programs that promote the adoption of effective health curricula
identified by the *Programs That Work* project such as *Focus on Kids*
and *Safer Choices*. The aim is not to assess the effectiveness of
individual programs in changing the behaviors of youth, but rather to
assess the extent to which programs are disseminated.

This evaluation has three objectives:

-   Determine if the dissemination of effective prevention programs for
    youth can be enhanced by providing enhanced technical assistance
    involving the use of a dissemination planning guide.

-   Identify the determinants of successful and unsuccessful site
    implementations of the *Programs That Work* curricula (HIV
    prevention).

-   Determine the cost-effectiveness of providing dissemination
    trainings to promote *Programs That Work*.

The results of this evaluation will provide valuable information to
improve the effectiveness of DASH-funded trainings on *Programs That
Work*. Results will be used to help improve the content of trainings by
adding or adjusting activities aimed at increasing implementation rates
in the states and cities. In addition, the results will provide cost
information to better understand the benefits of this training model to
public health in the United States.

The evaluation of the two curricula was conducted separately in two
phases. From August 2000 to August 2001, Battelle evaluated the
dissemination of *Focus on Kids*, and from September 2001 to December
2002, Battelle focused on the dissemination of *Safer Choices*.

There are two parts of this report. In Part I, details regarding the
evaluation of *Focus on Kids* will be presented, and Part II will focus
on the evaluation of *Safer Choices*. Each part begins with a brief
introduction of the curriculum, followed by the methods used for
evaluation. In the methods section, we will introduce the instruments
and procedures adopted; tracking methods used; data entry, cleaning and
analysis procedure applied, and barriers encountered and how they were
resolved. The methods used for data collection with master trainers and
local health educators will be discussed separately. The results section
also consists of two components, that is, results on master trainer data
and results on local health educator data. Master trainer data include
both survey data and focus group data, and local health educator data
are from survey data only. For survey data, we will first introduce the
sample characteristics and response rate, followed by the major findings
of the frequency analyses. For focus group data, sample characteristics
and response rate will be described first, followed by a brief
description of the findings of the qualitative analyses. More detailed
information regarding the focus groups can be found in Appendix 1.

# 2.0 Introduction {#introduction .unnumbered}

*Focus on Kids* has been designated as a "Program that Works" by the
Centers for Disease Control and Prevention (CDC), meaning that it shows
evidence of leading to positive behavior change among young people as
demonstrated in a rigorous outcome evaluation. *Focus on Kids* is a
research-based intervention program that seeks to bring university-based
research to the community. The goal of the program is to reduce the risk
of HIV infection among urban youth. The program is rooted in an
educational philosophy that supports the idea that youths can protect
themselves from HIV infection if they are provided with the knowledge
and skills to do so. The program seeks to reach higher-risk youth by
focusing on community organizations as vehicles for reaching youth as
opposed to school settings. The program endorses a curriculum comprised
of eight sessions focused on: team building and cohesion, risks and
values, self-education (ways to obtain information and examine
consequences), skills building, sexual health information provision,
attitudes and skills for sexual health, and building self-efficacy.

Master trainers attended a national training on *Focus on Kids* in
October 1999. This training qualified them to train local health
educators and to teach young people. They also were encouraged to
develop action plans for the dissemination of the curriculum (e.g., ways
of reaching target groups, strategies for approaching those responsible
for implementing programs in schools).

Four of the seven states that attended the training received an
additional site visit from ETR Associates[^1]. During the
one-and-a-half-day site visit, the master trainers were provided
additional training using a dissemination package. Master trainers from
these four states were therefore in the enhanced group. The remaining
states were assigned to the standard group. The selection of the four
out of seven states was on a random basis.

This evaluation answers three questions:

-   Can the process of adopting and implementing new prevention
    curricula in schools be accelerated through detailed action planning
    activities and enhanced follow-up technical assistance to those
    trained?

-   How are the barriers to adoption addressed effectively by those
    successfully adopting HIV prevention *Programs That Work*?

-   How cost-effective is the training process in the adoption and
    implementation of *Programs That Work*?

# 3.0 Methods {#methods .unnumbered}

## 3.1 Master Trainer Methods {#master-trainer-methods .unnumbered}

### Telephone Survey {#telephone-survey .unnumbered}

All master trainers (standard and enhanced) completed a telephone survey
12 months and 18 months after attending a national training in *Focus on
Kids* curriculum. Data were collected to measure adoption of the
programs at two different time points. The minimum standard for adequate
state-level adoption is one regional or local training provided, or a
planned regional or local training, including a date on which it is to
be held and list of attendees, one year after the national training. For
master trainers, the primary outcome to be measured was the number of
local and regional personnel trained on the *Programs That Work*.
Secondary outcome measures included: costs associated with training,
amount of informational materials distributed, the number of conferences
or events in which the *Programs That Work* is discussed, and type and
number of administrative agreements to allow training on the program at
regional and local levels. The state-level context for adoption and
implementation of *Programs That Work* was assessed by telephone survey
questions about state standards or guidelines for the content of HIV
risk prevention curricula and the amount of time allocated to teach
them, and through focus group interviews.

The telephone survey was administered by project staff who were trained
in telephone survey administration procedures. A variety of procedures
were implemented to assure high completion rates, confidentiality and
valid answers. For example, Battelle contacted each master trainer
before the telephone survey to arrange for the best time to administer
the survey, and followed up with each master trainer to assure its
completion. The survey questionnaire includes questions that provide
contact information that will assist project staff in locating them for
later survey and focus group data collection. Information requested is
limited to name, place of employment, and job title. Each participant
was informed about voluntary participation and confidentiality. Written
or verbal consent was obtained from each participant prior to the
telephone survey.

The national training was held on October 8, 1999. The first wave of the
master trainer telephone survey was conducted between November 2000 and
February 2001. The interval between the first wave of the telephone
survey and the national training ranged from 13 months (416 days) to 16
months (482 days) with an average of 14 months (433 days). The second
wave of the master trainer telephone survey was conducted between April
2001 and May 2001. The interval between the first and second waves of
the master trainer survey ranged from 3 months (93 days) to 7 months
(212 days) with an average of 5 months (152 days). The interval between
the national training and the second wave of survey ranged from 18
months (565 days) and 20 months (599 days) with an average of 19 months
(580 days).

In January 2001, the Task Leader contacted the two master trainers who
had voiced a complaint during the survey. Both of them expressed
frustration with their own inability to implement the training for a
broader audience due to lack of support for the program in their states.
They felt that the survey did not really address all of these issues.
Both of the respondents said they would continue to participate in the
study and completed the second round of the survey.

###  {#section-5 .unnumbered}

### Tracking {#tracking .unnumbered}

In January 2001, the tracker telephoned those trainers who had been
conducting trainings in the first half of the school year. At that
point, no new trainings had been conducted. The tracker telephoned 6
trainers in February and 13 trainers in March. Again, at that point, no
new trainings had been conducted. No preliminary tracking calls were
made prior to the second round of master trainer CATI interviews.

###  {#section-6 .unnumbered}

### Data Entry, Cleaning and Analyses {#data-entry-cleaning-and-analyses .unnumbered}

Both surveys with *Focus on Kids* master trainers were administered
using Computer-assisted Telephone Interviewing (CATI) methods.
Interviewers recorded respondent answers on hardcopies of the survey
instruments. These instruments were then submitted to the editing
department for editing, coding, and cleaning. The editing department
worked closely with the primary data cleaner who documented any coding
decisions in a coding decision log. The cleaned data was then entered
into an Excel spreadsheet and sent to the primary data cleaner, who
conducted a follow-up cleaning of the data, again noting any coding
decisions in the coding decision log.

Frequency analyses have been conducted for all variables. A number of
summary variables have also been generated to group related items and to
capture the activities that had been conducted by master trainers to
disseminate the curriculum. SAS program has been used to conduct all
statistical analyses and data management.

### Focus Group {#focus-group .unnumbered}

In addition, master trainers were invited to participate in a focus
group. The focus groups were held in order to better understand the
context in which the dissemination of curricula and training occur.
Specifically, the following topics were covered:

-   Dissemination Process

-   Barriers

-   Enablers

-   Recommendations

-   Participation in Dissemination Study

    -   Enhanced Site Visit (enhanced group only)

    -   Participation in Survey

-   Other

Each focus group was homogeneous with respect to the condition (enhanced
or standard). Each participant was asked to join only one of the
teleconference focus groups. The telephone focus groups were held
shortly after completion of the second wave of surveys for master
trainers. Participants were also informed about voluntary participation
and confidentiality. Written or verbal consent was obtained from each
participant prior to the telephone focus group.

The focus group for members of the enhanced condition was held on May
30, 2001. A total of five people attended representing three states. The
focus group for members of the standard condition was held on June 25,
2001. Due to scheduling difficulties, only two people attended, each
from a different state. However, the information they shared was very
rich.

Upon completion of each group discussion, a taped record was sent to a
transcriptionist who provided Battelle with a verbatim account of the
focus group. At this point, the focus group moderator analyzed each tape
using NUD\*IST software by searching for each of the themes noted above
and organizing the information according to those themes.

##  {#section-7 .unnumbered}

## 3.2 Local Health Educator Methods {#local-health-educator-methods .unnumbered}

### Mail Survey  {#mail-survey .unnumbered}

Further data were collected among local personnel (local health
educators or LHEs) who attended regional or local trainings provided by
master trainers during the one-year follow-up period. Data were
collected to measure local adoption and implementation of the programs.
Implementation is the actual use of the program in youth group settings.
The primary outcome measured is the number of youth reached through
program implementation. Secondary outcome measures include: costs
associated with implementation, number and type of administrative
agreements to allow local implementation of the programs, number of
contacts with master trainers for problem-solving of adoption and
implementation issues, and amount of informational materials distributed
locally. Regional and local contexts for adoption and implementation of
*Programs That Work* were assessed through mailed survey questions about
school and community policies regarding the content and time allotted
for HIV prevention curricula, as well as community resources currently
devoted to risk reduction for HIV.

In the mailing of the Self-Administered Questionnaire (SAQ), Battelle
also provided the LHEs with an 800 telephone number for a project staff
member who had been trained on the survey and could answer respondent
questions and concerns about it. As an incentive to complete and return
the SAQ, each local health educator received, in addition to the SAQ, an
abstinence promotion poster produced by ETR Associates. The local health
educators were also informed about voluntary participation and
confidentiality. A signed and completed written consent was obtained
from each respondent. The respondents sent back the consent form and the
survey in one mailing. Battelle staff separated the consent form and the
survey so that they could not be associated in any way.

Master trainers identified 205 local health educators that they trained.
All 205 comprised the population for inclusion in the study.

The first mailing of local health educator surveys was sent out on May 7
and May 8, 2001. During May 2001, a concern surfaced regarding
increasing the response rate for local health educators at the end of
the school year. It was decided not to use reminder postcards but to go
directly to the next step of our follow-up protocol and to send a second
mailing to all non-responders before the end of May. Therefore, the
second mailing of the surveys was sent out on May 29 and May 30, 2001.

In June 2001, Battelle staff trained the interviewers to conduct
telephone follow-up and interviewing of local health educators. Through
a combination of telephone interviewing and mailed SAQs, Battelle was
able to reach additional local health educators. However, Battelle and
CDC continued to be concerned about the low response rate from the local
health educators. In the latter half of June, Battelle focused on
educators who were employed outside of school systems (i.e., year-round
employees). It was decided that Battelle would re-contact those
educators who had not yet responded to the SAQ and were employed in
schools during the month of September. The interviewers also discovered
that, despite a clear request in the introductory letter regarding our
wish to hear from everyone regardless of whether they had trained, many
educators did not believe that the survey applied to them if they had
not been training. Battelle agreed to make the need for response from
all members of the sample clearer in materials for Phase 3.

In September 2001, Battelle began contacting local health educators not
reached in June 2001. Follow-up packages were mailed on September 18,
2001. By the end of September 28, two additional surveys were received
completed.

In early October 2001, a decision was made to end local health educator
follow-ups by October 18. After October 18, on the 22^nd^, an additional
questionnaire was received in the mail, bringing the total completed to
121 and the completion rate to 59.0%.

One minor issue related to the packaging of local health educator
materials arose and was resolved during April 2001. Due to the bulkiness
of the packaging, a decision was made to fold the poster incentives
(versus shipping them in a rolled format) and to place them in flat
envelopes along with the cover letter and survey.

### Data Entry, Cleaning, and Analyses {#data-entry-cleaning-and-analyses-1 .unnumbered}

Upon receipt, completed self-administered questionnaires were submitted
to the editing department for editing, coding, and cleaning. The editing
department worked closely with the primary data cleaner, who documented
any coding decisions in a coding decision log. The cleaned data were
then entered into an Access database. The Access database provided
quality control advantages due to its user-friendly data entry forms and
structural alignment with hardcopy questionnaires. The database was then
sent to the primary data cleaner, who imported the data into an Excel
spreadsheet and conducted a follow-up cleaning of the data, again noting
any coding decisions in the coding decision log.

Frequency analyses have been conducted for all variables. A number of
summary variables have also been generated to group related items and to
capture the activities that had been conducted by local health educators
to disseminate the curriculum. SAS program has been used to conduct all
statistical analyses and data management.

# 4.0 Results  {#results .unnumbered}

## 4.1 Master Trainer Results {#master-trainer-results .unnumbered}

### Telephone Survey {#telephone-survey-1 .unnumbered}

***Sample characteristics and response rate.*** Seventeen master
trainers (MTs) comprised the population for inclusion in the surveys.
Two were determined to be ineligible. One of these MTs became ill and
was no longer working in his/her position as a trainer. The other MT
left his/her position at his/her job for personal reasons. Fourteen out
of the fifteen eligible MTs completed Survey One, yielding a response
rate of 93.3%. The other master trainer who did not participate in
Survey One insisted that her/his responses exactly replicated the
responses of another master trainer, one with whom s/he works closely.

These 14 master trainers were from 7 different states: California,
Louisiana, Minnesota, Nevada, North Carolina, Oregon, and Washington.
Fifty percent (50%, n=7) of them were in the enhanced group[^2], and the
other 50% were in the standard group (n=7). Seventy-one percent (71%) of
the participants were female (n=10), and 29% were male (n=4). For all 14
participants, we obtained their consent for participating in the study.
The length of the interview ranged from 12 minutes to 40 minutes, with
an average of 21 minutes.

Forty-three percent (43%) of the participants were working for state
education agencies (n=6), 22% were working for community agencies (n=3),
14% for state health agencies (n=2), 7% for local/district education
agencies (n=1), 7% for public/private/alternative schools (n=1), and 7%
for other types of organizations (n=1).

Twenty-one percent (21%) of the participants were state education agency
administrators (n=3), 21% were state-level master trainers (n=3), 14%
were community agency administrators (n=2), 7% were local/district
education agency administrators (n=1), 7% were community agency staff
(n=1), and the other 30% had other types of roles or functions in their
organizations (n=4).

Fifty percent (50%) of the participants had more than 10 years of
experience in providing HIV prevention training to health educators or
to young people (n=7), 14% had eight to ten years of experience (n=2),
22% had five to seven years of experience (n=3), and the other 14% had
two to four years of experience (n=2).

For the second round of the survey (18 months after the training), all
15 master trainers participated in the survey, yielding a response rate
of 100%. Among these participants, 53% of them (n=8) were in the
enhanced group, and the other 47% were in the standard group (n=7). The
length of the interview ranged from 8 minutes to 21 minutes, with an
average of 11 minutes.

***Frequency analyses**.* Frequency analyses have been conducted for the
data collected at the first round (12 months after the national
training), at the second round (18 months after national training), and
both rounds aggregated, respectively. Only major findings are reported
in this section (Table 4.1).

It was found that within 12 months after the national training, four
local trainings had been held, which involved a total of 205 local
health educators in 4 states. Specifically, one training was held in
State10 (ID#10), where 57 local health educators were trained. One
training was held in State20 (ID#20), and 71 local health educators
participated in the training. One training was held in State30 (ID#30),
where 46 local health educators participated. Finally, 31 local health
educators participated in the training held in State50 (ID#50). Master
trainers from State10 and State20 were in the enhanced group, and master
trainers from State30 and State50 were in the standard condition.
Therefore, 128 local health educators were trained under the enhanced
condition, and the remaining 77 were trained under the standard
condition.

However, over the next six months (between 12 months after the national
training and 18 months after the training), no local training was held.
When asked whether and how they provided people with information about
*Focus on Kids* after the national training, each master trainer
answered that s/he used more than one way to get the word out about the
program within 12 months after the training (min=2, max=9,
mean=5.1).[^3] The three most frequently used methods were presentations
or briefings (93% of master trainers used this method), networking
(86%), and informal workshops (79%). Between 12 months and 18 months
after training, these activities had decreased, but most master trainers
still used a variety of methods to provide people information about the
program (min=0, max=7, mean=3.1). The three most frequently used methods
were presentations or briefings (73%), networking (67%), and one-to-one
meetings (60%).

When asked if the master trainers informed different agencies/groups
(e.g., state education agency, state health agency, etc.) about the
program, most master trainers said that they informed multiple agencies
within 12 months after the training (min=1, max=11, mean=6.3).[^4] The
top three agencies/groups that had been informed were local education
agencies (86%), community organizations (86%), and school teachers or
health educators (79%). Although the trend had decreased after 12
months, most master trainers still informed more than one agency (min=1,
max=9, mean=4.4). Between 12 months and 18 months after the national
training, the top three agencies/groups that had been informed were
community organizations (60%), school teachers or health educators
(60%), and HIV prevention community planning groups (53%).

The master trainers were also asked if the agencies/groups they informed
of the program agreed to support the implementation of the curriculum.
It was found that within 12 months after the training, an average of 3.6
agencies agreed to support the implementation of the curriculum (min=0,
max=7). The four most frequently mentioned agencies/groups that agreed
to support implementation of the curriculum were state education
agencies (50%), community agencies (50%), state health agencies (36%),
and young people (36%). Between 12 months and 18 months after the
training, an average of 2.9 agencies/groups (min=0, max=7) agreed to
support implementation. The four most frequently mentioned
agencies/groups that agreed to support implementation of the curriculum
were state education agencies (53%), school teachers or health educators
(47%), local health agencies (33%), and community agencies (33%).

The master trainers were asked whether they had directly taught *Focus
on Kids* to young people, how many times they had taught the entire
program, and how many young people they had taught. It was found that
within 12 months after the training, 1 master trainer had taught the
entire program to young people once, and 1 master trainer had taught
twice. In total, 63 young people had been taught the curriculum by
master trainers directly. However, between 12 months and 18 months after
the training, no master trainers had taught the program to young people
directly.

Table 4.1 Major Findings from Master Trainer Telephone Survey for Focus
on Kids

  ---------------------------- ------- ------- ------- -------- ------- -------
                               First                   Second           
                               Wave                    Wave             

                               Min     Max     Mean    Min      Max     Mean

  Number of ways used to       2       9       5.1     0        7       3.1
  provide people with                                                   
  information about *Focus on                                           
  Kids*                                                                 

  Number of agencies informed  1       11      6.3     1        9       4.4
  about *Focus on Kids*                                                 

  Number of agencies agreed to 0       7       3.6     0        7       2.9
  support *Focus on Kids*                                               
  ---------------------------- ------- ------- ------- -------- ------- -------

###  {#section-8 .unnumbered}

### Focus Group  {#focus-group-1 .unnumbered}

***Sample characteristics and response rate***. Of the 15 master
trainers that participated in the 12- and 18-month surveys, a total of 7
participated in focus groups. The focus group with master trainers in
the enhanced condition consisted of 5 participants, but only 2 master
trainers from the standard condition participated in a focus group. Of
the remaining 8 master trainers who did not participate in focus groups,
3 refused and 5 could not participate due to scheduling conflicts.
Although we were unable to coordinate the timing of a focus group with
these latter 5 master trainers, it is important to note that they were
extremely cooperative, and attempted to schedule more than once. The
other 3 "refusers" also cited time constraints as the main reason for
their inability to participate.

The focus group data were also examined in terms of response by
responder's characteristics, i.e., type of organization, role at
organization, years of experience. Differences in responses by these
characteristics were not evident nor did they emerge from the data
through careful analysis.

***Findings.*** The focus groups provided much useful information on the
dissemination of HIV prevention programs, such as *Focus on Kids.*
Detailed information regarding the focus group can be found in
Appendix 1. Only a summary of the findings will be presented in this
section.

In general, respondents had a positive regard for *Focus on Kids* and
for the *Programs That Work* in general, but expressed many concerns
about implementing the programs, particularly in school settings.

Both within and across groups there were individual differences in the
degree of local autonomy in curriculum choice, and in the degree to
which local districts needed to adhere to an abstinence-only message.
Even so, schools generally took a conservative stance and avoided
addressing condom use, in particular condom demonstrations, even where
such discussions were allowed. The process of selecting and
disseminating curricula in school districts was cumbersome, and
sometimes including comprehensive sexuality education would entail
special hearings. This is one reason why *Focus on Kids* was considered
to be better suited to community settings or alternative schools, and
not to mainstream public schools.

Respondents shared many concerns, as well as solutions and
recommendations. They felt that the curriculum was generally a good one,
but that it was difficult to implement with fidelity because it had been
designed for a community setting. Respondents also shared that the
process of rolling out *Programs That Work* can be quite cumbersome and
advocated for more advance information about what kinds of curricula
were being considered, and a more streamlined training program that
could be attended by more people who would actually be doing the
training. The recommendation for streamlining the training applied to
both national and local trainings.

Some concerns go beyond the scope of the current study. For example,
respondents pointed out that educators need certain skills for
implementing programs like *Focus on Kids* that may be lacking among the
pool of educators being trained. Master trainers and state staff can
supply technical assistance, but there is a limit to what they can
reasonably offer.

Apparently the enhanced intervention did not have an impact upon
dissemination and training activities, mainly because the bulk of
activities in the enhanced states occurred before the intervention took
place. Rather, differences between the enhanced and standard groups were
linked to differences in the individual states.

##  {#section-9 .unnumbered}

## 4.2 Local Health Educator Results {#local-health-educator-results .unnumbered}

### Mail Survey  {#mail-survey-1 .unnumbered}

***Sample characteristics and response rate.*** Contact information on
205 local health educators was provided by 12 master trainers (MT) using
rosters. One hundred and twenty-one local health educators sent back the
survey, yielding a response rate of 59%. Only one person of the
remaining 84 LHEs refused to participate in the survey. Thirteen survey
packages were returned undeliverable. We were not able to locate 29
people and never heard back from them. Battelle attempted to invite the
remaining 41 people whom we were able to locate to complete the survey
over the telephone. For those people we were able to reach by telephone,
none of them agreed to complete the survey over the telephone. For those
people we were unable to reach, we left a message on their voicemail,
and none of them called us back and none of them returned the mail
survey. Consent was obtained from all the LHEs who participated in the
survey.

All these local health educators who participated in the survey were
trained by multiple master trainers jointly. Specifically, 39 local
health educators were trained by 5 master trainers from State10 (ID#10)
and State40 jointly, 38 LHEs were trained by 2 MTs from State20 jointly,
24 LHEs were trained by 3 MTs from State30, and the other 20 LHEs were
trained by 2 MTs from State50. Among these 121 LHEs, 64% of them (n=77)
were in the enhanced condition, and 36% (n=44) were in the standard
condition.

Among all the participants,[^5] 62% were working for schools
(public/private/ alternative) (n=75), 13% for community agencies (n=16),
19% for health agencies (n=23), 2% for other education
agencies/organizations (n=2), and 4% for other government agencies
(n=4).

Thirty-one percent (31%) of the participants were classroom teachers
(n=37), 7% were teacher supervisor/curriculum specialist (n=8), 4% were
community agency administrators (n=5), 15% were community agency staff
(e.g., youth worker, recreation leader) (n=18), 4% were other
administrators (e.g., school administrator, administrative assistant,
state agency administrator) (n=5), 10% were school nurses (n=12), 1% was
AIDS program staff (n=1), 12% were counselors, social workers, or
therapists (n=15), 12% were other educators (e.g., student assistant,
librarian, para-educator) (n=14), and the other 4% of LHEs had other
types of role or function in their organizations (n=6).

Among all the participants, 18% had more than 10 years of experience
providing HIV prevention training to health educators or to young people
(n=22), 4% had eight to ten years of experience (n=5), 12% had five to
seven years of experience (n=14), 29% had two to four years of
experience (n=35), 20% had one year or less experience (n=24), and the
other 17% had no experience (n=21).

***Frequency analyses.*** Frequency analyses have been conducted for the
data collected from local health educator survey. Only major findings
are reported in this section (Table 4.2).

The local health educators were asked whether they provided other people
with information about the *Focus on Kids* program since they attended
the training and what methods they had used to provide information. Most
local health educators reported that they used more than one way to
provide information to others about the curriculum (min=0, max=6,
mean=2.0).[^6] The three methods used most frequently were one-to-one
meetings (56% of the LHEs used this method), presentations or briefings
(50%), and networking (36%).

The local health educators were also asked whether and how many
agencies/groups they had informed of the program since the training.
Most LHEs informed more than one group or agency (min=0, max=9,
mean=2.7).[^7] The top four agencies/groups informed about the program
were school teachers or health educators (55%), school principals (37%),
community organizations (31%), and students' parents (31%).

The LHEs were asked whether they were required to obtain permission to
implement the program from different agencies/groups. It was found that
most LHEs were required to obtain permission from more than one
agency/group (min=0, max=9, mean=3.0). [^8] The top three
agencies/groups mentioned by LHEs from which they were required to
obtain permission were school principals (44%), students' parents (37%),
and local education agencies (36%).

The LHEs were also asked if the agencies they informed about the program
agreed to support implementation of the curriculum. It was found that an
average of 3.1 agencies agreed to support implementation of the
curriculum (min=0, max=9). The top three agencies/groups that agreed to
support the program were school teachers or health educators (27%),
school principals (26%), and students' parents (24%).

The LHEs were asked if they had directly taught *Focus on Kids* to young
people, how many times they had taught it, and how many young people
they had taught. Sixty-seven percent (67%, n=81) of the LHEs said that
they did not teach the program to young people, 21% (n=25) said that
they had taught the whole program to young people, and 11% (n=13) said
that they did not teach the whole program, but taught parts of the
program to young people. Ten percent (10%, n=12) of the LHEs said they
had taught the program to young people once, 7% (n=8) said that they had
taught the program twice, 1% (n=1) had taught it 5 times, 1% (n=1) had
taught it 10 times, and 1% (n=1) had taught it 11 times. Finally, for
those LHEs who had taught young people directly, the number of young
people taught by each LHE ranged from 5 to 438, with a total of 1,486
young people taught and an average of 59 young people taught by each
LHE.

Table 4.2 Major Findings from Local Health Educator Survey for Focus on
Kids

  ---------------------------------------------- ------------ -------- --------
                                                 LHE Survey            
                                                 Statistics            

                                                 Min          Max      Mean

  Number of ways used to provide people with     0            6        2.0
  information about *Focus on Kids*                                    

  Number of agencies informed about *Focus on    0            9        2.7
  Kids*                                                                

  Number of agencies whose permission is needed  0            9        3.0
  to implement *Focus on Kids*                                         

  Number of agencies agreed to support *Focus on 0            9        3.1
  Kids*                                                                
  ---------------------------------------------- ------------ -------- --------

# 5.0 Introduction {#introduction-1 .unnumbered}

*Safer Choices* is a more recently identified "Program That Works" than
*Focus on Kids*. *Safer Choices* is a research-based program targeted
towards high school youth. The program was developed with the intent of
becoming a permanent element of school curricula. *Safer Choices* is
focused on preventing HIV, other sexually transmitted diseases (STDs)
and pregnancy. The program is comprised of five major components: school
organization, curriculum and staff development, peer resources and
school environment, parent education, and school-community linkages. The
curriculum is made up of 20 lessons, 10 to be implemented with ninth
grade students, and 10 be implemented with 10^th^ grade students. It
provides information on: HIV and other STDs, condoms and other methods
of protection, norms and other influences on teen sexual behaviors,
risks, decision-making, negotiation and refusal skills, impacts of
HIV/AIDS and pregnancy, media influences, and community resources for
HIV, STD, and pregnancy testing.

Master trainers attended a national training on *Safer Choices* in May
2001, which qualified them to train local health educators and to teach
young people. The master trainers were also encouraged to develop action
plans for the dissemination of the curriculum.

All 12 states that participated in the training are in the standard
group. This evaluation intended to answer two questions:

-   How and how well was this new prevention curriculum disseminated?

-   What are the barriers and enablers to successful dissemination?

*Safer Choices* is a more complicated program with more components,
compared to *Focus on Kids*. Therefore, the evaluation of this
curriculum has specific questions regarding the implementation of each
component of the curriculum.

# 6.0 Methods {#methods-1 .unnumbered}

##  {#section-10 .unnumbered}

## 6.1 Master Trainer Methods {#master-trainer-methods-1 .unnumbered}

###  {#section-11 .unnumbered}

### Telephone Survey  {#telephone-survey-2 .unnumbered}

All Master trainers completed a telephone survey six months and one year
after attending a national training in *Safer Choices* curriculum. Data
were collected to measure adoption of the programs at two different time
points. The minimum standard for adequate state-level adoption is one
regional or local training provided, or a planned regional or local
training, including a date on which it is to be held and a list of
attendees, one year after the national training. For master trainers,
the primary outcome to be measured was the number of local and regional
personnel trained on the *Programs That Work*. Secondary outcome
measures included: costs associated with training, amount of
informational materials distributed, the number of conferences or events
in which the *Programs That Work* is discussed, and type and number of
administrative agreements to allow training on the program at regional
and local levels. The state-level context for adoption and
implementation of *Programs That Work* was assessed by telephone survey
questions about state standards or guidelines for the content of HIV
risk prevention curricula, and the amount of time allocated to teach
them, and through focus group interviews.

The telephone survey was administered by project staff who were trained
in telephone survey administration procedures. A variety of procedures
were implemented to assure high completion rates, confidentiality, and
valid answers. For example, Battelle contacted each master trainer
before the telephone survey to arrange for the best time to administer
the survey, and followed up with each master trainer to assure its
completion. The survey questionnaire includes questions that provide
contact information to assist project staff in locating them for later
survey and focus group data collection. Information requested is limited
to name, place of employment, and job title. Each participant was
informed about voluntary participation and confidentiality. Written or
verbal consent was obtained from each participant prior to the telephone
survey.

The national training of *Safer Choices* was held on May 21, 2001. The
first wave of the master trainer telephone survey was conducted between
November 2001 and January 2002. The interval between the first wave of
the telephone survey and the national training ranged from 6 months (179
days) to 8 months (236 days) with an average of 6 months (193 days). The
second wave of the master trainer telephone survey was conducted between
April 2002 and June 2002. The interval between the first and second
waves of the master trainer survey ranged from 4 months (134 days) to 7
months (210 days) with an average of 6 months (173 days). The interval
between the national training and the second wave of the survey ranged
from 11 months (337 days) and 13 months (389 days) with an average of 12
months (366 days).

### Tracking  {#tracking-1 .unnumbered}

In November 2001, the project team decided to individualize the tracking
procedures. Those who had already trained would be called in early
December 2001; those who had not yet trained but planned to would be
called in February 2002; and those who did not intend to train would be
called in February as well, with the exception of one master trainer who
said she did not want to be tracked because she will never train.

Tracking with these master trainers began on January 23, 2002. Ten
master trainers had trained prior to survey one, and an additional four
planned to train in January 2002. Thus, a total of 14 master trainers
required tracking calls. By the end of January, attempts were made to
reach all 14 master trainers; 8 of them were reached. No rosters had
been received by the end of January.

In February 2002, the interviewers learned that some master trainers had
concerns about sending rosters without further explanation as to the
purpose of sharing this information. The task leader telephoned the
master trainers in question at the end of the reporting period. She
discovered that three states would like a letter from CDC further
explaining the project. It was decided that CDC would prepare the letter
and Battelle would disseminate it in order to preserve the
confidentiality of the master trainers. On March 15, Battelle
disseminated a letter to those master trainers or supervisors who had
requested one from CDC. The letter provided information on the purpose
of the LHE survey in order to apprise master trainers and supervisors as
to the reason why we were requesting rosters to contact the trainees. By
the end of March, CDC had received one roster from a master trainer
since the mailing of the letter.

In February 2002, Battelle discovered that in one state, there was
disagreement among trainers as to whether the program offered was merely
an information workshop or a training. Through follow-up calls the task
leader discovered that in the state in question, the program delivered
was intended for informational sharing and not to train educators on how
to deliver the *Safer Choices* program. Appropriate corrections were
made to the data and noted in the coding decision log.

### Data Entry, Cleaning and Analyses {#data-entry-cleaning-and-analyses-2 .unnumbered}

Both surveys with *Safer Choices* master trainers were administered
using Computer-assisted Telephone Interviewing (CATI) methods.
Interviewers recorded respondent answers on hardcopies of the survey
instruments. These instruments were then submitted to the editing
department for editing, coding, and cleaning. The editing department
worked closely with the primary data cleaner who documented any coding
decisions in a coding decision log. The cleaned data were then entered
into an Access database. The Access database provided quality control
advantages due to its user-friendly data entry forms and structural
alignment with hardcopy surveys. The database was then sent to the
primary data cleaner who imported the data into an Excel spreadsheet and
conducted a follow-up cleaning of the data, again noting any coding
decisions in the coding decision log.

Frequency analyses have been conducted for all variables. A number of
summary variables have also been generated to group related items and to
capture the activities that had been conducted by master trainers to
disseminate the curriculum. SAS program has been used to conduct all
statistical analyses and data management.

### Focus Group {#focus-group-2 .unnumbered}

In addition, master trainers were invited to participate in a focus
group. The focus groups were held in order to better understand the
context in which the dissemination of curricula and training occur.
Specifically, the following topics were covered:

-   Dissemination Process

-   Barriers

-   Enablers

-   Recommendations

-   Participation in Dissemination Study

    -   Enhanced Site Visit (enhanced group only)

    -   Participation in Survey

-   Other

Each participant was asked to join only one of the teleconference focus
groups. The telephone focus groups were held shortly after the
completion of the second wave of surveys for master trainers.
Participants were also informed about voluntary participation and
confidentiality. Written or verbal consent was obtained from each
participant prior to the telephone focus group.

Three focus groups were held. The first was held on May 29, 2002.
Another group was held on September 5, and a final focus group on
September 13, 2002. Six master trainers participated in the first focus
group, representing the states of Indiana, Iowa, Kansas, Massachusetts,
Maine, and New York. The second focus group was comprised of three
master trainers, representing Kansas and Texas. The final group was also
comprised of three master trainers representing Florida and New York.
Two states were represented across groups: Kansas and New York.

Upon completion of each group discussion, a taped record was sent to a
transcriptionist who provided Battelle with a verbatim account of the
focus groups. The primary focus group moderator then analyzed the data
through the use of a qualitative analysis software package, QSR N6. The
transcripts were imported into N6, and coded according to the themes
noted above. Reports on each theme were then generated.

## 6.2 Local Health Educator Methods {#local-health-educator-methods-1 .unnumbered}

### Mail Survey {#mail-survey-2 .unnumbered}

Further data were collected among local health educators who attended
regional or local trainings provided by master trainers during the
one-year follow-up period. Data were collected to measure local adoption
and implementation of the programs. Implementation is the actual use of
the program in youth group settings. The primary outcome measured is the
number of youth reached through program implementation. Secondary
outcome measures include: costs associated with implementation, number
and type of administrative agreements to allow local implementation of
the programs, number of contacts with master trainers for
problem-solving of adoption and implementation issues, and amount of
informational materials distributed locally. Regional and local contexts
for adoption and implementation of *Programs That Work* were assessed
through mailed survey questions about school and community policies
regarding the content and time allotted for HIV prevention curricula, as
well as community resources currently devoted to risk reduction for HIV.

In the mailing of the survey, Battelle provided the LHEs with an 800
telephone number for a project staff member who had been trained on the
survey and could answer respondent questions and concerns about it. As
an incentive to complete and return the SAQ, each local health educator
received, in addition to the SAQ, an abstinence promotion poster
produced by ETR Associates. The local health educators were also
informed about voluntary participation and confidentiality. A signed and
completed written consent was obtained from each respondent. The
respondents sent back the consent form and the survey in one mailing.
Battelle staff separated the consent form and the survey so that they
could not be associated in any way.

Master trainers identified 73 local health educators that they trained.
All 73 comprised the population for inclusion in the study.

In March 2002, Battelle found that the contact information on LHEs was
very sparse. Battelle proposed sending surveys throughout the month of
April, such that the survey would be sent out as further contact
information was received. This was proposed in place of having a single
mailing date. In April 2002, Battelle sent advance letters to local
health educators and mailed survey packets to all local health educators
for which we had contact information.

On May 7, 2002, postcard reminders were sent to a second batch of LHEs.
Questionnaires were sent on the 5th and postcard reminders were sent on
the 20th to a third batch of LHEs. On May 31, Battelle sent the second
survey packages to 47 LHEs who had not yet returned the survey. Battelle
also contacted some of the LHEs via telephone and left messages, asking
them to return the survey. By the end of May, 34% of the LHEs had
returned the survey.

Because the response rate of local health educators was still below
acceptable level, in addition to sending a second package, Battelle
attempted to locate telephone numbers and email addresses of unreached
local health educators in order to contact them by telephone and email.

The third batch of local health educator questionnaires (n=31) were sent
out on August 14 and August 15, 2002. By the end of August, 42 surveys
were received, yielding a response rate of 58%. The team agreed to
discontinue mailing surveys to LHEs on September 30. During the month of
September 2002, Battelle followed up with local health educators who had
not sent back the survey by calling and inviting them to complete the
survey over the telephone. While only one of them agreed to complete the
survey over the telephone, two additional questionnaires were received
in the mail, bring the total received to 44, and raising our response
rate to 60.3%.

With regard to LHE poster incentives, Battelle returned 166 unused
posters to ETR and was told in May 2002 that they could not be reused.
Battelle was given the option of either taking credit from ETR for
future purchasing or bearing the cost and getting the posters back. In
July 2002, Battelle was informed that a credit of \$780.72 had been
awarded by ETR for the posters.

### Data Entry, Cleaning, and Analyses {#data-entry-cleaning-and-analyses-3 .unnumbered}

Upon receipt, completed self-administered questionnaires were submitted
to the editing department for editing, coding, and cleaning. The editing
department worked closely with the primary data cleaner who documented
any coding decisions in a coding decision log. The cleaned data was then
entered into an Access database. The Access database provided quality
control advantages due to its user-friendly data entry forms and
structural alignment with hardcopy questionnaires. The database was then
sent to the primary data cleaner who imported the data into an Excel
spreadsheet and conducted a follow-up cleaning of the data, again noting
any coding decisions in the coding decision log.

Frequency analyses have been conducted for all variables. A number of
summary variables have also been generated to group related items and to
capture the activities that had been conducted by local health educators
to disseminate the curriculum. SAS program was used to conduct all
statistical analyses and data management.

# 7.0 Results  {#results-1 .unnumbered}

##  {#section-12 .unnumbered}

## 7.1 Master Trainer Results {#master-trainer-results-1 .unnumbered}

### Telephone Survey  {#telephone-survey-3 .unnumbered}

***Sample characteristics and response rate.*** Twenty-six master
trainers comprised the population for inclusion in the surveys. They are
from 13 different states: Arizona, Connecticut, Florida, Indiana, Iowa,
Kansas, Missouri, Massachusetts, Maine, New Jersey, New York, Ohio, and
Texas.

For the first round of surveys (6 months after the national training),
all 26 master trainers participated in the telephone survey, yielding a
response rate of 100%. For all 26 participants, we obtained their
consent for participating in the study. The length of the interview
ranged from 5 minutes to 34 minutes, with an average of 15 minutes.

Thirty-three percent (33%) of the participants were working for state
education agencies (n=9), 23% for local/district education agencies
(n=6), 12% for community agencies (n=3), 4% for state health agencies
(n=1), 4% for private health care organizations (n=1), 4% for regional
education agencies (n=1), 4% for state public universities (n=1), 4% for
colleges (n=1), 4% for grant-funded programs (n=1), 4% for
non-professional educational companies (n=1), and 4% for private
universities (n=1).

Nineteen percent (19%) of the participants were state education agency
administrators (n=5), 19% were state-level master trainers (n=5), 23%
were local- or district-level teacher trainers (n=6), 7% were teacher
supervisors/curriculum specialists (n=2), 4% were community agency
administrators (n=1), 4% were community agency staff (n=1), 4% were
public/community health educators/nurses (n=1), 4% were professors
(n=1), 4% were regional-level master trainers (n=1), 4% were regional
trainers of teacher administration (n=1), 4% were college teachers
(n=1), and 4% were regional professional developers (n=1).

Fifty percent (50%) of the participants had more than 10 years of
experience in providing HIV prevention training to health educators or
to young people (n=13), 15% had eight to ten years of experience (n=4),
23% had five to seven years of experience (n=6), 8% had two to four
years of experience (n=2), and the other 4% had one year or less
experience (n=1).

For the second round of the survey (12 months after the training), 25
out of 26 master trainers participated in the survey, yielding a
response rate of 96%. The person who was not surveyed (ID \# 422) could
not be located and was recorded as lost to follow-up. For all 25
participants, we obtained their consent for participating in the study.
The length of the interview ranged from 4 minutes to 26 minutes, with an
average of 11 minutes.

***Frequency analyses**.* Frequency analyses have been conducted for the
data collected at 6 months after the national training, 12 months after
national training, and the aggregated datasets (6 months and 12 months),
respectively. Only major findings are reported in this section (Table
6.1).

It was found that within 6 months after the *Safer Choices* national
training, four local trainings had been held, which involved 65 local
health educators. Specifically, one training was held in State14
(ID#14), where 2 local health educators were trained. One training was
held in State32 (ID#32), and 21 local health educators participated in
the training. One training was held in State51 (ID#51), where 11 local
health educators participated. Finally, 31 local health educators
participated in the training held in State61 (ID#61). In the next six
months (between 6 months after the national training and 12 months after
the training), one local training was held in State32, and 8 local
health educators participated in the training.

When asked whether and how they provided people with information about
*Safer Choices* after the national training, most master trainers
answered that they used more than one way to get the word out about the
program within 6 months after the training (min=0, max=9, mean=2.9).[^9]
The four most frequently used methods were presentations or briefings
(62% of master trainers used this method), one-to-one meeting (58%),
networking (46%), and informal workshops (46%). Between 6 months and 12
months after training, these activities had decreased, but most master
trainers still used a variety of methods to provide people with
information about the program (min=0, max=6, mean=2.0). The top three
methods used were presentations or briefings (40%), networking (40%),
and one-to-one meeting (32%).

When asked if the master trainers informed different agencies/groups
(e.g., state education agency, state health agency, etc.) about the
program, most master trainers said that they informed multiple
agencies/groups within 6 months after the training (min=0, max=12,
mean=4.4).[^10] The top three agencies/groups that had been informed of
the program were school teachers or health educators (61%), local
education agency (54%), and state education agency (46%). Although the
trend had decreased after 6 months, most master trainers still informed
more than one agency/group (min=0, max=9, mean=3.9). The top three
agencies/groups that had been informed about the program were school
teachers or health educators (64%), local education agency (48%), and
HIV prevention community planning group (48%).

The master trainers were also asked if the agencies/groups they informed
about the program agreed to support implementation of the curriculum. It
was found that within 6 months after the training, an average of 2.0
agencies/groups agreed to support the implementation of the curriculum
(min=0, max=10). The most frequently mentioned agencies/groups were the
state education agencies (38%), followed by local education agencies
(19%), HIV materials review boards (19%), school principals (19%), and
school teachers or health educators (19%). Between 6 months and 12
months after the training, an average of 2.1 agencies/groups (min=1,
max=9) agreed to support. The most frequently mentioned agencies/groups
were local education agency (24%) and parents or parent groups (24%),
followed by HIV prevention planning group (20%), HIV materials review
board (20%), school board (20%), school principals (20%), and school
teachers or health educators (20%).

As stated before, the *Safer Choices* curriculum is a more complicated
curriculum with more components; therefore, the master trainers were
also asked whether they had implemented any components of *Safer
Choices* directly with young people. Within six months after the
national training, only 1 master trainer had implemented all five
components of *Safer Choices* curriculum directly with young people, and
1 MT had implemented one component. Between 6 months and 12 months after
the training, 3 MTs had implemented 1 component and 1 MT had implemented
3 components.

Table 6.1 Major Findings from Master Trainer Telephone Survey for Safer
Choices

  ---------------------------- ------- ------- ------- -------- ------- -------
                               First                   Second           
                               Wave                    Wave             

                               Min     Max     Mean    Min      Max     Mean

  Number of ways used to       0       9       2.9     0        6       2.0
  provide people with                                                   
  information about *Safer                                              
  Choices*                                                              

  Number of agencies informed  0       12      4.4     0        9       3.9
  about *Safer Choices*                                                 

  Number of agencies agreed to 0       10      2.0     1        9       2.1
  support *Safer Choices*                                               
  ---------------------------- ------- ------- ------- -------- ------- -------

### Focus Group  {#focus-group-3 .unnumbered}

***Sample characteristics and response rate.*** Of the 26 master
trainers that participated in the 6- and 12-month surveys, a total of 12
participated in focus groups. Of the 14 master trainers who did not
participate in focus groups, 2 could not be located, 3 refused, and the
remaining 9 could not participate due to scheduling conflicts. Although
we were unable to coordinate the timing of a focus group with these
latter 9 master trainers, it is important to note that they were
extremely cooperative and attempted to schedule more than once.

The focus group data were also examined in terms of response by
respondent characteristics, i.e., type of organization, role at
organization, years of experience. Differences in responses by these
characteristics were not evident nor did they emerge from the data
through careful analysis.

***Findings.*** The focus group discussions revealed a great deal of
pertinent and useful information on the dissemination and implementation
of HIV prevention programs, such as *Safer Choices.* Detailed
information regarding the focus groups can be found in Appendix 1. Only
a summary of the findings will be presented in this section.

In general, the master trainers had a positive regard for *Safer
Choices* and for the *Programs That Work*, but expressed many concerns
about implementing *Safer Choices* in particular, due its large scope
and the resources that would be required to implement it with fidelity.

While there were individual differences within and across groups in
terms of the actual implementation process, in most cases, states
provided information to districts or schools but could not make
recommendations on programs to implement. In addition, the choice of a
program and decision to implement such a program is generally up to
local-level representatives such as districts or schools. Also, the
implementation process usually involves a number of approvals, including
review by a material review board or panel.

Respondents shared many concerns as well as solutions and
recommendations. They felt that the curriculum was generally a good one,
but that it was difficult to implement with fidelity because it is a
large program with five components, therefore requiring more resources
than other programs to implement. The lack of resources in general was a
barrier, given the state of budget crisis in most states. Respondents
also felt that the program was not targeted towards the appropriate
population, deeming it more appropriate for 6^th^- to 8^th^- grade
youth.

The master trainers also reported on the strengths of the program and on
other factors that contributed to the dissemination of the program.
While they mentioned that it is difficult to implement such a large
program with numerous components, they also felt that this was one of
the strengths of the program. The various components enable the program
to offer a comprehensive package. The trainers also provided
recommendations on ways to improve the dissemination and implementation
process. In general, they felt that the training could be shortened,
that a local assessment piece could prove to be fruitful, and that
further research is needed, particularly on the outcomes related to
youth attitude and behavior change.

## 7.2 Local Health Educator Results {#local-health-educator-results-1 .unnumbered}

### Mail Survey  {#mail-survey-3 .unnumbered}

***Sample characteristics and response rate.*** Contact information on
73 local health educators was provided by 6 master trainers using
rosters. Forty-three local health educators sent back the survey. Of the
remaining 30, no LHEs refused to complete the questionnaire. None were
returned undeliverable. Battelle attempted to invite the remaining 30
people to complete the survey over the telephone. For those people we
were able to reach by telephone, only one agreed to complete the survey
over the telephone. For those people we were unable to reach, we left a
message on their voicemail. None of them called us back and none of them
returned the mail survey. As a result, 44 LHEs completed the survey (43
by mail, 1 by telephone), yielding a response rate of 60.3%. Consent was
obtained from all of them.

Among these LHEs who returned the survey, some were trained by multiple
master trainers jointly. Specifically, 1 LHE was trained by 1 master
trainer from State14, 14 LHEs were trained by 2 MTs from State32
jointly, 4 LHEs were trained by 1 MT from State51, and 19 LHEs were
trained by 2 MTs from State61. These LHEs participated in trainings held
within 6 months after the national training. In addition, 6 LHEs were
trained by 2 MTs from State32 between 6 months and 12 months after the
national training.

Among all the participants, 93% were working for school (public/private/
alternative) (n=41), 2% for community agencies (n=1), and 5% for health
agencies (n=2).

Seventy-one percent (71%) of the participants were classroom teachers
(n=31), 2% were teacher supervisors/curriculum specialists (n=1), 2%
were community agency administrators (n=1), 5% were community agency
staff (e.g., youth worker, recreation leader) (n=2), 2% were other
administrators (e.g., school administrators, administrative assistants,
state agency administrators) (n=1), 16% were school nurses (n=7), and
the other 2% were counselors, social workers, or therapists (n=1).

Among all the participants, 32% had more than 10 years of experience
providing HIV prevention training to health educators or to young people
(n=14), 9% had eight to ten years of experience (n=4), 18% had five to
seven years of experience (n=8), 16% had two to four years of experience
(n=7), 9% had one year or less experience (n=4), and the other 16% had
no experience (n=7).

***Frequency analyses.*** Frequency analyses have been conducted for the
data collected from local health educator survey. Only major findings
are reported in this section (Table 6.2).

The local health educators were asked whether they provided other people
with information about the *Safer Choices* program since they attended
the training and whether they used multiple methods to provide
information. Most local health educators reported that they used more
than one way to provide information to others about the curriculum
(min=0, max=7, mean=2.3). [^11] The three methods mentioned most
frequently were one-to-one meetings (61%), presentations or briefings
(52%), and networking (39%).

The local health educators were also asked if and how many agencies and
groups they had informed of the program since the training. Most LHEs
informed more than one group or agency (min=1, max=9, mean=3.7).[^12]
The three agencies/groups that were mentioned most frequently were
school teachers or health educators (95%), school principals (84%), and
students' parents (59%).

The LHEs were asked whether they were required to obtain permission to
implement the program from different agencies/groups. It was found that
most LHEs were required to obtain permission from more than one
agency/group (min=0, max=9, mean=3.0). [^13] The top 3 agencies/groups
mentioned by LHEs from which they were required to obtain permission
were school principals (44%), students' parents (37%), and local
education agencies (36%).

The LHEs were also asked if the agencies/groups they informed of the
program agreed to support the implementation of the curriculum. It was
found that an average of 3.8 agencies/groups agreed to support the
implementation of the curriculum (min=0, max=9). The three
agencies/groups that were most frequently mentioned were school
principals (73%), students' parents (43%), and school board (39%).

The LHEs were asked whether they had implemented any components of
*Safer Choices* directly with young people. Twenty-four percent (24%) of
the LHEs (n=10) said that they did not implement any components of
*Safer Choices* directly with young people. Thirty percent (30%) said
that they implemented one component (n=13), 12% said that they
implemented two components (n=5), 14% implemented three components
(n=6), 10% implemented four components (n=4) and 10% implemented all
five components (n=4). Two people did not answer this question. When
asked how many young people they had taught, it was found that a total
of 8,132 young people were taught, with an average of 301 young people
per LHE who answered the question (n=27).

Table 6.2 Major Findings from Local Health Educator Survey for Safer
Choices

  ---------------------------------------------- ------------ -------- --------
                                                 LHE Survey            
                                                 Statistics            

                                                 Min          Max      Mean

  Number of ways used to provide people with     0            7        2.3
  information about *Safer Choices*                                    

  Number of agencies informed about *Safer       1            9        3.7
  Choices*                                                             

  Number of agencies whose permission is needed  0            9        3.0
  to implement *Safer Choices*                                         

  Number of agencies agreed to support *Safer    0            9        3.8
  Choices*                                                             
  ---------------------------------------------- ------------ -------- --------

# 8.0 Conclusions  {#conclusions-1 .unnumbered}

This evaluation adopted both qualitative and quantitative methods to
evaluate the dissemination of two CDC *Programs That Work: Focus on
Kids* and *Safer Choices.* Data were collected from multiple sources,
that is, master trainers and local health educators. Information from
the surveys supplement information uncovered through the focus groups,
and vice versa. This variation in data collection activities has enabled
us obtain a more comprehensive understanding of dissemination and
implementation activities with regard to HIV prevention programs for
youth such as *Focus on Kids* and *Safer Choices*.

In general, the response rate is acceptable. Across 6 data collections,
the average response rate is 85% (min=59%, max=100%). The sample also
covers a variety of agencies/groups, occupations, and years of
experience. All participants were informed of voluntary participation
and consent was obtained from each participant.

Results from master trainer telephone survey data indicated that, for
both curricula, (1) multiple local trainings had been held by master
trainers after the national training; (2) most master trainers used a
variety of methods to get words out about the program; (3) a variety of
agencies/groups had been informed by the master trainers about the
program; (4) some of the agencies/groups informed agreed to support
implementation of the program; and (5) few young people were taught by
master trainers directly. Overall, the reported dissemination activities
had slightly decreased during the second wave of the survey, compared to
the first wave of the survey, but remained at a quite high level.

Focus group data showed that in general, the master trainers had a
positive regard for both curricula. However, the major concern was the
difficulty to implement the curricula with fidelity. For *Focus on
Kids*, this concern was raised because it was designed for a community
setting or alternative schools, not for mainstream public schools. For
*Safer Choices*, this concern was raised because it is a large program
with 5 components, which requires more resources to implement. The
participants of the focus groups also provided other comments on the
strengths of the program, as well as concerns, solutions and
recommendations.

The results from local health educator questionnaires are similar to the
results of master trainer survey. For both curricula, it was found that
(1) the local health educators used multiple methods to provide people
with information on the curricula; (2) multiple agencies/groups were
informed; (3) most local health educators were required to obtain
permission from more than one agency/group to implement the curricula;
(4) most agencies/groups informed agreed to support implementation of
the curricula; and (5) a large number of young people were taught by
local health educators.

To conclude, the dissemination of *Focus on Kids* and *Safer Choices*
had been a success: multiple local trainings had been held; information
of the curricula had been delivered to a variety local agencies/groups;
and a large number of young people had been taught. However, there are
still some barriers to the implementation of the program that can be
addressed. The curricula may need to be tailored in order to better suit
the needs of varying audiences. In addition, a cost-effectiveness
analysis is needed to further determine the success of the programs.

**APPENDICES**

Appendix 1 The Master Trainer Focus Group Report

Appendix 2 Focus on Kids Master Trainer (MT) Survey Documentation

> Appendix 2.1 The codebook (Word file)
>
> Appendix 2.2 The coding decision rules (Word file)
>
> Appendix 2.3 The attention items (Word file)
>
> Appendix 2.4 The MT survey one data (SAS file)
>
> Appendix 2.5 The MT survey two data (SAS file)
>
> Appendix 2.6 The concatenated MT survey one and two data (SAS file)
>
> Appendix 2.7 The open-ended response data (Excel file)
>
> Appendix 2.8 The frequency program (SAS file)
>
> Appendix 2.9 The frequency output (SAS file)
>
> Appendix 2.10 The ad hoc analysis program (SAS file)

Appendix 2.11 The ad hoc analysis output (SAS file)

Appendix 2.12 The value label program (SAS file)

Appendix 3 Focus on Kids local health educator (LHE) SAQ Documentation

> Appendix 3.1 The codebook (Word file)
>
> Appendix 3.2 The coding decision rules (Word file)
>
> Appendix 3.3 The attention items (Word file)
>
> Appendix 3.4 The data entry database (Access file)
>
> Appendix 3.5 The LHE data (SAS file)
>
> Appendix 3.6 The frequency program (SAS file)
>
> Appendix 3.7 The frequency output (SAS file)
>
> Appendix 3.8 The ad hoc analysis program (SAS file)

Appendix 3.9 The ad hoc analysis output (SAS file)

Appendix 3.10 The value label program (SAS file)

Appendix 4 Safer Choices Master Trainer (MT) Survey Documentation

 

> Appendix 4.1 The codebook (Word file)
>
> Appendix 4.2 The coding decision rules (Word file)
>
> Appendix 4.3 The prominent coding decisions (Word file)
>
> Appendix 4.4 The attention items (Word file)

Appendix 4.5 The MT survey one data entry database (Access file)

Appendix 4.6 The MT survey two data entry database (Access file)

> Appendix 4.7 The MT survey one data (SAS file)
>
> Appendix 4.8 The MT survey two data (SAS file)
>
> Appendix 4.9 The concatenated MT survey one and two data (SAS file)
>
> Appendix 4.10 The frequency program (SAS file)
>
> Appendix 4.11 The frequency output (SAS file)
>
> Appendix 4.12 The ad hoc analysis program (SAS file)

Appendix 4.13 The ad hoc analysis output (SAS file)

Appendix 4.14 The value label program (SAS file)

Appendix 5 Safer Choices local health educator (LHE) SAQ Documentation

> Appendix 5.1 The codebook (Word file)
>
> Appendix 5.2 The coding decision rules (Word file)
>
> Appendix 5.3 The attention items (Word file)

Appendix 5.4 The data entry database (Access files)

> Appendix 5.5 The LHE data (SAS file)
>
> Appendix 5.6 The frequency program (SAS file)
>
> Appendix 5.7 The frequency output (SAS file)
>
> Appendix 5.8 The ad hoc analysis program (SAS file)

Appendix 5.9 The ad hoc analysis output (SAS file)

Appendix 5.10 The value label program (SAS file)

Appendix 6 Instruments: Surveys and Questionnaires

> Appendix 6.1 The Focus on Kids Master Trainer Survey One Instrument
> (original)
>
> Appendix 6.2 The Focus on Kids Master Trainer Survey One Instrument
> (final)
>
> Appendix 6.3 The Focus on Kids Master Trainer Survey Two Instrument
> (original and final)
>
> Appendix 6.4 The Focus on Kids Local Health Educator Self-Administered
> Questionnaire (original)
>
> Appendix 6.5 The Focus on Kids Local Health Educator Self-Administered
> Questionnaire (final)
>
> Appendix 6.6 The Safer Choices Master Trainer Survey One Instrument
> (original)
>
> Appendix 6.7 The Safer Choices Master Trainer Survey One Instrument
> (final)
>
> Appendix 6.8 The Safer Choices Master Trainer Survey Two Instrument
> (original)
>
> Appendix 6.9 The Safer Choices Master Trainer Survey Two Instrument
> (final)
>
> Appendix 6.10 The Safer Choices Local Health Educator
> Self-Administered Questionnaire (original)
>
> Appendix 6.11 The Safer Choices Local Health Educator
> Self-Administered Questionnaire (final)

[^1]: ETR Associates is a private, not-for-profit corporation that
    provides leadership, educational resources, training, and research
    in health promotion focused on sexuality and health education. The
    enhancement condition for the *Focus on Kids* evaluation involved a
    1 ½ day site visit from ETR, during which they provided additional
    technical assistance and training using the ETR dissemination
    package.

[^2]: Master trainers in enhanced group received a one-and-a-half-day
    site visit, where they were provided additional training using a
    dissemination package besides the national training, whereas master
    trainers in standard group only participated in the national
    training on the curriculum.

[^3]: Min=2 means that the lowest number of methods used by the master
    trainer to provide people information about the program is 2. One
    person used 9 different methods to get the word out about the
    program, which makes the maximum equal to 9. On average, master
    trainers used 5 ways to introduce the program.

[^4]: Min=1 means that the lowest number of agencies informed by master
    trainer is 1. One master trainer informed 11 different agencies,
    which makes the maximum equal to 9. On average, master trainers
    informed 6.3 agencies.

[^5]: One person did not describe his/her organization.

[^6]: Min=0 means that some local health educators did not use any
    method to provide people with information about the program. One
    person used 6 different methods to get the word out about the
    program, which makes the maximum equal to 6. On average, local
    health educators used 2 different ways to introduce the program.

[^7]: Min=0 means that some LHEs did not provide information to any
    agencies or groups. Some LHEs provided information to as many as 9
    different agencies or groups, which makes the maximum equal to 9. On
    average, LHEs provided information to 2.7 agencies or groups.

[^8]: Min=0 means that some LHEs did not need permission from any
    agencies or groups. Some LHEs needed permission from as many as 9
    different agencies or groups. On average, LHEs were required to
    obtain permission from 3 agencies or groups.

[^9]: Min=0 means that some master trainers did not use any method to
    provide people with information about the program. One person used 9
    different methods to get the word out about the program, which makes
    the maximum equal to 9. On average, master trainers used 3 different
    ways to introduce the program.

[^10]: Min=0 means that some master trainers did not inform any
    agencies/groups. One master trainer informed 12 different
    agencies/groups, which makes the maximum equal to 12. On average,
    master trainers informed 4.4 agencies/groups.

[^11]: Min=0 means that some local health educators did not use any
    method to provide people with information about the program. Some
    LHEs used 7 different methods to get the word out about the program,
    which makes the maximum equal to 7. On average, local health
    educators used 2.3 ways to introduce the program.

[^12]: Min=1 means that some LHEs only provided information to one
    agency or group. Some LHEs provided information to as many as 9
    different agencies or groups, which makes the maximum equal to 9. On
    average, LHEs provided information to 3.7 agencies or groups.

[^13]: Min=0 means that some LHEs did not need permission from any
    agencies or groups. Some LHEs needed permission from as many as 9
    different agencies or groups. On average, LHEs were required to
    obtain permission from 3 agencies or groups.