# Presentation: 062949

## The Continuing Adventures of Fidelity & Adaptation

- Implementing Science-based Programs Effectively:
- A Forum on Fidelity and Adaptation Issues
- Paul Brounstein
- April 16, 2003

## Fidelity and Adaptation

- Two sides of the same coin or 2 cents to rub together?
- Best thought of as complementary concepts
- Implementers should strive for balance between the two when adopting or adapting effective prevention programs

## Fidelity

- Fidelity is high when the number and nature of activities remain the same over implementations.
- In R&D efforts, when implemented as planned—optimally when activities are clearly linked to need and desired outcomes and dose is sufficient to yield effects

## Adaptations

- Adaptations may be made to number and/or regularity of activities, time spent in specific activities, personnel implementing, specific activities used in order to enhance effectiveness by:
- Allowing implementers to use resources efficiently
- Adjusting for specific needs of the client population
- The benefits of adaptation include greater inclusion of local contextual information in the implementation process and increased local buy-in

## Why You Should Give a Sou

- Adaptations can also attenuate program effectiveness
- Insufficient dosage
- Changing time schedules to compress program
- Inappropriate changes in activities given theoretical model of change
- Poor translation in attempts to make interventions culturally, age, or gender appropriate.

## How do we know  Adaptations are Advantageous?

## HRY X-Site:  Design Characteristics

**Large Multi-Site Sample (48 sites, over 10,000 youth)**
**Comparison/Control Groups**
**Common Instrument**
**Four Points In Time**
**Exposure to Prevention Services - (Dosage)**
**Program Level Variables**

## Internal Risk and Protective Factors by Age in the Baseline Sample

**More Protection**

**More Risk**

**(N= 10,473)**

## Average 30-Day Substance Use Over Time for Males*

**8%**

**22%**

**0%**

**29%**

***Low comparison group exposure to prevention services sites only.  Averages include covariate adjustments.**

**(N=3,307)**

## Average 30-Day Substance Use Over Time for Females*

***Low comparison group exposure to prevention services sites only.  Averages include covariate adjustments.**

**1%**

**3%**

**6%**

**9%**

**(N= 1,888)**

## Finding 

***	******Although female adolescent programs tended to emphasize affective content, the girls who attended programs focusing on behavioral- or life-skills demonstrated significantly greater positive outcomes for substance use over the four time points than did girls in other programs (female or coeducational).***

## Finding 

***	******For boys, programs that emphasize interactive styles of delivering program content are more effective the four time points of the study.  Emphasis on different types of content (e.g., affective or life-skills) did not make a significant difference in sustaining effects. ***

## Findings

- Culturally adapted programs produced effect sizes twice that of programs that were not so adapted.
-  Cultural adaptations served to increase satisfaction, interest and exposure to the programs, effecting outcomes as a mediating variable

## Trajectory of Innovations

- Classic R&D Model
- Innovation is developed
- Prototyped technology is implemented and tested
- If successful, information is disseminated
- Those receiving the information and predisposed to change/improvement adopt the technology
- Diffusion

## Reality Intervenes Again

- Some people adopt “fads” that resonate with their values and/or their belief system but that just don’t work
- Some people make the decision to adopt an innovation but fail to comprehend the model or become overwhelmed and never implement it properly
- Research on adoption suggests adoption is facilitated by compatibility, perceived relative advantage,  and complexity (negatively) of the innovation

## One question:  Can programs be replicated with Fidelity:  CSAP Replication Initiative Goals

- Identified Innovative, Promising Programs
- GAO selection criteria for exemplary programs**: **
- FY ‘94 and FY ‘95
- 17 models of which 9 were selected by grantees for replication
- 16  5-year awards
- 6 self-replications

## Fidelity Instruments

- Fidelity instruments were developed in consultation with the program developer to quantify the degree to which the new projects replicated the original project models in a manner that closely resembled the original

## Slide 18

## Interpreting Multi-Site Evaluation Results

- In general, the pattern of results observed replicated those of the original programs, not all of whom had consistent positive outcomes
- e.g., where results were mixed or flat, they tended to mirror the results achieved by the original program

## Fidelity-Effectiveness Relation

- Some replicants provided data indicating a positive relationship between fidelity and outcomes
- One replicant implemented with greater fidelity to the program plan than did the original user – and had better outcomes

## Post Mortem w/Replicators

- all said they would replicate again
- self-replicants
- opportunity to increase what we know about particular program
- makes good sense once we know original works
- new replication grantees
- good use of resources
- great way to introduce new services to community
- provided mechanism to network with other programs
- original proof of effectiveness was invaluable

## Core Components Analysis

- Reviewed 21 model and effective programs
- Teams of prevention researchers independently reviewed and coded programs.
- Developers interviewed.
-  Program materials reviewed included:
- Fidelity instruments
- Findings papers, publications
- Final reports
- Program curricula

## Rating and Coding

- Teams of raters independently reviewed programs to answer the question:
*What elements are fundamental to this program?*
- Focused on participant and program characteristics, facets of evaluation design and implementation, and factors that enhance program adaptation and/or replication under differing contextual conditions.
- Detailed coding protocol.
- Inter-rater reliability assessed and improved with consensus meetings.

## Results

- Besides imparting new knowledge and skills, effective prevention programs provide participants with opportunities to use this information.
- Effective programs move beyond change at the individual level.  Emphasis is placed on creating lasting changes within individual, family, and school domains in an effort to create “caring communities” that share accountability for change.

**Notes:**
Among programs reviewed, opportunities for practice were incorporated into curriculum-based activities or through the addition of intervention components intended to reinforce curriculum content.  Commonly employed curriculum-based strategies include: 

Modeling and behavioral rehearsal (facilitator demonstrates a new skill; participants then perform the skill within session).
Assigned out-of-session activities intended to reinforce concepts (journaling, identification of issues to be raised in subsequent sessions, practice of skills at home with parents or others).
Cueing (teachers cue students to use new behaviors in specific situations).
Placing participants in the role of expert and having them demonstrate new knowledge and skills (e.g., participants create an anti-drug advertising campaign that would be effective with their peer group).
Use of self-monitoring techniques to enhance awareness and enactment of desired behaviors. 

## Results

- Program developers consider parental involvement a critical factor for success.
- Efforts to include parents focus toward two interrelated goals: enhancing parenting skills and self-efficacy, and increasing the level of parental involvement in children’s lives.

**Notes:**
Close to one-half (48%) of reviewed programs incorporate a parenting component:

Fully 60% of programs with a parenting component utilize structured activities and experiential activities (social, cultural, recreational events) to foster increased interaction between parents and youth.

The remaining 40% of programs provide one or more forms of parenting skills training. 

## Results

- Successful programs promote a consistent message sent through multiple channels (e.g., parents, teachers, and peers).
- Effective programs attend to characteristics of the target population that place them at risk for alcohol, tobacco, and other drug use.

**Notes:**
Incredible Years, the Child Development Project, and Project Achieve, for example, employ a “whole school reform” approach.  A consistent message is sent to parents, teachers, and students, and students consistently hear this message in settings where a majority of their time is spent – at home and school. 

## Results

- Successful programs emphasize relationship building as a precursor to delivery of program content.  Number of sessions provided and activities that comprise the intervention vary; but the first step of gaining influence is common.
- Importance of entering into the world of the  client and integrating services into it are stressed.

**Notes:**
For example, Family Effectiveness Training, Fairfax Leadership and Resiliency, and Communities Mobilizing for Change on Alcohol stress the importance of relationship building across individual and agency levels.  Effective programs establish relationships with agencies in which services will be offered, and nurture these relationships throughout the life of the program.
Teachers, coaches, and other individuals delivering program content receive ongoing support and direction.
Initial sessions focus on joining participants together, prior to introducing program content.
Critical to the success of Project Achieve was “buy in” on the school and district level prior to program implementation.
The positive effects of relationship were observed among participants in the Across Ages program: 
The greatest gains were observed among participants in the mentoring component of the program who engaged in consistent and ongoing contact with caring adult mentors. 

## Results

- Successful programs use naturally occurring social networks.  Services are delivered via the school, community-based agencies or through other networks already in place (e.g., sports team setting).
- Process evaluation data reveal that successful programs enjoy high fidelity to the program plan or curriculum, dosage adequacy, and consistency.

## Results

- Educational attainments and experience levels of persons delivering intervention are highly variable; yet programs consistently require the training of delivery agents (self-instructional, curriculum-based or in-person) prior to implementation.

**Notes:**
One-half of reviewed programs do not require delivery agents to have specific educational attainments; two-fifths require that agents hold a bachelor’s degree in a relevant field.  Two-thirds of reviewed programs require facilitators to have prior employment experience in an area relevant to the target population and/or target problems/issues to be addressed through the program. 

Four-fifths of facilitators received advance training to acclimate them to the goals and philosophy of their respective programs and to standardize practices employed over the duration of  intervention:
Remote site training is the most common type of training participants receive prior to implementing the intervention.

## Conclusion

- Knowledge of the active ingredients in a science-based program provides guidance on where and how to adapt a program without losing fidelity.
- Programs adapted in line with CCA may in fact prove more effective than those unaltered from their original form.

## Prevention Programs Outcomes Monitoring System (PPOMS)

- Goals

** ****Quantify extent of science-based program implementation**

-  **Measure fidelity and adaptation levels**
** ****Document outcome results**

-   **Relate fidelity and adaptation to outcomes**

## Prevention Programs Implemented

- What was done?
- (Adaptations)

- How Much?
- (Dosage)

- What results?
- (Outcomes)

*Accountability--The *Prevention Programs Outcomes Monitoring System (PPOMS)*—in Pursuit of Science*

## What You Can Do To Enhance Effectiveness

- Maximize fidelity to program model whenever possible
- Add adaptations if at all possible, don’t substitute for regular activities
- Review program philosophy and logic model; make sure changes made are consonant
- Talk to the developer and/or certified trainers about modifications, discuss core components/active ingredients

## Final Recommendations, Con’t.

- Perform assessments
- Pre, post and at intervals along the way
- Process and outcome
- Feedback Results to further improve program
- Cooperate with National Surveys (PPOMs)