# Presentation: 382502

## Assessing Intervention Fidelity in RCTs: Concepts and Methods

- Panelists:
- David S. Cordray, PhD
- Chris Hulleman, PhD
- Joy Lesnick, PhD
- Vanderbilt University
- Presentation for the IES Research Conference
- Washington, DC
- June 12, 2008

## Overview

- Session planned as an integrated set of presentations
- We’ll begin with:
- Definitions and distinctions;
- Conceptual foundation for assessing fidelity in RCTs, a special case.
- Two examples of assessing implementation fidelity:
***Chris Hulleman*** will illustrate an assessment for an intervention with a single core component
***Joy Lesnick*** illustrates additional consideration when fidelity assessment is applied to intervention models with multiple program components.
- Issues for the future
- Questions and discussion

## Definitions and Distinctions

## Dimensions Intervention Fidelity

- Little consensus on what is meant by the term “intervention fidelity”.
- But Dane & Schneider (1998) identify 5 aspects:
- Adherence/compliance– program components are delivered/used/received, as prescribed;
- Exposure – amount of program content delivered/received by participants;
- Quality of the delivery – theory-based ideal in terms of processes and content;
- Participant responsiveness – engagement of the participants; and
- Program differentiation – unique features of the intervention are distinguishable from other programs (including the counterfactual)

## Distinguishing Implementation Assessment from Implementation Fidelity Assessment

- Two models of intervention implementation, based on:
- A purely descriptive model
- Answering the question “What transpired as the intervention was put in place (implemented).
- An *a priori* intervention model, with explicit expectations about implementation of core program components.
- Fidelity is the extent to which the realized intervention (**t****Tx**) is “faithful” to the pre-stated intervention model (**T****Tx**)
- Fidelity = **T****Tx**** – t****Tx**
- We emphasize this model

## What to Measure?

**Adherence** to the intervention model:
***(1) Essential*** or ***core*** components (activities, processes);
***(2) Necessary, but not unique to the theory/model****,* activities, processes and structures (supporting the essential components of T); and
***(3) Ordinary features*** of the setting (shared with the counterfactual groups (C)
- Essential/core and Necessary components are priority parts of fidelity assessment.

## An Example of Core Components” Bransford’s HPL Model of Learning and Instruction

- John Bransford et al. (1999) postulate that a strong learning environment entails a combination of:
- Knowledge-centered;
- Learner-centered;
- Assessment-centered; and
- Community-centered components.
- Alene Harris developed an observation system (the VOS) that registered novel (components above) and traditional pedagogy in classes.
- The next slide focuses on the prevalence of Bransford’s recommended pedagogy.

## Challenge-based Instruction in “Treatment” and Control Courses: The VaNTH Observation System (VOS)

**Percentage of Course Time Using Challenge-based Instructional Strategies**

- Adapted from Cox & Cordray, in press

## Implications

- Fidelity can be assessed even when there is no  known benchmark (e.g., ***10*** Commandments)
- In practice interventions can be a mixture of components with strong, weak or no benchmarks
- Control conditions can include core intervention components due to:
- Contamination
- Business as usual (BAU) contains shared components, different levels
- Similar theories, models of action
- But to index “fidelity”, we need to measure components within the control condition

## Linking Intervention Fidelity Assessment to Contemporary Models of Causality

- Rubin’s Causal Model:
- True causal effect of X is (YiTx – YiC)
- RCT methodology is the best approximation to the true effect
- Fidelity assessment within RCT-based causal analysis entails examining the difference between causal components in the intervention and counterfactual condition.
- Differencing causal conditions can be characterized as “achieved relative strength” of the contrast.
- Achieved Relative Strength (ARS) = tTx – tC
- ARS is a default index of fidelity

## Achieved Relative Strength =.15 

***Achieved Relative Strength =.15****** ***

***Expected****** ******Relative Strength =.25***

## In Practice....

- Identify core components in both groups
- e.g., via a Model of Change
- Establish bench marks for TTX and TC;
- Measure core components to derive tTx and tC
- e.g., via a “Logic model” based on Model of Change
- With multiple components and multiple methods of assessment; achieved relative strength needs to be:
***Standardized***, and
***Combined*** across:
- Multiple **indicators**
- Multiple **components**
- Multiple **levels** (HLM-wise)
- We turn to our examples....

## Assessing Implementation Fidelity in the Lab and in Classrooms: The Case of a Motivation Intervention

- Chris S. Hulleman
- Vanderbilt University

## The Theory of Change

- INTEREST

- PERFORMANCE

- MANIPULATED RELEVANCE

- Adapted from:
- Hulleman (2008); Hulleman, Godes, Hendricks, & Harackiewicz (2008);
- Hulleman & Harackiewicz (2008); Hulleman, Hendricks, & Harackiewicz (2007); Eccles et al. (1983); Wigfield & Eccles (2002); Hulleman et al. (2008)

**The Theory of Change**

**Notes:**
Set up: As an example, we will use the motivational intervention designed by Hulleman et al intended to enhance student motivation and subsequently interest and performance.
Based on expectancy-value models of motivation, one component of motivation that has been identified to impact both interest and performance is utility value, or the perceived relevance of a task, activity, or course topic to the person’s life.
The motivational intervention designed and tested by Hulleman and colleagues was therefore aimed at increasing perceptions of utility value (for a course topic, lab activity). These increases in utility perceptions were then expected to lead to increases in performance and interest.

## Methods

|  | Laboratory | Classroom |
| --- | --- | --- |
| Sample | N = 107 undergraduates | N = 182 ninth-graders
13 classes
8 teachers 
3 high schools |
| Task | Mental Multiplication Technique | Biology, Physical Science, Physics |
| Treatment manipulation | Write about how the mental math technique is relevant to your life. | Pick a topic from science class and write about how it relates to your life. |
| Control manipulation | Write a description of a picture from the learning notebook. | Pick a topic from science class and write a summary of what you have learned. |
| Number of manipulations | 1 | 2 – 8 |
| Length of Study | 1 hour | 1 semester |
| Dependent Variable | Perceived Utility Value |  |

## Motivational Outcome

*g* = 0.05
- (p = .67)

**?**

**Notes:**
What factors were responsible for the diffusion in treatment effectiveness in the field? We investigated implementation fidelity.

## Fidelity Measurement and Achieved Relative Strength

- Simple intervention – one core component
-  Intervention fidelity:
- Defined as “quality of participant responsiveness”
- Rated on scale from 0 (none) to 3 (high)
- 2 independent raters, 88% agreement

## Quality of Responsiveness

|   | Laboratory |  |  |  | Classroom |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|   | C |  | Tx |  | C |  | Tx |  |
| Quality of Responsiveness | N | % | N | % | N | % | N | % |
| 0 | 47 | 100 | 7 | 11 | 86 | 96 | 38 | 41 |
| 1 | 0 | 0 | 15 | 24 | 4 | 4 | 40 | 43 |
| 2 | 0 | 0 | 29 | 46 | 0 | 0 | 14 | 15 |
| 3 | 0 | 0 | 12 | 19 | 0 | 0 | 0 | 0 |
| Total | 47 | 100 | 63 | 100 | 90 | 100 | 92 | 100 |
| Mean | 0.00 |  | 1.73 |  | 0.04 |  | 0.74 |  |
| SD | 0.00 |  | 0.90 |  | 0.21 |  | 0.71 |  |

**Notes:**
This table needs formatting work!

## Indexing Fidelity

- Absolute
- Compare observed fidelity (**t**Tx) to absolute or maximum level of fidelity (**T**Tx)
- Average
- Mean levels of observed fidelity (**t**Tx)
- Binary
- Yes/No treatment receipt based on fidelity scores
- Requires selection of cut-off value

**Notes:**
Somewhere in this section I may want to refer back to the original figure that Dave presents on Achieved Relative Strength and show each of these indices (absolute, average, and binary) as they map onto that figure.

## Fidelity Indices

|  |  | Conceptual | Laboratory | Classroom |
| --- | --- | --- | --- | --- |
| Absolute | Tx | 
 |  |  |
|  | C | 
  |  |  |
| Average | Tx | 
 | 1.73 | 0.74 |
|  | C | 

 | 0.00 | 0.04 |
| Binary | Tx | 
 |  |  |
|  | C | 
 |  |  |

**Notes:**
Could create a figure to go along with this table that mirrors the original figure that Dave will present in the intro. What do you guys think? Is that necessary? It seems like it might help people get the concept.

## Indexing Fidelity as Achieved Relative Strength

- Intervention Strength = Treatment – Control
- Achieved Relative Strength (ARS) Index
- Standardized difference in fidelity index across Tx and C
- Based on Hedges’ *g *(Hedges, 2007)
- Corrected for clustering in the classroom (ICC’s from .01 to .08)

**Notes:**
By itself, treatment fidelity does not help us distinguish the strength of the intervention – it is only in comparison to the control condition that we can determine the achieved relative strength of the intervention.
Achieved Relative Strength (ARS) Index = standardized difference in the fidelity index across treatment and control conditions (expressed in SD units). Based on Hedges’ g (Hedges, 2007).
Intra-class correlations (ICC’s) were actually quite small:
	ICC for quality of responsiveness = 0.08
	ICC for perceived utility value = 0.01

## Average ARS Index

- Where,
- 	    = mean for group 1 (**t**Tx )
- 	    = mean for group 2 (**t**C)
- 	ST = pooled within groups standard deviation
- 	nTx = treatment sample size
- 	nC = control sample size
- 	n = average cluster size
- 	*p* = Intra-class correlation (ICC)
- 	N = total sample size

**Group Difference**

**Sample Size Adjustment**

**Clustering Adjustment**

## Absolute and Binary ARS Indices

*Where, *
*p*Tx = proportion for the treatment group (**t**Tx )
*p*C = proportion for the control group (**t**C)
- nTx = treatment sample size
- nC = control sample size
- n = average cluster size
*p* = Intra-class correlation (ICC)
- N = total sample size

**Group Difference**

**Sample Size Adjustment**

**Clustering Adjustment**

**Notes:**
It seems like we’ll want to at least flash these formulas so that people can get an idea of how we calculated them. 

## Average ARS Index

- Infidelity

- “Infidelity”

-      TTx

- TC

- 100
- 66
- 33
- 0

**Treatment Strength**

**t****C**

**t ****tx**

- 3
- 2
- 1
- 0

**Average ARS Index**

- (0.74)-(0.04) = 0.70

**Notes:**
Not sure if this slide is necessary, but it more explicitly demonstrates how the ARS indices connect to the theory of change as outlined by Dave earlier.

## Achieved Relative Strength Indices

|  |  | Observed Fidelity |  | Lab vs. Class Contrasts |
| --- | --- | --- | --- | --- |
|  |  | Lab | Class | Lab - Class |
| Absolute | Tx | 0.58 | 0.25 |  |
|  | C | 0.00 | 0.01 |  |
|  | g | 1.72 | 0.80 | 0.92 |
| Average | Tx | 1.73 | 0.74 |  |
|  | C | 0.00 | 0.04 |  |
|  | g | 2.52 | 1.32 | 1.20 |
| Binary | Tx | 0.65 | 0.15 |  |
|  | C | 0.00 | 0.00 |  |
|  | g | 1.88 | 0.80 | 1.08 |

**Notes:**
This is an alternative way to present the ARS Indices (compare with previous slide - #20).

## Linking Achieved Relative Strength to Outcomes

**Notes:**
I’ll talk about the trend in the classroom in comparison to the significant effect in the lab. This leads to a consideration of the factors that may have contributed to infidelity and thus decreased treatment effectiveness in the classroom.

## Sources of Infidelity in the Classroom

- Student behaviors were nested within teacher behaviors
- Teacher dosage
- Frequency of responsiveness
- Student and teacher behaviors were used to predict treatment fidelity (i.e., quality of responsiveness).

**Notes:**
How can fidelity and achieved relative strength be improved in field settings?
Teacher dosage: the number of opportunities teachers provided for students to write essays
Frequency of responsiveness: number of essays completed by students

## Sources of Infidelity: Multi-level Analyses

- Part I: Baseline Analyses
- Identified the amount of residual variability in fidelity due to students and teachers.
- Due to missing data, we estimated a 2-level model (153 students, 6 teachers)
*Student:**	**Y**ij* = b0*j* + b1*j*(TREATMENT)*ij** *+ *r**ij*,
*Teacher:*	b0*j* = γ00 + *u*0*j*,
- 		b1*j* = γ10 + *u*10*j*

## Sources of Infidelity: Multi-level Analyses

- Part II: Explanatory Analyses
- Predicted residual variability in fidelity (quality of responsiveness) with frequency of responsiveness and teacher dosage
*Student:**	**Y**ij* = b0*j* + b1(TREATMENT)*ij** *+
**	****	****	****	****b****2****(RESPONSE FREQUENCY)*****ij****** ***+ *r**ij*
*Teacher:**	*b0*j* = γ00 + *u*0*j*
- 			b1*j* = γ10 + **b****10****(TEACHER DOSAGE)*****j****** ****+ u*10*j*
- 			b2*j* = γ20 + **b****20****(TEACHER DOSAGE)*****j****** ****+ u*20*j*

## Sources of Infidelity: Multi-level Analyses

|  | Baseline Model |  | Explanatory Model |  |
| --- | --- | --- | --- | --- |
| Variance Component | Residual Variance | % of Total | Variance | % Reduction |
| Level 1 (Student) | 0.15437* | 52 | 0.15346* | < 1 |
| Level 2 (Teacher) | 0.13971* | 48 | 0.04924 | 65* |
| Total | 0.29408 |  | 0.20270 |  |

- * *p* < .001.

## Case Summary

- The motivational intervention was more effective in the lab (*g* = 0.45) than field (*g* = 0.05).
- Using 3 indices of fidelity and, in turn, achieved relative treatment strength, revealed that:
- Classroom fidelity **< **Lab fidelity
- Achieved relative strength was about 1 SD less in the classroom than the laboratory
- Differences in achieved relative strength = differences motivational outcome, especially in the lab.
- Sources of fidelity: teacher (not student) factors

## Joy Lesnick

- Assessing Fidelity of Interventions with ***Multiple Components***: A Case of Assessing Preschool Interventions

## What Do We Mean By Multiple Components in Preschool Literacy Programs?

- How do you define preschool instruction?
- Academic content, materials, student-teacher interactions, student-student interactions, physical development, schedules & routines, assessment, family involvement, etc. etc.
- How would you measure implementation?
- Preschool Interventions:
- Are made up of  components (e.g., sets of activities and processes) that can be thought of as constructs;
- These constructs vary in meaning, across actors (e.g., developers, implementers, researchers);
- They are of varying levels of importance within the intervention; and
- These constructs are made up of smaller parts that need to be assessed.
- Multiple components makes assessing fidelity more challenging

**Notes:**
Thus far, we’ve been talking about measuring fidelity in an intervention with one component.  Things get a little more complicated when we look at programs with multiple components, but it is these kinds of programs that are most common in education and social settings.

In this portion of the presentation, I’ll be using the evaluation of a preschool curriculum as a case study for talking about fidelity.  So I’d like to start by thinking broadly about preschool instruction.  How would you define it?

There are lots of different parts – academic content, materials, interactions between student-teacher and between student-student, physical development, schedules and routines, assessment, family involvement, and so on.

When measuring the fidelity of the implementation of the program, we have to think systematically about all of the different components that make up a program.

Preschool interventions:  SLIDE

## Overview

- Four areas of consideration when assessing fidelity of programs with multiple components:
-   Specifying Multiple Components
-   Major Variations in Program Components
-   The ABCs of Item and Scale Construction
-   Aggregating Indices
- One caveat:  Very unusual circumstances
- Goal of this work:
- To build on the extensive evaluation work that had already been completed and use the case study to provide a framework for future efforts to measure fidelity of implementation.

**Notes:**
For this part of the presentation, I will use the evaluation of a preschool literacy curriculum as a case study for exploring fidelity of implementation.

We’ll discuss four areas of consideration when assessing fidelity of programs with multiple components.  Each of the four areas is illustrated by an example that arose in secondary analyses of the case study evaluation, which was conducted by Dale Farran, Mark Lipsey, Carol Bilbrey and their research team.

The four areas I’ll discuss are:
Specifying Multiple Components
Major Variations in Program Components
The ABCs of  Item and Scale Construction
Aggregating Indices

But first, one important caveat:
We worked under very unusual circumstances.   These analyses were undertaken after the evaluation had been completed.  It is rare to have the opportunity to do it ahead of time.  

Our intention in doing this work was to build on the extensive evaluation work that had already been completed and use the case study to provide a framework for future efforts to measure fidelity of implementation.  We are grateful to the PIs and research team for providing their data for this further examination into fidelity

## 1.  Specifying Multiple Components

- _Our Process_
- Extensive review of program materials
- Potentially hundreds of components
- How many indicators do we need to assess fidelity?

**Notes:**
The first area of consideration is specifying the multiple components of the program.

We began by conducting an extensive review of the program materials (including professional development materials when available).  We identified major and minor headings in the developers language when specifying components and created a diagram of the relationship between levels and components.

We were not surprised to find hundreds of components in the preschool curriculum.  We worked to differentiate the treatment and control conditions to help identify core differences between the two programs.  

We wondered – how many indicators do we need to assess fidelity?  Let’s look at the diagram  

## 1.  Specifying Multiple Components

- Constructs	     Sub-Constructs  	Facets	        Elements       Indicators**     **

**Notes:**
Here is a diagram showing selected constructs, sub-constructs, facets, and elements of the preschool curricula under investigation.

In the first column, there are six different constructs.  I focus in on instruction (in red), and point out three sub-constructs – materials, content, and processes.

The third column specifies 9 facets for instruction-content and 2 elements for processes-content.

The fourth column specifies 6 elements of the facet instruction-content-literacy.  4 indicators are displayed for each.  

So how many indicators do we need to assess fidelity?
If we have an indicator for each element – we’d have 6 constructs * 3 sub-constructs * ~5 facets * ~5 elements = 1800 items on a fidelity instrument.  Whew!  

## Conceptual differences between programs may happen at micro-levels 

Empirical differences between program implementation may happen at more macro levels

Theoretically expected differences vs. empirically observed differences
Must identify conceptual differences between programs at the smallest grain size at the outset, although may be able to detect empirical differences once implemented at higher macro levels 

- Conceptual differences between programs may happen at micro-levels
- Empirical differences between program implementation may happen at more macro levels
- Theoretically expected differences vs. empirically observed differences
- Must identify conceptual differences between programs at the smallest grain size at the outset, although may be able to detect empirical differences once implemented at higher macro levels

- Grain Size is Important

**Notes:**
Do we need 1800 items?  Probably not, BUT...

Conceptual differences between similar types of programs are more likely to occur at the more micro levels – so the grain size matters.   When we were specifying the program models in the case study, the two treatment programs were extremely similar at levels 1 and 2.  The only difference at level 3 occurred in terms of processes.  Both programs included literacy circles and thematic units in their design.  However, in one condition, the literacy circles and thematic units were scripted or highly structured.  In the other condition, they were not.  The content was the same but the processes – and specifically the structured nature of the programs was different.

This conceptual difference was the key difference between programs, so it became very important to develop indicators to assess that difference.

In reality, the empirical differences between programs may be evident at more macro levels once the programs are implemented and indicators are aggregated to higher levels, but in designing the fidelity assessment, we should start with the conceptual differences between programs and make sure the indicators are that specific. 

## 2.  Major Variations in Program Components

- One program often has some combination of these different types of components:
- Scripted (highly structured) activities
- Unscripted (unstructured) activities
- Nesting of activities
- Micro-level (discrete) activities
- Macro-level (extended) activities
- What you’re trying to measure will influence *how* to measure it -- and *how often* it needs to be measured.

**Notes:**
The first area for consideration when conducting fidelity assessment was the specification of multiple components.

The second area is in specifying the different types of program components and how fidelity assessment methods can be applied.

In programs with multiple components, it is common for one program to have different types of components, most notably scripted (or highly structured) activities – prescribed by a daily lesson plan, for example – and unscripted (or unstructured) activities – work with children in small groups or centers, for ex.

These scripted and unscripted activities are nested within larger, more extended activities – such as month-long thematic units.  We have to look at both the micro and macro levels.

## 2.  Major Variations in Program Components

| Type of Program Component | Example from the Case Study
 | Implications | Abs | Avg | Bin | ARS |
| --- | --- | --- | --- | --- | --- | --- |
| Scripted (highly structured) activities | In the first treatment condition, four scripted literacy circles are required.  There is known criteria for assessing fidelity. | Fidelity is the difference between the expected and observed values
TTx – tTx | Yes | Yes | ? | Yes |
| Unscripted (unstructured) activities | In the second treatment condition, literacy circles are required, but the specific content of those group meetings is not specified.   | There is unknown criteria for assessing fidelity.  We can only record what was done, or in comparison to control
tTx | No? | Yes? | ? | Yes |

**Abs:**	“Absolute Fidelity” Index:  what happened as compared to what should have happened – highest standard
**Avg:**  	Magnitude or exposure level; indicates what happened, but it’s not very meaningful – how do we know if level is good or bad?
**Bin**:	Binary Complier:  Can we set a benchmark to determine whether or not program component was successfully implemented? >30% for example? Is that realistic? Meaningful?
**ARS :**	Difference in magnitude between Tx and C – relative strength – is there enough difference to warrant a treatment effect?

**Notes:**
Looking first at the types of activities, we used the fidelity indices presented by Chris(?) earlier to explore whether it’s possible to get a fidelity index when assessing scripted and unscripted activities.

First, we looked at the scripted – or highly structured activities.  In one treatment condition in the case study, four literacy circles are required, and the content and goals of every literacy circle are provided in the teachers materials.  There is known criteria for assessing fidelity.  Fidelity is the difference between the expected and observed values.  

We can calculate an absolute fidelity index – comparing what happened to what should have happened – the highest index we have.
We can calculate the magnitude or exposure level – indicating what happened.  This measure may not be very meaningful – how do we know if it’s a good magnitude or not – but we can compute it.
We can calculate the difference in magnitude between Tx and C – using relative strength.  We are looking to see if there is enough difference to warrant a treatment effect.
We could potentially calculate a binary complier index by deciding on a benchmark to determine whether or not the program component was successfully implemented.  If >30% implementation, then successful?  Or if <30% failure, for example.  But is that realistic or meaningful?  Probably not in this circumstance.  It’s not as clear cut as taking a pill or not.

Next, we looked at the unscripted or unstructured activities.  These happened in both treatment conditions, but one example that contrasts the previous example is that in the second condition, literacy circles were required, but the specific content of those group meetings was not specified.  There is unknown criteria for assessing fidelity.  We can only record what was done, or in comparison to the control.
Calculating an absolute fidelity index is now more difficult – we are not quite sure what the absolute value is – the criteria is unknown. 
We can still calculate the magnitude or exposure level – indicating what happened.  As I mentioned early, this measure may not be very meaningful – how do we know if it’s a good magnitude or not – but we can compute it
We can calculate the difference in magnitude between Tx and C – using relative strength.  We are looking to see if there is enough difference to warrant a treatment effect.
Same problem with binary complier index that I indicated previously.

The question then becomes – how do you take these indices and work them into a full index when they’re based on different conceptions of fidelity?  A common metric is needed.  Achieved Relative Strength is the only one that is common across all (and is metric free), but it isn’t really fidelity any more.  But it may the be best that we can do.

- Dots under a microscope – what is it???

**Notes:**
So far, we’ve put our case study program under the microscope.  Here’s a short illustration of that.  Let’s imagine that perhaps the light colored squares are unscripted components and the darker color squares are scripted components .  Together they come together in some kind of picture.  Any ideas about what it is?

The eye of a monster? Clouds?  A hat with ears? Whale skin?

We’re looking at the fine grains of programs, but these fine grains are nested within a larger image – a larger image that we also have to take into account...

## Starry Night, Vincent Van Gogh, 1889

**Notes:**
This image under the microscope is Van Gogh’s Starry Night.  

In assessing fidelity, we’re looking at the fine grains, but also at the big picture of which the fine grains are a part.

## We must measure the trees... and also the forest...

- We must measure the trees... and also the forest...

- Micro-level (discrete) activities
- Depending on the condition, daily activities  (i.e. whole group time, small group time, center activities) may be scripted or unscripted and take place within larger structure of theme under study.
- Macro-level (extended) activities
- Month long thematic unit (is structured in treatment condition and unstructured in control)  is underlying extended structure within which scripted or unscripted micro activities take place.
- In multi-component programs, many activities are nested within larger activity structures.  This nesting has implications for fidelity analysis – what to measure and how to measure it.

**Notes:**
The point here is that when assessing fidelity, we have to measure the trees, but also the forest of which the trees are a part.

Fidelity assessment has to be done at both levels – the micro- and the macro.  

Micro-level or discrete activities are daily activities such as whole group time, small group time, center activities, etc., that may be scripted or unscripted and take place within the larger structure of the theme under study.

The macro-level activities are extended – such as a month-long thematic unit.  It is within the macro-structure that the discrete activities take place.

In short, in multi-component programs, many activities are nested within larger activity structures.  This nesting has implications for fidelity analysis – what to measure and how to measure it.

## 3. The ABCs of Item and Scale Construction

**A**im for one-to-one correspondence of indicators to component of interest
**B**alance items across components
**C**overage and quality are more important than the quantity of items

**Notes:**
The third area of consideration is Item and Scale Construction.  It is not an understatement to say that Measurement Matters.  We encountered three points along these lines – and they just happened to be ABC... (groan).

Aim for one-to-one correspondence of indicators to component of interest

Balance items across components

Coverage and quality are more important than the quantity of items

## Aim for one-to-one correspondence

_**A**_im for one-to-one correspondence

- Example of _more than one component_ being assessed in one item:
- [Does the teacher] Talk with children throughout the day, modeling correct grammar, teaching new vocabulary, and asking questions to encourage children to express their ideas in words? (Yes/No)
- Example of _one component_ being measured in each item:
- Teacher provides an environment wherein students can talk about what they are doing.
- Teacher listens attentively to students’ discussions and responses.
- Teacher models and/or encourages students to ask questions during class discussions.

- Diff bw T & C (Oral Lang)*:
- T:  1.80 (0.32)
- C:  1.36 (0.32)
**ARS ES:  1.38**
- T:  3.45 (0.87)
- C:  2.26 (0.57)
**ARS ES:  1.62**

***Data for the case study comes from an evaluation conducted by Dale Farran,      **
** ****Mark Lipsey, Carol Blibrey, et al.**

**Notes:**
One-to-one correspondence of items to components means that each item is attempting to assess only one program component. 

In the case study example, we encountered the following questions in the “Oral Language” element.

First, an example of more than one component being assessed in one item:
[Does the teacher] Talk with children throughout the day, modeling correct grammar, teaching new vocabulary, and asking questions to encourage children to express their ideas in words? (Yes/No)

And three examples of one component being measured in each item:
Teacher provides an environment wherein students can talk about what they are doing.
Teacher listens attentively to students’ discussions and responses.
Teacher models and/or encourages students to ask questions during class discussions.

We can see that the first question is assessing too many things with one question, and cannot possibly be nuanced enough with a yes/no response to capture the variation in observations.

The data confirm our suspicions.  The achieved relative strength ES between treatment and control for the first example is 1.38, while the ARS ES for the second set of examples is 1.62.

It’s not dramatic, but every little bit makes a difference.  (Alicia Keys agrees)

But there’s more than just better item construction going on here – the first set of questions uses a 2-point scale and the second set of questions is also on a 4-point scale.  Not only does measuring just one component make a difference, but using a more sensitive measure makes a difference.

## Balance items across components

_**B**_alance items across components

- How many items are needed for each scale?
- Oral-language over represented
- Scales with α<0.80 not reliable

| Literacy Content | #items | α |
| --- | --- | --- |
|     Oral language | 20 | 0.95 |
|     Language, comprehension, and response to text | 7 | 0.70 |
|     Book and print awareness | 2 | 0.80 |
|     Phonemic awareness | 3 | 0.68 |
|     Letter and word recognition | 7 | 0.76 |
|     Writing | 6 | 0.67 |
| Literacy Processes: |  |  |
|     Thematic Studies | 4 | 0.62 |
|     Structured Literacy Circles | 2 | 0.62 |

**Notes:**
The second lesson we learned about item and scale construction is to balance the number of items across components.

How many items are needed for each scale?  It depends.  But just looking at the number of items for each component, we see imbalance.  We also see imbalance across the reliabilities in terms of the alpha levels.

Scales with alphas<0.80 are not reliable.  

This leads to the C in our ABCs....

## Coverage and quality more important than quantity

_**C**_overage and quality more important than quantity

| Literacy Content | #items | α |
| --- | --- | --- |
|     Oral language | 20 | 0.95 |
|     Language, comprehension, and response to text | 7 | 0.70 |
|     Book and print awareness | 2 | 0.80 |
|     Phonemic awareness | 3 | 0.68 |
|     Letter and word recognition | 7 | 0.76 |
|     Writing | 6 | 0.67 |
| Literacy Processes: |  |  |
|     Thematic Studies | 4 | 0.62 |
|     Structured Literacy Circles | 2 | 0.62 |

- Two scales each have 2 items, but very different levels of reliability
- How many items are needed for each scale?
- Oral Language:  20 items.  Randomly selected items and recalculated alpha:
- 10 items: α = 0.92
- 8 items:   α = 0.90
- 6 items:   α = 0.88
- 5 items:   α = 0.82
- 4 items:   α = 0.73

**Notes:**
The C of Item and Scale construction says that coverage and quality are more important than quantity.

This is the same table as in the previous slide, and I have highlighted two scales with two items each.  Book and print awareness has 2 items and alpha = 0.80 while Structured Literacy Circles has 2 items and alpha = 0.62.  

The question remains – how many items are needed for each scale? It depends. 2 is enough for book and print awareness, but not for structured literacy circles.

What about oral language?  20 items have an alpha of 0.95.  We randomly selected a decreasing number of items to determine how many were necessary to have a reliable scale.  You can see that the alpha decreased as the items increased, but with 5 of the 20 items (randomly selected), the alpha was 0.82, still higher than our cut-off.

The point here is that you can overwork yourself by not trying to achieve balance between numbers and precision.  

It’s important to note that this information is currently unknown.  What is good practice?  What are good fidelity scales?  What is achievable in this area?  We don’t yet know, nor will we know for a decade – we don’t have an accumulated history yet to decide.

But we can do some things now – even a small pilot test is better than nothing for developing scales – if we’re expecting an 0.80 alpha, then a small sample will be ok.

And what will be very helpful is to keep track of this kind of information after evaluations are completed – so the data can be used for the next round.  We really see this as an iterative process.

## Aggregating Indices

- To weight or not to weight?  How do we decide?
- Possibilities:
- Theory
- Consensus
- $$ spent
- Time spent
- Case study example – 2 levels of aggregation  within and between:
- Unit-weight within facet:  “Instruction – Content – Literacy”
- Hypothetical weight across sub-construct:  “Instruction – Content”

**Notes:**
The final area of consideration deals with aggregating indices.  

In programs with multiple components, we are often interested in an overall fidelity index, so we must aggregate from the micro-levels up to the more macro-levels.  The question then becomes:  to weight or not to weight?  And How do we decide?

Some possibilities include:
Theory (program, research base)
Consensus (bw researchers, developers, implementers?)
$$ spent on different components of the program (does $$ = importance?)
Time spent on different components (is the most time spent on the most important components?)

In our case study example, we explored aggregation at two levels – although that only got us back up to the facet level.
We decided to unit-weight within the facet of Literacy content, and then explore some hypothetical weights across the subconstruct of instruction - content

## YOU ARE HERE....

- UNIT WEIGHT

- THEORY WEIGHT

- HOW WEIGHT?

- HOW WEIGHT?

**Notes:**
So, to revisit this diagram from earlier, in the next two slides I’ll describe how we unit-weighted the elements of the Instruction-content-literacy (blue shaded box) and how we used hypothetical theory weights for the facets of Instruction-content (green shaded box).  

Little yellow guy is where we are

## Aggregating Indices

| Literacy Content | Average Fidelity Index:  
Tx | Average Fidelity Index:  
C
 | “Absolute Fidelity” Index:  
Tx | “Absolute
Fidelity” Index:  
C | Achieved Relative Strength Fidelity Index
(Average) | Achieved Relative Strength Fidelity Index
(Absolute) |
| --- | --- | --- | --- | --- | --- | --- |
| Oral language | 1.82 | 1.40 | 91% | 70% | 1.36 | 0.53 |
| Language, comprehension, and response to text | 1.74 | 1.37 | 87% | 69% | 1.45 | 0.44 |
| Book and print awareness | 1.91 | 1.39 | 96% | 70% | 1.38 | 0.73 |
| Phonemic awareness | 1.73 | 1.48 | 87% | 74% | 0.74 | 0.32 |
| Letter and word recognition | 1.75 | 1.36 | 88% | 68% | 1.91 | 0.50 |
| Writing | 1.68 | 1.37 | 84% | 69% | 1.22 | 0.34 |
| Average – unit weighting | 1.77 | 1.38 | 89% | 75% | 1.34 | 0.48 |

- **clustering is ignored

- Unit-weight within facet:  Instruction – Content – Literacy

**Notes:**
The six elements in the first column represent the elements of the facet:  “Instruction – Content – Literacy.”  

The headings in the first row indicate the different fidelity indices for each of the elements.

In the bottom row, we have averaged each of the element’s fidelity measurements weighting each with the same unit weight.  It didn’t make sense to us to decide that one of the elements was more important than another – but perhaps developers or implementers would disagree.  Just for the sake of example, averages were calculated for each column individually

Unlike the findings presented earlier, there is a big difference in the ARS when using average fidelity and absolute fidelity.  I mentioned earlier that using the absolute index is difficult in unscripted programs because no “absolute” is available.  This big difference is an important contrast to the findings of the program with only one component that Chris described.  We find that the Tx group implemented at higher levels than the control, which is what we would expect, but deciding how to index the relative difference still requires consideration, given the difference between avg and abs ARS calculations.

## Aggregating Indices

| Instruction - Content | Treatment | Control | Hypothetical
Weight |
| --- | --- | --- | --- |
|     Literacy | 1.77 | 1.38 | 40% |
|     Math | 1.51 | 1.80 | 5% |
|     Social and Personal Development | 1.79 | 1.58 | 35% |
|     Scientific Thinking | 1.57 | 1.71 | 5% |
|     Social Studies | 1.84 | 1.41 | 5% |
|     Creative Arts | 1.66 | 1.32 | 5% |
|     Physical Development | 1.45 | 1.50 | 3% |
|     Technology | 1.45 | 1.57 | 2% |
|  |  |  | 100% |
| Unweighted Average | 1.63 | 1.53 |  |
| Weighted Average | 1.74 | 1.49 |  |

- Theory-weight across sub-construct (hypothetical)

**Notes:**
In this table, we attached hypothetical weights to the facets of the sub-construct “Instruction-Content”

There is a slight difference between the weighted and unweighted averages, and of course this would shift depending upon the weights assigned to each of the content areas.  I gave the highest weights to literacy and social and personal development, but the next step would be to decide on a method and strategy for specifying weights based on a decision made about the importance of each of the content areas in contributing to the overall program.

## YOU ARE HERE ...

- UNIT WEIGHT

- THEORY WEIGHT

- HOW WEIGHT?

- HOW WEIGHT?

**Notes:**
So we unit-weighted within the elements of literacy, and theory-weighted between the facets of instruction-content, but we’re still only at the 3rd level of our diagram. How would we weight at the subconstruct level?  Are materials, content, and processes equivalent?  Maybe or maybe not.  And at the construct level, the six constructs are not of equal weight – how would we aggregate up?  These are questions we are still exploring using the case study data.

There is a mix of scripted and unscripted components in this diagram, so in order to aggregate up, we have to have an index capable of ignoring the measurement component of the values by using a COMMON METRIC.  

When considering how to assess fidelity and aggregate indices to a larger fidelity index, it seems as if there are many options.  But in fact, if you want to avoid aggregation aggravation, you’ll need an index that ignores metric and allows aggregation across different types of components.  We have identified two possibilities in this case study of programs with multiple components.

ARS - absolute 
ARS - average

## Key Points and Future Issues

- Identifying and measuring, at a minimum, should include model-based core and necessary components;
- Collaborations among researchers,  developers and implementers is essential for specifying:
- Intervention models;
- Core and essential components;
- Benchmarks  for TTx  (e.g., an educationally meaningful dose; what level of X is needed to instigate change); and
- Tolerable adaptation

## Points and Issues

- Fidelity assessment serves two roles:
- Average causal difference between conditions; and
- Using fidelity measures to assess the effects of variation in implementation on outcomes.
- Should minimize “infidelity” and weak ARS:
- Pre-experimental assessment of TTx in the counterfactual condition...Is TTx > TC?
- Build operational models with positive implementation drivers
- Post-experimental (re)specification of the intervention: For example:
-  MAPARS = .3(planned prof.development)+.6(planned use of data for differentiated instruction)

## Points and Issues

- What does an ARS of 1.20 mean?
- We need experience and a normative framework:
- Cohen defined a small effect on outcomes as 0.20; medium as 0.50, and large as 0.80
- Overtime this may emerge for ARS