Funded by the VA HSR&D (Project# SDR 10-012)

Spotlight on Women Cyberseminar Series
Methodological Issues in 

Psychotherapy (and OthePsychotherapy (and Othe

Nonpharmacological) Treatm

Research

Paula P. Schnurr, Ph.D

VA National Center for PTSD, Executive Divis

Dartmouth Medical School

ADVANCING SCIENCE AND PROMOTING UNDERSTANDING OF TRAU
Context

• Growing number of women Veteran

- % among VA users is growing too

• Few studies of psychotherapy and 

studies of psychotherapy and 

non-drug interventions in women V

- Few studies in samples that permit ma

Few studies in samples that permit ma

female comparisons either

• P i i l f t d i h th Principles of studying psychothera

apply broadly, e.g., surgery, physic

th d ti therapy, education
CSP #494: CBT for PTSD in W

Question: Are the benefits of Prolo

Exposure greater than the benefits oExposure greater than the benefits o

therapy?

284 Female Veterans and Active-Duty Personn

Random Assignment

141 Total 143 T

143 T

Comparison 

Present C

141 Total

Prolonged Exposure (PE)

Trauma Focused

Exposure Therapy Therapy 
It’s all about con

“The fundamental goal of any betgroup experimental design and itassociated methodology is to hofactors consistent other than the variable about which cause variable about which cause

-and

-e

conclusions are to be drawn.

”

–T B k 

Tom 

Bor

kovec
We Have “Control Issues

• e.g., choosing a comparison condit

- No placebos and many options

• e.g., equating a comparison conditi

e.g., equating a comparison conditi

- Many factors to control, or not

• e.g., assigning therapists to conditi

- Need to control therapist effects
Characteristics of Psychothe

Characteristics of Psychothe

and Dru

g Interventions

Psychotherap

Treatment involves collaboration between participants & providers

+++ 

Blinding patients & providers is virtually impossible +++ Provider expertise can influence results

+++ 

Therapist (provider) adherence Therapist (provider) adherence +++

must be assessed

+++

Control conditions often have Control conditions often have +++ 

active elements +++ 
Special Considerations i

Special Considerations i

Ps

y py chotherapy Trials

• Choosing a comparison condition• Equating a comparison condition• Assi

gg p nin

g thera

pists to conditions

• Manualization• T i i i i d it i Training, supervision, an

d monitori

• Additional treatment

• Group-based treatments
Choosing a Comparison Con
Elements of Pre Elements of Pre

-Post Change 

Post Change 

Function of Treatment Gro
Control for Elements of Chan

Control Control for Elements Elements of Chan

Ps

y py g chothera chotherapy Desi

gns

Wait-list: controls for factors unrelated 

treatment (most threats to internal valtreatment (most threats to internal val

provides no information about mecha

Nonspecific/treatment as usual: also co

nonspecific therapeutic elements

Component control: also controls for no

s

p; g ecific elements

; isolates active in

gr

Other active treatment: control varies; t

whether treatments differ but not why whether treatments differ but not why
What to Choose? What to Choose?

What

’s the Question?

Wait list: does the treatment have bene

• Not for effectiveness questions

Nonspecific comparison/usual care Nonspecific comparison/usual care

: is 

effect greater than the effect of simp

to therapy or getting usual treatmentto therapy or getting usual treatment

Component control: why does it work? 

are the acti e ingredients? are the acti

v

e ingredients?

Other active treatment: is tx A better o

efficient or cost-effective than tx B?
Choosing a Comparison Treatm

Choosing

a Comparison Treatm

CSP #494

• Chose a nonspecific, present-centere

comparison group because: group because:

- PE shown to work in wait-list studies; lim

evidence from nonspecific designs evidence from nonspecific designs

- many VA patients seek help for current p

present focus enhances clinical validity apresent focus enhances clinical validity a

similarity to VA treatment 

- Possible to manualize and to equate with 

Possible to manualize and to equate with 

of sessions, format, some content, and h

- Possible to exclude active in

gredients of 
Effect Size as a Function of Com

Effect Size as

a Function of Com

Grou

p g , e.

g., vs. Treatment 

Effect s
Effect Sizes for Eye Moveme

Effect Sizes for Eye Moveme

Desensitization & Reprocessing fo

-0.2 0.8 1.8 

R

oth

b

a

m 19

9

7

1.32

0.97

R

oth

b

a

u

m 19

9

7

Wilson 1995Jensen 1994

0.92

0.75

0.7

Marcus 1997Scheck 1998Vaughn 1994

0.62

0.47

0.41

Vaughn 1994Carlson 1998Carlson 1998

0.03

0.31

-0.03

Devilly 1998Pitman 1996D

evilly 19

9

8

–Chemtob et al., 2000
Effect Sizes for Eye Moveme

Effect Sizes for Eye Moveme

Desensitization & Reprocessing fo

-0.2 0.8 1.8 

Rothbaum 1997

1.32

0.97

Rothbaum 1997Wilson 1995Jensen 1994M 1997 0.92

0.75

0.7

Marcus 1997Scheck 1998Vaughn 1994

0.62

0.47

0.41

0 03

Vaughn 1994Carlson 1998Carlson 1998 Comparison Group Key

Bl itli t t l

0.03

0.31

-0.03

Devilly 1998Pitman 1996Devilly 1998

Blue = waitli

s

t con

trol

Red = nonspecific controAqua = component contr

–Chemtob et al., 2000)
Rules of Thumb for Effect Si

Rules of Thumb for Effect Si

a Function of Com

p 

arison G

Comparison

G

Expected

Eff t Si

N Nee

roup Effec

t Size P G er 

Gr

Waitlist Large

2

6

Nonspecific/TAU Medium 64Component/Active Small 393

Estimates for 2-group, 2-tailed t-test, at .80 power, p = 
Equating a Comparison Con

• Amount of therapy, e.g., number of 

sessions

• Format, e.g., homework, group or Format, e.g., homework, group or

individual

• Type of therapist, e.g., MD vs. PhD• Rationale/credibility• Treatment overlap/what is 

“common
Meta-Analysis of Equivalen

Analysis of Equivalen

Effects in Psy py chotherapy

# of 

st di u es d 

Inequivalent 8 .47 

structureEilt qu valent 13 15 

structure

13 .15 

Inequivalent > equivalent, p = .00

–Baskin et al., 2003
How Much to Equate?

• Depends on which differences could 

(not 

p y ossibl

y

) bias outcomes

• Control to the point of 

“reasonablene

- A bl ld t diff A reasonable person would accep

t differ

between treatments as true differences

- Treatment integrity is preserved Treatment integrity is preserved

• Consider:

- 10 90-mi t i it t inu

te sessions in 

targe

t

tx vs. 10 

sessions in comparison tx?

- if CBT vs relaxation no written homewo

if CBT vs. relaxation, no written homewo

relaxation?
Example:

Equating Treatments in CSP 

Conditions equated on:• # and duration of sessions• Individual format Individual format• Use of manual• Initial psychoeducation in first sess• R ti l ( lth h t diff d) Rationale (although type differed)• Homework ( g yp ) although type differed)
Assigning Therapists to Cond

• (There should always be > 1 the• Outcomes within therapists are 

to be clustered

• Important to control for therapis

effects due to differential skill effects due to differential skill, 

enthusiasm, warmth
Meta-Analysis of Therapist E

Analysis of Therapist E

(Crits-Cristop ) h et al., 1991)

Simple rs P

CBT (vs. Dynamic) -.33* Use of manual Use of manual -.45** Therapist experience -.41** Treatment length .28 

Variance due to therapist effects = 

(range = 0%-48.7%)

N=15 studies, 27 treatment conditions, 141 therapists*p<.10 **p<.05
Approaches to Assigning The

• Each therapist delivers 

both/all txs

• CSP #494: 2 

both/all txs per condition

• Different therapists 

deliver each tx

per condition

- Not PE expe

deliver each tx had CBT tra

- Assigned by 

convenience or

had CBT tra

- Therapists r

convenience or

- Sa

m

ple

siz

e

expertise 

- Randomized

Sa p e s e analysis accclustering

- ICC 05 ~.
Each Therapist Delivers Bot

• Useful if few therapists (usually sm

single

-site design) site design)

• Problems: therapist delivers treatm

with differential skill, enthusiasm, wdifficulty separating treatmentss

• Remedies: Careful training and 

su

p ;p y ervision

; inde

pendent fidelit

y

monitoring is essential
Different Therapists Deliver 

• Useful if experts needed or more th

therapists (usually large N multisite therapists (usually large

N, multisite

design)

• Problems: therapist effects still pos

discordant preferences (if randomiz

compensat tt i ory s

tra

tegies

• Remedies: Randomization

; g

 training

supervision, and fidelity monitoring important
Manualization

• Manuals ensure consistent treatme

delivery and permit replication delivery and permit replication

• Detail is essential

- Session by session guidelines- Specific prompts and suggestions- How to address noncompliance, crises

• If treatment can If treatment can’t be manualized, t be manualized,

alternative strategies may be neede

- e.g, y p , 

., content analysis of tapes, chart no

patient reports
Training vs Supervision v

Training vs. Supervision v

Monitorin

g

• Training: teaching therapists how to 

deliver the treatment deliver the treatment

• Supervision: providing feedback to 

therapists during the study

• Monitoring: independently checking 

: independently checking 

fidelity and competence

- checking alliance and process are type

checking alliance and process are type

monitoring too
Study Training and Supervi

• Training

-

Type and amount

- Training cases

• Supervision

- Audiotaping/videotaping- F / requency/type

• Should be more rigorous in efficacy • In CSP #494, in-person workshops fo

b

y 1-2 trainin

g ; cases

; individual 

supervision throughout
Monitoring Adherence an

Monitoring Adherence an

Com

petence

• Format

- C id id / di t % f i Consider video

/audi

o

tape, 

% o

f session

monitored, and basis for selecting 

• Monitor should be independent of tr

Monitor should be independent of tr

and supervision process

• M dt t i Measures nee

d

to cap

ture unique, c

and proscribed elements and allow 

comparison of treatments comparison of treatments

• In CSP #494, ~8% of sessions monit

oversampli f k i ling o

f

key sessions
Additional Treatment

• Many patients entering trials are rec

concurrent therapy concurrent therapy

• Medications? - Most psychotherapy studies allow medica• Other treatment for the same disord

Other treatment for the same disord

- Usually contraindicated for scientific o

reasons

• Other psychotherapy?

- Supportive and self Supportive and self

-help usually allowe

help usually allowe
Suggestions for Allowin

Suggestions for Allowin

Additional Treatment

• In efficacy studies, allow what is neces

ethical reasons ethical reasons

• In effectiveness studies, also allow trea

that do not interfere with treatment that do not interfere with treatment

• Medications should be stabilized befor• Discourage/disallow concurrent PTSD 

treatment 

• Discourage change unless clinically ne• Measure co Measure co

-therapy to check for comp

therapy to check for comp

strategies
Group-Based Treatment

• Group clustering should be accounte

sample size projection and data analysample size projection and data analy

whether randomization is by group or 

individual

• Same principle applies for clustering 

therapists
Effect of Group Format on Sa

Effect of Group Format on Sa

Size Calculation

The whole is less than the sum of the pVariance Inflation Factor to adjust for 

nonindependence:

VIF = 1 + (# group members - 1)ICC

Examples for groups of 6 assuming desired Examples for groups of

6, assuming desired

α 

power = .80 to find an effect of d = .50:

ICC VIF Sample needed (per group) Sample needed (per group).00 1.00 64.10 1.50 96.20 2 00 . 128.30 2.50 160
Effects of Corrected Analys

Effects of Corrected Analys

Results of Grou

p Treatme

• Baldwin et al. 

( ) 2006

)

reanalysis of si

gnificant tests for 

group evidencebased treatments

- Corrected dfs-

Varied ICC assumptions
Recommendations

1

. Address unique methodological iss

Address unique methodological iss

addition to the usual issues in trials

manuals, therapist assignment, addtreatment, training, and supervision

2. Ensure adequate statistical power a

address thera

p gp ist effects and 

grou

p

clustering in analysis and sample sprojections

3. Choose the comparison group that 

appropriate for your question appropriate for your question
Questions? Comment

For more information: Schnurr, P.P. (2007). Thand hard places in psychotherapy outcome resJ l f T ti St Journa

l o

f

Traumatic Stress, 20, 779

-792.

paula.schnurr@dartmouth.edu