**Methodological Issues in Psychotherapy (and other Nonpharmacological) Treatment Research**

>> We are approaching the top of the hour.  I would like to provide the introduction to our speaker today, who as I mentioned, is Dr. Paula Schnurr.  And Dr. Paula Schnurr is the Deputy Executive Director of the VA National Center for PTSD.  She is research professor of psychiatry at Dartmouth Medical School.  Her research focuses on substantive questions, including as those regarding physical health and PTSD, as well as methodological and physical issues.  More recently, she has spoken on clinical trials, PTSD treatment.  She has written or co-edited over 100 chapters, books, or journal articles.  And Dr. Schnurr is also the editor-in-chief of the Journal of Post Traumatic Stress which you can view online.  And Dr. Schnurr, I would like to pass it over to you at this time. 

>> Thanks, Molly, for the nice introduction.  For those of you in the audience who have articles to submit, I stepped down as JPTS editor in December and now I need to update the bio I guess.  And the new editor is Daniel Weiss and he would be very glad to receive your papers.  Molly, I want to make sure I understand how to move the slides. If I scroll down, do people see the next slide? 

>> Correct, yes, we do see the second slide at this time. 

>> The talk today is about methodological issues in psychotherapy research, and I think this is an important topic in women's health because study studying psychotherapy in women veterans helps us accommodate the growing numbers of women we see in the VA healthcare system. There have been really few studies of psychotherapy and other nondrug interventions in women veteran populations and few studies in samples that are large enough for with a large enough women to compare men and women, and now my focus is psychotherapy.  That’s what I am talking about today, but the principles that I will be reviewing really apply broadly to any nondrug intervention, even surgery, but also physical therapy, education, and so on. 

Now, bear with me and I will try to conduct a survey so I can ensure that I pitch the content appropriately for the audience. It looks like we have a few respondents to the survey. Which of the following best describes your primary role?  And Molly tells me we just wait a little while and then move on.  

>> Correct,  you’ll see the numbers start to slow down in just a minute.  <silence> It looks like we’ve probably reached the number of respondents at this time, if you’d like to read the answers.

>> Sure.  It looks like about, well we're still growing, but about 40% are VA mostly clinical, and about the same number are VA mostly research, and about another 9% are both VA clinical and research, and then there is very view from outside the VA more in research or research clinical positions. I think what I want to say and how I want to approach it will apply to both of you. If you do research, this is advice about how to do it. If you read research, if you are a clinician wanting to absorb it, especially this talk  -- certain parts of this I hope will be valuable to helping you evaluate the literature. 

Let's move on now. I am going to use a psychotherapy example. It is one of my studies. It is a VA cooperative study number 494 which was a randomized clinical trial of cognitive behavioral therapy – CBT, you may hear me use that acronym -- for treating post traumatic stress disorder in almost 300 female veterans and active duty personnel. In this study we -- I am learning how to use the pointer here. Forgive me, folks.  We compared a type of cognitive behavioral therapy known as prolonged exposure that focuses on past trauma with a present centered treatment that focused on current problems caused by past trauma. In many ways the present centered therapy actually reflected kind of the best of in a standardized version of usual care. That was our hope at least. 

>> Paula, if I may interject, I did not see the pointer come up on the screen. Did you have it depressed and held over? 

>> Can you see it up there now? 

>> No. You need to click on the bottom center of the screen on the arrow pointing at the red dot and hold it over the slide. 

>> Do you see it now? 

>> No I do not.

>> I am seeing it on my screen. 

>> Are you seeing your arrow or the laser pointer? 

>> I am seeing the little red dot. 

>> Keep it held down while you hover over. There we go. Now we can see it. 

>> Thank you, Molly. 

>> So this quote from Tom Borkovec sums up the challenge we face in psychotherapy research and in any clinical trial. It is all about control. I would like to read it just to emphasize what he has to say. “The fundamental goal of any between-group experimental design and its associated methodology is to hold all factors consistent other than the one variable about which cause-and-effect conclusions are to be drawn.” 

Now, in psychotherapy research we have control issues, choosing a comparison condition, because we don't have placebos and we actually have a lot of choices, how to equate the comparison condition with the treatment we’re studying.   There are many factors to control or not in many cases and even assigning therapists to conditions because we need to control therapist effects. 

Now, in truth even though we have control issues, there is no truly unique issues in psychotherapy research. The same issues that we face arise in drug studies but they're more prominent in psychotherapy, so one of them, collaboration between the patient and the provider is more important in psychotherapy research, and in fact it is the actual conversation between the patient and the provider that is the intervention. 

Another, although complete blinding isn't always possible in a drug study, providers and patients can guess sometimes. Blinding is virtually impossible in psychotherapy studies and even would be undesirable. Can you imagine a therapist not knowing what treatment they're delivering or a patient not knowing what treatment he or she is receiving?  So instead what we do in psychotherapy research is we ensure blinding of assessors at least in the best of studies. 

Now, therapy like surgery is a set of skills that vary each time they're performed, so expertise can significantly affect outcomes. Also, adherence must be assessed for quality control in psychotherapy, surgery, and any non-pharmacologic intervention.  And lastly, control conditions often have active therapeutic elements even supportive counseling which is a common control condition involves skills that we teach people when they go to graduate school. 

So here is a summary of what I want to review today, the special considerations in psychotherapy trials. Molly, are you seeing that pointer now? 

>> No, I am not. 

>> Each time I come onto the screen I have to hold it down? 

>> You need to reclick on the icon and then, yes, hold it down for as long as you want us to view it. 

>> How is that? Is it there? 

>> Yes, it is. 

>> That's great. This is sort of like a can you hear me now thing, and, folks, please bear with me. I don't have great eye-hand coordination. 

What we'll focus on is how to choose a comparison condition, how to equate a comparison condition with an active treatment, how to assign therapists to conditions, manualization, training, supervision, and monitoring, additional treatment and group based treatment. 

Now, the first issue is how to choose a comparison condition. Facing this task can make investigators sort of feel out of control which is why I love this slide, but in fact there are systematic ways to make a choice and get back in control. 

First, let's review factors that cause change from pre-to post treatment in three types of comparison groups. Wait list, in which there is no treatment, a nonspecific treatment group, and such as supportive counseling or treatment as usual and then active treatment, here I have A and B. In wait list group there is no treatment, so any change that occurs is due to factors that are unrelated to treatment. It might be just the natural course of illness, measurement, regression to the mean, and people are likely to change from before to after treatment but when they do, since there is no treatment, that isn't the explanation for the change. 

When people get a nonspecific treatment such as supportive counseling or treatment as usual that consists of a variety of techniques, you still have the factors unrelated to treatment but you also have nonspecific elements, the benefits of going to therapy, these might include support, instillation of hope, remoralization, attention, and so on, and these factors operate in active treatments, too but you also have the unique benefits of those active treatments and here I depicted A as having somewhat more benefit than B. We'll come back to this slide in a few minutes. 

Now, in this next slide, it talks about how different comparison groups control for these elements of change. A wait list group controls for factors unrelated to treatment. This is most threats to internal validity, but it doesn't provide any information about mechanism. A nonspecific treatment comparison such as supportive counseling or treatment as usual also controls for nonspecific therapeutic elements.  A component control group helps you determine the active ingredients and controls for the nonspecific elements as well.  By component control, I mean groups that consist of additions of effective therapy so adding cognitive restructuring to exposure therapy or dismantle an effective therapy that has multiple components, cognitive processing therapy has a cognitive component and a written component and in fact Patty has done a study comparing the combined treatment with the cognitive piece alone and the written treatment alone. Lastly, another active treatment control really gives you variable control. It tells you whether treatments differ and often not why. The most extreme example would be psychotherapy versus medication where you couldn't equate those interventions on many dimensions. 

So the question that arises, then, and probably this is the heart of what I want to talk about today or what's most important and what I have been talking about today is how do you choose among these groups if you can't use a placebo, what do you use, and the answer is actually a question which is what is your question? What do you want to ask with the study? 

A waitlist design enables you to ask whether a treatment has benefit, and it is appropriate at the early stages of investigation and should be used for efficacy studies only. If it is a question of effectiveness, whether the treatment works in usual practice, how it works in the real world, they be a more active treatment is necessary. A nonspecific comparison group is used to determine whether is anything special about a given treatment beyond the effects of just going to therapy or getting some kind of good, usual care. 

A component control group is used to ask the question of why a treatment works, so usually you're doing this after you know it works, and it works beyond -- the benefit is beyond the benefit of just going to therapy so here component control design should only be used when there is a good reason and an evidence-based to support a question about mechanism.  And lastly, another active treatment you should be using to determine whether one is better than the other or one is more cost effective is also appropriate to use this kind of design when you want to know whether a new treatment compares so the standard of care. 

Okay. In our VA cooperative study 494 we chose a nonspecific present center comparison group because prolonged exposure or target treatment had already been shown to work in waitlist studies, but there was very limited evidence from nonspecific designs, most control groups had been waitlist. 

Also, many VA patients seek help for current problems and so this focus on the present actually had good clinical validity and was very similar to usual VA care. It was also possible to manualize the treatment so we could standardize it across all of our sites and we had twelve sites, and to equate it with the prolonged exposure on format, number of sessions, some of the content even and the requirement of doing homework, and it was possible to exclude the active ingredients of the prolonged exposure so we could determine whether it was a specific exposure work that made the difference. 

Now we have come back to the slide you saw before, and now let's talk about how the choice of a comparison group affects the effects side.  Assume -- I will try to use pointer again here. Molly, can you see the pointer? 

>> Yes, we saw it briefly over active treatment A. 

>> Okay. Do you see it now? Do I have to press it it down to show you? 

>> Correct. You need to hold it down the entire time. 

>> Got it. Okay. 

>> Well, assume active treatment A is the treatment you're evaluating and its effect is due to the factors unrelated to treatment, the nonspecific elements, and then what's unique about treatment A. So a waitlist design is controlling only for the factors unrelated to treatment, and so the effect size is going to be a measure of the nonspecific and the unique elements of treatment A. 

A nonspecific design is also controlling for these nonspecific elements so the effect size is going to be smaller because it is only going to reflect the difference between this treatment and treatment A's unique effects. Lastly, the effect size is going to be smallest for comparison with treatment B because here you are controlling for factors unrelated to treatment, nonspecific elements, and the unique elements of treatment B, so your effect size will be smallest in that kind of active comparison. 

All right. So this is theory so far even if it makes sense, it is good to ask how this works in practice and what you are looking at here are data that come from a literature review performed for the first edition of the International Society for Traumatic Stress Studies (ISTSS) practice guidelines, and you're looking at effect sizes represented at ( indiscernible ) either between group effect sizes and the treatment is eye movement desensitization and reprocessing (EMDR). EMDR involves having a patient make rapid saccadic eye movements by following the therapist fingers back and forth while recalling a traumatic event, so the patient keeps the head fixed and the therapist moves the hands and the patient's eyes are supposed to follow the hand while recounting the traumatic memory, and the idea behind this, the hypothesis is that the eye movements induce a brain wave state that is more receptive to facilitate a trauma processing. 

So I have arranged these effect sizes in order with one exception at the bottom that I will explain in a minute, but you can see that the effect sizes essentially vary at the bottom from a minus .03 to a a .268, almost 2.7 at the top. 

Now, it was hard to determine whether EMDR worked or how well it worked but look what happens when you sort the effect sizes by the type of comparison groups. The dark blue bars at the top which indicate the largest effects come from studies that use the waitlist comparison group. 

So here I am talking about these studies. You can see that they're extremely large ranging from a .92 to a 2.68. The red bars in the middle are smaller, and they come from studies that use the nonspecific comparison group and then the two aqua bars at the bottom come from component control designs that attempted to determine whether the eye movements were necessary. So breaking down the data this way is very useful because if I can go back for a minute. The effect sizes were all over the place, and even if you could get a median effect size or average effect size, it would be hard to know what to make of this. Furthermore, it would be hard to know why EMDR was working, whether it was working any better than any other treatment and so on just looking at the data this way. 

But when you go back to stratifying the effect sizes by the type of comparison groups, you can make some very sensible conclusions. I think these data say that EMDR is beneficial and more so than just going to therapy, so EMDR works. That's why it is actually in the VA DOD practice guidelines. However, it doesn't appear that the eye movements are necessary, so why EMDR is working is not clear. The mechanism of action, at least in my view, is yet to be determined from looking at data like this. 

Now, because the type of comparison group affects the size of the treatment effect, it is very important to consider the type of comparison group when you are projecting sample size needed to achieve adequate statistical power. Cohen defines a large effect as a D of .8 and this is the proportion of a standard deviation by which the two groups differ, so by 8/10 of a standard deviation.  Medium effect is D equals .5 and small effect is D equals .2.  The sample size projections here are the N needed for a 2 group comparison to have 80 percent power or .05 two tailed test, and this is a very common scenario.

This slide is suggesting some rules of thumb that I derived from looking at met-analyses of psychotherapy for a variety of problems, not just PTSD, and please remember these are just suggestions, and they're not hard and fast rules, but I feel confident in sharing them with you because they make sense in light of a lot of the numbers I have seen. 

So for waitlist design it is appropriate to, I think, expect a large effect requiring only an N of 26 per group, so it is easy to power a study for a waitlist comparison which you should be using only at the earliest stages of research. For nonspecific design, expect a medium effect and this bumps up the needed sample size to 64 per group.  For component control or active comparison design, you have to expect a small effect, and plan for a very large number of participants per group, 393 if you are going by the book. This is a lot, and it is no surprise that we rarely see differences between active psychotherapy. Few studies are powered to detect these kind of differences. Now, this is for a 2 point comparison. You can bump up your power in other ways, for example, looking at slopes rather than end point comparisons, but if you are doing something simple like a T test or F test where you are looking at point by point comparisons, you really need a lot of subjects for an active comparison between active treatments. 

So now let's move on to talk about equating a comparison condition with an intervention you're trying to study. You need to consider a variety of factors, such as the amount of therapy, the number of sessions, format, whether people had to do homework, or you’re requiring group or individual formats, the type of therapist, whether people have to be MD versus a PhD, or could have a masters level or even not that, and the rationale between the comparison and the active treatment as well as the credibility and then treatment overlap, what's common to both treatments and what's different between them. 

Equivalence matters. This next slide depicts the results of a meta-analysis by Baskin and colleagues that examines the effects of equivalence on effect size in psychotherapy studies.  The investigators developed a scale for rating studies in terms of their equivalence between the comparison and the target treatment and then they divided the studies into equivalent and nonequivalent groups. So inequivalent, and then nonequivalent. What you see here is that the effect size, these again are D effect sizes, were much larger in the nonequivalent groups, .47 versus only .15 in studies that had equivalent structure, and this was the statistically significant difference. Now, this makes sense in terms of our prior discussion because equivalent comparison groups offer more control, so the more you control for, the smaller the effect. 

So this brings us to the question of how much you should equate a comparison treatment and an active treatment. I would like to propose that this depends on which differences could plausibly and not possibly bias outcomes. There is a lot of things you can control for between an active treatment and a comparison group. I think it makes sense to control to the point of reasonableness. What this means is a reasonable person would accept differences between the treatments as true differences and at the same time treatment integrity is preserved.  Let me give you an example of how you could operationalize this. So consider and this is actually a real comparison that we had discussed in one of our studies, in 10 90 minute sessions in a target treatment versus 10 60 minute sessions in a comparison treatment. The target treatment was 90 minutes, but the concern was in the VA we typically have 60 minute sessions.  Well, this would mean people in the active treatment got 50% more treatment. If the active treatment is better, you don't really know if it is because they got 50% more treatment. In that particular case we bumped up the comparison treatment to be 90 minutes because we couldn't shrink the active treatment down to fit into a 60 minute session.   Here is another case. If you're comparing cognitive behavioral therapy versus relaxation, should you have no written homework in relaxation, CBT requires written homework. For me written homework would be the opposite of relaxation, so a compromise would be to have relaxation homework, something that preserves the integrity of relaxation treatment. 

In 494 I will give you an example of how we equated the present-centered treatment with the prolonged exposure. The conditions were equated on number and duration and were both 10-90 minute sessions, in individual format, and the treatment was manualized, and we had initial psycho education in both sessions and we also had a very strong rationale for patients to help them understand why we were taking the particular treatment approach although the rationale differed between the two treatments. 

We also had homework in both treatments although this differed in the prolonged exposure homework required listening to audio tapes of the treatment sessions and also doing real world in Vivo exposure to situations that invoke trauma reminders. In the present center therapy, homework involved journaling instead about current problems. 

Now let's move to talking about how to assign therapists to conditions. At the outset I want to say there should always be more than one therapist in a study. A one therapist study is the analog of a one subject study and even if your study is small you should always have more than one therapist. Now, this is an important consideration because outcomes within therapists are likely to be clustered assuming that the therapist vary in effectiveness, some therapists will have better outcomes than other therapists, so it is important to control for these differential effects that might be due to differences in skill and enthusiasm and warmth, some variable that creates different outcomes. 

This slide shows the results of a meta analysis of therapist effects that of the done by Crist Kristoff awhile ago and I think still the data are very current in terms of more recent studies. They looked at 15 psychotherapy studies that included a total of 141 therapists, and they found that almost 9% of the variance due to in these studies was due to therapist effects.  Failing to account for this variance winds up biasing the estimate of the treatment effect because the therapist variance is bundled with the treatment effect, so you overestimate the benefits of the treatment. However, in this study the size of the treatment effect varied as a function of study characteristics, particularly the use of a manual was associated with decreased therapist effects, either in unit variant, the simple Rs column or multi-variant, that's the partial Rs column, both in both analyses using a manual was associated with smaller effects. Also, more therapeutic experience, more experienced therapists had smaller effects in both the unit variant and the multi-variant analyses.   CBT had a marginally significant effect in unit variant analysis that this appeared in multi-variant analysis because almost all of the CBT studies used the manual, so the most important take away here is that manualization can significantly help you decrease variants due to therapists. 

So there is several approaches you can use to assigning therapists to the treatment that they deliver. Each therapist can deliver both or all of the treatments or different therapists can deliver each. They could be assigned by convenience, who happens to be interested or available or by expertise and you can also randomize therapists which is something that we have done in our studies. For example, in our VA cooperative study 494 we had two therapists per condition per site, so a four per site across the twelve sites and they weren't P.E. experts and few had CBT training even, so they accepted randomization to the treatment that they delivered.  We accounted for the therapist effect in sample size, projection, and analysis, and we found a relatively small intercorrelation, only about .05 which makes sense we used the manual and had very careful supervision throughout the study which can also help you reduce variants due to therapist effect. 

So let's now talk about the scenario of each therapist delivering both or all of the treatments. This is useful if you have a few therapists, usually in a small study, single site design and not only useful, it is really necessary. This is what you have to do.  It is typically thought that this strategy where the therapist delivers both treatments controls for therapist effects, but it actually can create them because therapists may deliver treatments with differential skills, enthusiasm or warmth, so it doesn't really eliminate the problem. Therapists who are doing both treatments also may have difficulty separating the treatments, so that the remedy here because this is a necessity, the remedy is managing through careful training and supervision during the entire study.  You also want to do independent fidelity monitoring, having someone outside of the supervision looking at tapes or listening to tapes to ensure the distinction and the fidelity to the treatments. 

Now, the other strategy is having different therapists deliver each treatment. This is useful if you need experts. For example, if you are comparing say psychoanalysis with cognitive behavioral therapy, probably a study that won’t be done because it would be quite difficult but in that case it is very hard to teach people to be psychoanalysts, so you would take experts.  Also, if you have a large study and you can randomize therapists, will you be having different therapists deliver each which is really nice because you get the benefits of randomizing therapists which is are the same of randomizing subjects.  The problem is that therapist effects are still possible and the therapist who accepts randomization, just like our patients who accept it, may find they have discordant preferences. In our experience this happens quite rarely.  In VA cooperative study 494 we had one therapist who is a cognitive behavioral therapist assigned to present center therapy who just felt it didn't work with her therapeutic approach and values and decided that she wanted to leave the study but out of a couple studies that we have done using this strategy, that is the only case. Still, it is possible. It is also possible that therapist assigned to the comparison condition may engage in compensatory strategies although if you're monitoring therapists, you can determine this and eliminate it.  So again, just as when you are having therapists deliver all of the treatments, you have to use very careful training, supervision, and fidelity monitoring to ensure that people deliver the treatment as safely as possible. 

We have been talking about manualization. It is really important to ensure the replicability of the treatment delivery within a given study or to permit people to replicate your findings.  Detail is essential. You want session by session guidelines, specific prompts and suggestions and how to address noncompliance and crises.  I would emphasize the last point is really important because much manual deviation comes when patients come in with a crisis that they want to talk about that day rather than doing session 3. Or who didn't do their homework because of some crisis and therapists trying to improvise what to do in these cases often proceed to in some cases extreme deviation from what the content of the session is supposed to be.  You may ask what if I have some kind of treatment that can't be manualized.  And if that's the case, I would still say try to write on outline and then use alternative strategies such as content analysis of tapes, re-examining chart notes, patient reports, and so on. I would always try to provide some kind of guideline insofar as possible. 

We have also been talking about training, supervision, and monitoring. Let me break that down to define these terms because they're sometimes used a bit interchangeably. Training refers to teaching therapists how to deliver a treatment. Supervision refers to providing feedback to therapists during the study so that they can correct their delivery and monitoring is independently checking on fidelity and competence. Here I am talking about outcome and checking therapeutic alliance or therapeutic process are also types of monitoring.  In terms of study training and supervision, you need to consider the type of training you want to use to do it live, by the web, reading, some combination of these things, with or without role playing and examples, the length of training needs to be determined, too, and you need to consider whether people should treat practice cases or training cases. This is quite common in many psychotherapy studies particularly efficacy studies requiring people to come to a high level of proficiency.   Supervision now occurs after training during the study to examine how therapists are doing and correct the any problems that they're having. Usually it is done by a supervisor reviewing audio or video tapes and the typical model has been individual therapist supervisor sessions. Increasingly we're seeing group supervision sessions, and frequency may start out as high as 1 to 1, 100% of the session being monitored and in individual phone calls and but now it typically declines as the therapist gets more proficient. It should be more rigorous in efficacy studies because here you're trying to establish whether the treatment works. In an effectiveness study you know the treatment works and you are trying to see how it works in the real world and the real world means more variability.  In CSP 494 we conducted in person workshops at the outset of the study followed by 1 to 2 training cases for the therapist and then we provided individual supervision throughout although we titrated down the amount of supervision as people saw more cases and got more proficient. 

Now, there are also several considerations when designing a protocol for monitoring adherence and competence. Regarding the format you need to consider whether you want to have your fidelity monitors examine video versus audio tape.  Video is preferable if possible because sometimes it is the nonverbal behaviors that are very important and you won't see that on an audio tape. You need to determine what percent of sessions will be monitored, and the basis for selecting sessions, whether you just randomly pull a percentage or you over sample a higher percentage of target sessions. In CSP 494 we monitored 8% of the sessions but then we over sampled some key sessions. The first time the exposure therapy was introduced, for example.   The fidelity monitor should be independent of the training and supervision process.  I need to emphasize this because the fidelity monitor is checking how well you did or how well the therapist did and therefore the monitor's feedback, good or bad, should not be given back to the supervisors or the therapist. I would say the one exception there would be if the fidelity monitor noted something that seemed to be an ethical violation but presumably the supervision process would have caught that.   Fidelity measures need to capture unique, common and prescribed elements and they need to allow the treatments to be compared so that you can determine whether there was equivalent fidelity across the active and the comparison treatment. 

Additional treatment is an issue because many patients entering trials are receiving concurrent therapies.  Often in drug studies it’s quite remarkable that they will ask people to come off all drugs, not only off all drugs but to stop seeing longstanding therapist and stop engaging in any kind of psychotherapy. Psychotherapy research is not as fussy, and many patients on some kind of treatment, concurrent treatment are allowed. Most psychotherapy studies allow medication, and I think the logic is that as people still have PTSD and meet the criteria, whatever the medication is doing isn't enough, and we're not going to worry about it.  In contrast, other treatment for PTSD is usually contra-indicated for scientific or more commonly for safety reasons because safety is a big concern. It could harm the patient if he or she receives PTSD care from two different therapists who aren't collaborating. This is really something important, so a patient may discontinue active PTSD treatment with an existing therapist, but continue to see that therapist for support of check-ins while undergoing an active PTSD treatment. Other psychotherapies such as supportive counseling as I just said and self help is usually allowed and I would encourage this to the extent possible. 

Now, in terms of suggestions for allowing additional treatment, I recommend that if you are doing an efficacy study you allow what's necessary for ethical reasons. Here you want some stricter control.  In effectiveness studies I would also allow treatments that don't interfere with whatever treatments you're testing. Medications I think should be allowed but should be stabilized before study entry.  I would discourage or disallow concurrent PTSD treatment in almost all cases. Also discourage changing these treatments unless they're clinically necessary, that medication changes are required to help a patient who is deterioriating significantly. Another thing is that it is important to measure co-therapy to check for compensatory strategies. For example, in 494, participants who are assigned to the present center condition were more likely to have medication changes during the study's therapy and we think what was happening is the providers were trying to compensate for the fact that these patients, although on average they got a lot better, they weren't getting as better as the patients in the prolonged exposure.  By the way, that medication change didn’t sent seem to make a difference in outcome but it was happening. 

So now I will start wrapping up by going to the last topic of the day which is group based treatment. Group clustering should be accounted for in sample size projection and data analysis whether the randomization is by group or by individual patients. The same principle applies for clustering due to therapists, clusters are clusters. Essentially the problem is that the whole is less than the sum of the parts.  If outcomes are clustered each individual patient contributes less than 100% information, and you can calculate what's known as a variance inflation factor to adjust for nonindependence. It is calculated as one plus the number of group members minus one, times the interclass correlation for the group.  Again, variance inflaction factor is one plus a number of group members minus one times the ICC. So here below I am giving you some examples of the variance inflation factor as a function of different interclass correlations. We're assuming the groups are six people, desired P value alpha .05 and you want 80% power to find an effect of .5. Assuming that there is no correlation within groups or that you have individual patients, the ICC is there for zero, there’s no variant inflation. It is one. Assuming that you have an interclass correlation of as small as .1, the variant inflation factor is now 1.5 which means you need 60% more subjects. You start out needing 64 and if you have an ICC of just .1 you will need 50% more subjects and you can see here how with the variance inflation factor going up as a function of the interclass correlation, the sample size can increase dramatically. Generally in psychotherapy manualized studies we see something more in this ballpark, somewhere between the .05 that we found in our study and a .1 but still if it is only .1, 50% more subjects is a lot more subjects. You need to take into account. 

Again, this is in principle but it also pans out in reality. This is a study done by Baldwin and colleagues where is they reanalyze data on test of group based treatments, the data come from the APA's list of emperically supported group treatments and all of the treatments on the list have been shown to be effective. That is they had had statistically significant methodologically sound studies backing them up and none of them had accounted for the clustering in the data analysis. So what Baldwin and colleagues did is they corrected the degrees of freedom so that the degrees of freedom selected the number of groups rather than the number of patients, and then they made varying assumptions about the interclass correlations, so what you are looking at now in the graph is all studies were statistically significant at the outset.  When they corrected the degrees of freedom using the number of groups rather than the number of participants, only less than 70% of the studies were still significant, so only 70% of the studies would have made the list even assuming the interclass correlation was zero just doing the correct using the correct statistical model.  If the interclass correlation had been .05 which is what we found in our cooperative study, about 35% of the studies would have remained on the list so you would have had -- if these data had been correctly analyzed, only 35% of the studies would have made the cut, and this is very significant because we are over interpreting the effects of group psychotherapy. 

So let me now try to wrap up with some recommendations of how to pull this together. First of all, you should address in psychotherapy research address unique methodological issues in addition to the usual issues in clinical trials. These include the use of manuals, attention to assigning therapists to conditions, how to handle additional treatment, training, and supervision. 

Also make sure you have adequate statistical power and address therapist effects and group clustering in analysis and sample size projections.  And lastly, and most importantly, choose the comparison group that's appropriate for your question. Now, if you are sitting out there as a clinician, my recommendation is that you evaluate the studies that you are reading based on these criteria and see how well the investigators did.

So at this point I think we will turn things over for questions or comments. Molly, I am not able to get to the very last slide. 

>> Okay. Let me see if I can get there. Here we are. Okay. So before we begin the question and comments section, I know a lot of people joined the session after the start, so I would like to let you know that to type in your question or comment, please go to the upper left-hand corner of your screen at this time, and click on the Q&A tab and then simply type your question or comment into the top box and press ask.  Also, I would like to solicit a little feedback while we do the verbal Q&A. I have put up our feedback survey at this time. Please take a moment to answer this as we do take into account your opinions and we try and do future planning based on them. 

We'll begin the Q&A portion. The first is a comment: I want to say thank you for arranging this talk. It is very valuable information and we seldom get to hear it. Thank you already, Dr. Schnurr, and the second question is: regarding patient preferences followed by several question marks, so I believe they want a little further explanation on that. 

>> If there are specific -- I will try to answer and if there are specific questions about patient preferences, please type that in. The patient preferences are extremely important. They're not so much a methodological issue as a substantive issue. I think we're starting to learn at least in some cases patient preferences can affect outcome as people get what they want. I think that in terms -- you may be asking also about whether they should be measured at the very least and I think personally if you had asked me that a few years ago I would have said, well, okay. I think increasing the data that are emerging I would say, yeah, it is not a bad idea because we are seeing that in some cases they may affect the patient's compliance and actual benefit. Now, I am more familiar with this showing up in terms of medication versus psychotherapy and less so in terms of the type of psychotherapy received, but if you have a quick way to assess this, it could be a useful thing to add to a treatment battery. 

>> Thank you for that answer.  We do have quite a few questions flowing in at this time. The next one is: what is an acceptable percentage of sessions/cases for fidelity or quality control? 

>> This is a very hotly debated question. I think that the answer is more a number rather than a percentage. I don't know why I like to see this, but I like to see a couple hundred or so sessions, and anything is better than nothing, but the most important thing you want to do is get a representative sample of the underlying data, so if your study is really small and there aren't many sessions, it will be a much higher percentage. 

>> Thank you for that answer. The next question is: could you elaborate a bit on variant inflation factor? 

>> Sure. Can we go back to that slide? 

>> Let's hold the question for just a moment. I do want to give people a chance to fill out this form. 

>> I will try to elaborate without the slide. I thought that would help. This is a measure that is a calculated as a function of the clustering of outcomes within a group such as a group therapy or within a therapist as well as the number of individuals assigned to that group.  In a clustered situation the real unit of analysis is not the individual participant, it is the actual number of clusters. So you want to -- rather than adding a lot more people to a group, you want to add more groups because the whole is less than the sum of the parts in any given group, adding more groups gets you more bang for your buck, and adding more participants to a given group. This inflation factor is a way of gauging based on what you know or assume about the clustering of outcomes within groups how correlated they are, and how many more subjects you're going to need and in order to get the same statistical power you would have if the observations were independent either that the there was no group clustering even in a group design or you were doing individual randomization, individual treatment, sorry. 

>> Thank you very much for that answer. The next question is: what kind of limitations might there be on additional research studies a subject might participate in? 

>> I think that additional treatment studies should be considered very carefully with the same considerations about interfering with the treatments being contra-indicated. I also think the other issue is participant burden and so if doing a new study would make a person less likely to either do your treatment or do your assessments, I would think long and hard about allowing that. Personally, I like to allow as much freedom as possible, though, so I wouldn't be that conservative about this. I would also measure it just to see if it makes any difference, but think of the patient first, and then think of yourself second, being your ability to get the study done. 

>> Thank you for that answer. We do have quite a few more questions. If we do not get to them all, I would be happy to send them to you off line. 

>> Sure, and if we're allowed to stay online and people can stay online, I will do that either way. 

>> Absolutely. I do have the line open for ten minutes past the top of the hour. 

>> Okay. 

>> The next question is: in the ICCs, in parantheses regarding group comparisons, what are you correlating? Could you elaborate? 

>> That's a question I think I would rather address written that can be sent back -- I can send some formulas. Essentially you are looking at variance due to the therapist relative to the total therapist, and it is a way of breaking down the sums of squares to get that as a correlation. 

>> Thank you for that answer. The next question is: how can we measure the quality of supervision received? I am sorry, did I already read that one? 

>> No. 

>>Oh, Okay.

>> How can you measure the quality of supervision received? That's a good question. I have never been asked that before. You perhaps could if you want to look at the supervision, you would have to observe the supervision somehow. I think that you would have to video or audio tape the supervision and then have someone examine it. It might be quite labor intensive, so you also might want to think of some other strategies such as looking at the adherence of assuming that people -- supervisors are randomly or quasi-randomly assigned to who they supervise. You can look at adherence across supervisors, perhaps better supervisors would have more adherent therapists.  You can also ask the therapists what they think of the supervision. 

>> Thank you for that answer. The next question is: would you please say a little bit more about how the results presented on the EMDR studies actually showed that the eye movements don't appear to be the mechanism? 

>> Molly, are we able to go back to that slide? 

>> Yes, you are. 

>> In the studies, the component controlled studies, there was a very elegant design in which the therapist moved their hands but the patients were instructed to keep eyes fixed in the comparison condition. Everything was the same except the eye movements, and because the eye movements are presumed to be critical to the therapeutic -- to the mechanism of action, comparing eyes fixed versus eyes moving in -- I said it is very elegant and one of the nicest comparisons I’ve seen for looking at mechanism really drilling down. What are you looking at here, and I will try to get the pointer up and going again, what you’re looking at here are the effect sizes, the .31 in the Pitman study between eyes fixed and eyes moving. Eyes fixed should have deactivated the active ingredient of the EMDR and likewise should have deactivated that active ingredient.  So why I am saying I think from these studies the eye movements don't matter is that there was a small difference between the two comparisons even though there was a big difference in presumably what was active in the intervention. 

>>Thank you for that clarification. The next question is: starts with a comment, very helpful and informative. Thank you. The question is: do you know of any similar analyses, for example, accounting for training and integrity, group versus individuals, et cetera, using odds ratios, risk ratios or other epidemiological measures and parameters versus effect size for psychotherapy? 

>> I am not sure I understand the question, so I am sorry if I answer it incorrectly. If the question is about showing the effect sizes as a function of comparison group, using other measures of effect size, such as an odds ratio, I personally am not aware of that. I think it would be a great idea and I think we simply have not paid enough attention to the question of how effect sizes vary as a function of comparison group, or how the question that the study is answering varies as a function of comparison group as well. 

>> Thank you for that answer. We do have three questions remaining. The first being: given research findings of dropout rates for P.E. which is up to 40% or more and CPT, 20% or more, how do you address this specifically and methodologically? 

>> I think dropout is informative. What you should try to be doing, although we want patients to get as much benefit as possible, you should be measuring  everybody no matter what they do. Phil Lavori -- I heard him once give a very elaborate discussion of the issue of how to handle missing data that wound up with the punch line “you should avoid it”.  And so if patients drop out of treatment, you should at least measure what's happening to them. Sometimes they drop out because they're actually getting better and feel like they don't need it.  I think Pattie most recently is the developer of cognitive processing therapy found that in some of other work, so I think methodologically I am less concerned with trying to force it in a study but it is ideal if we can determine strategies that help patients stay in treatment however they can or however long they can so that they get more benefit of the treatment. With the strategy you still want to use an intention to treat analysis approach where you really confuse the main effect of the study based on randomization of all of the participants and then very carefully use strategy that is look at people with an adequate amount or who completed but remember once you're doing those analyses, are you not looking at randomized data. The ITT is the only analysis that's in the randomization, so that's why I say try to measure everybody as much as possible. 

>> Thank you for that answer. The next question is: could you speak more to medication stabilization prior to starting the study, the timeframe in which stabilization is said to occur? 

>> We tend to think about two months of stabilization, and this is in my hands at least has more to do with the patients and not to do with the science. If patients are on a new medication or new dose of medication, I think the last thing you want to do is be bothering them with what it takes to be in a psychotherapy research study, so we typically use two months. Some people use three months. At the very least I would say one month, but if it is actually new medications, I would give it a full two months. 

>> Thank you for that answer. Our final write-in seems to be either a follow-up to a previously written question or just a comment. I am sorry, it is hard to decipher. It starts with: for example, male patients may prefer male therapists, female patients may prefer female therapists especially for high stress states.  Or if the stress is partly due to sex based trauma and/or gender based discrimination etc. between patient therapist and inherent and cultural social factors that come into play and their male-female interactions effect. 

>> This is a follow-up to the question about patient preference, and I actually think that the person who wrote the question is articulating questions that would be great for research agenda.  So far we don't know very much about how these kind of factors, that is gender matching, affect outcome.  There was one study done using VA administrative data looking at how a patient and provider racial pairing affected outcome and that was only using administrative data, so by in large most studies haven't been designed to answer this question sufficiently. We know in the real world that some patients have very strong preferences. I think in research we're not seeing as much of that because patients who have those preferences vote with their feet and don't enter trials perhaps when they may not be able to choose the kind of therapist they have.  But as research in PTSD, psychotherapy and psychotherapy evolves, I think we are and should be doing more to understand how these patient matching factors can affect outcome and even affect participation. 

>> Excellent. Thank you for that very thorough response, and that does conclude our Q&A portion. That is all the questions that have been typed in so I would like to thank you very much for the comprehensive presentation and for staying after the top of the hour to answer questions and to the audience who stuck around and I will put back up your contact information for those who would like to contact you further.  And as I mentioned previously, this session has been recorded, so you can access it for later reference by going to the HSR&D website and click on the cyber seminar catalog and scroll through.  And thank you very much, Dr. Schnurr for your informative and excellent presentation. 

>> Thanks Molly. Thanks everybody. Bye bye. 

>> This does formally conclude the cyber seminar.