**Transcript of Cyberseminar**

**VIReC Clinical Informatics Seminar**

**PE-Web: A Web-Based Training Program for Providers in Prolonged Exposure for PTSD**

**Presenter: Kenneth J. Ruggiero, Ph.D.**

**January 21, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact: virec@va.gov*

*Moderator:*Good morning and good afternoon, everyone.  This session is part of the VA information resource center’s ongoing clinical informatics cyber seminar series.  The series’ aims are to provide information about research and quality improvement applications and clinical informatics, and also information about approaches for evaluating clinical informatics applications.  Thank you to CIDER for providing technical and promotional support for this series.

Questions will be monitored during the talk in the Q&A portion of Adobe Connect, and will be presented to the speaker at the end of this session.  A brief evaluation questionnaire will pop up when we close the session.  If possible, please stay until the very end and take a few moments to complete it.  Let us know if there is a specific topic area or a suggested speaker that you would like for us to consider for the future.

At this time, I would like to introduce our speaker for today, Dr. Kenneth Ruggiero.  Dr. Ruggiero is Associate Director and Research Health Scientist at the HSR&D Center of Innovation, Ralph H. Johnson VAMC.  He is also Professor and Director of Research Coordination in the Department of Psychiatry and Behavioral Sciences at Medical University of South Carolina.  Without further ado, may I present Dr. Ruggiero?

*Dr. Ruggiero:*Thank you, Erica.  I appreciate that.  I just wanted to—I’m glad that we have so many participants on the line.  I am excited to share with you Prolonged Exposure Web, PE-Web, which is an online web training program.  It takes about eight to ten hours to complete, and it’s designed primarily for VA providers who see veterans with PTSD, but also can be used by civilian providers who treat a wide variety of populations with PTSD or who might use prolonged exposure.

Most of the site is pretty veteran-centric, so a lot of the videos related to veteran scenarios and military backgrounds.  A lot of the cues are military related, so you certainly get that feel for it on the site.  At the same time, we’ve found that when we create sites like this, when we’re designing them, to increase training specific to a particular diagnosis like PTSD or depression, that they can be valuable when used with a variety of different populations.  Even those of you who might not spend a lot of your time treating veterans specifically would find benefit from this, I think.

As Erica said, I’m at the Charleston VA COIN, the HSR&D Center of Innovation here at the Ralph H. Johnson VAMC.  I’m also at the Medical University of South Carolina.  Here at the Medical University of South Carolina, we’ve actually had a long history—an eight-year, eight- or nine-year history—of developing these types of sites.

Let’s see if I could—there we go—remember how to go to the next slide.  What I’m going to cover in this presentation is I’m going to spend a very small amount of time talking about the behavioral health needs of veterans, different mental health diagnoses that are prevalent in the population.  I’m sure many of you already know that, so I won’t spend a lot of time with that.

Then I’ll talk about providers’ readiness to deliver evidence-based treatment.  In that context, I’ll talk about the VA dissemination initiative, where we’re rolling out prolonged exposure and also cognitive processing therapy and other interventions such as behavioral activation that have been shown to be effective in treatment of post-traumatic stress disorder and depression.

In that context, we are hoping that web training sites like PE-Web can be very valuable in the process.  I think there’s potential for it to have value in the pre-workshop phase and also in the post-workshop phase as a refresher course or as just an ongoing resource that providers can use.

I’ll talk about web-based training approaches in particular, and I’ll highlight one of the web-based resources that we developed here several years back, because we now have a lot of evaluation data over the years on that.  I’ll show you some of the user reactions that we’ve had, some of the knowledge change data that we have on that, and give you a little bit more insight into what we see in our databases about how people are using the site, how many people are accessing, how many people are completing, what types of knowledge change levels are we seeing, and so on and so forth.

I’ll talk briefly about the process of developing PE-Web.  I realize there are possibly some folks in the audience who are interested in developing sites like this, so learning a bit more about the steps that we took to put it together might be of value to you.  Finally, I’ll give you a brief description of the PE website.  I won’t spend a lot of time with that, because anybody can go to the site.  It is now fully publically accessible.  I’ll give you the website so that anybody can log on, if you’d like to, and use the training.  I’ll provide some screen shots and give an overview of each of the main sections of PE-Web, as well.

I have to also acknowledge that there were many—as you can imagine, a project like this is a major undertaking and it requires a lot of collaboration from a lot of different areas of the country.  First and foremost, Edna Foa was a key collaborator on this project.  She had her hand—I don’t know how she does it, given how busy she is and how senior she is, but she had her hand in every piece of this.

She is featured on videos on just about if not all of the components of the website—I believe every module on the website, or every one with the exception of one or two—she is, there is a video with her describing the component and explaining some of the key challenges that providers often encounter in those components.  Those, in and of themselves, are very valuable contributions, I believe, and unique.  In addition to that, with all of the content development that we had, with all of the outlining, all of the web design decisions; she had a role in all of those.  I thank her for that.

We also have several folks here.  We have content development experts—Alyssa Rheingold is the content development expert here locally at the Medical University of South Carolina.  Marty Strachan was also a major contributor to the content development at MUSC and at the Ralph H. Johnson VA.  I won’t go through everybody.

Ron Acierno is featured in a lot of our videos.  He’s the director of the PTSD clinical team at the Ralph H. Johnson VA Medical Center.  Our technology team included Mary Mauldin and Jonathan Coultas, who helped a lot with videos, and Cheryl-Lyn Samuels, who is our primary programmer.  She’s a person I go to when folks identify new issues that they’re encountering on the site as they’re using it currently.

I also had several external collaborators—Joe Ruzek who is the director of the dissemination branch at National Center for PTSD.  He obviously has major roles in the PE rollout, as well as Afsoon Eftekhari, who has been instrumental in alpha and beta testing and making sure that—she has reviewed the site I don’t know how many times, but an enormous number of times.  Also Ken Weingardt and Jessica Hamblen, who played a key role in helping to make some initial web design decisions and being involved at several steps throughout the process.

I’ll speak briefly about just a quick review, overview of veterans’ mental health needs.  As we know, veterans are at risk for a wide range of behavioral health problems.  The two in particular that are most relevant to this presentation are posttraumatic stress disorder and also depressed mood.  Substance use problems also are co-morbid, quite frequently, with posttraumatic stress disorder.  We also know that untreated PTSD and untreated depression, if they go for too long, may negatively affect overall health and wellness and will affect trajectories of recovery over time.

It’s important that we make sure that veterans have access to effective treatment for PTSD.  That’s been a major priory in VA for quite a while.  We really stepped up to the plate, I think, with the dissemination initiative, to make sure that providers across the nation have access to being able to deliver all the best treatments that are available for PTSD and depression.  I don’t know exactly how many people have gone through the rollout process, but I believe we’re well into the thousands at this point.

We know that prolonged exposure is one of the best practice interventions that were identified by the Institute of Medicine, so it’s important that we place priority on interventions like that that we know are effective in treating veterans.  That’s what we did here.

How does web-based training fit into all of this?  First of all, I mentioned the National Dissemination Initiative.  If you want to get a little bit more information on this, it’s about a three-year-old article now, so it’s a little bit outdated, but Brad Karlin, Joe Ruzek and others published a paper in *Journal of Traumatic Stress* where they talked a little bit more about some of the evaluation data they had on the National Dissemination Initiative.

Essentially, that entails providers going through about four-day workshop period for prolonged exposure.  They then need to go through an ongoing consultation process with several folks, where they—I don’t know too much about this, but they essentially follow two cases all the way through from start to finish and receive consultation on an ongoing basis as they complete those sites.

As I talk to you today about Prolonged Exposure Web, PE-Web, be aware that we strongly feel that PE-Web is not intended in any way, shape or form to serve as a stand-along training protocol.  It’s an eight- to ten-hour course; it’s meant to be used as an ongoing resource, as a preparation training resource for people who are about to enter a workshop, or as a refresher course, but certainly no way, shape or form as a stand-alone training protocol.

We set out, in this project, to identify sort of ways through technology that we can augment the training process.  We believe that if providers have access to ongoing training materials online that that can improve their preparation to deliver interventions like prolonged exposure.  There are lots of different ways that it can be used, as I mentioned, but in addition to that, before we end today I’ll tell you about some—an ongoing project that we have with Dr. Foa and her group to make smartphone resources available to supervisors and also to providers.  That’s just early on in the process at this point.  That is to say, essentially, that we’re looking for more ways beyond PE-Web technology can improve providers’ preparation and delivery of prolonged—interventions like prolonged exposure.

At this point, I think this opens it up for  a quick poll.  I just ask, “Have you completed a web-based training protocol in the past?”

*[Pause 11:52—12:06]*

Very interesting.  It looks like almost exactly 50-50.  I kind of expected that, but wasn’t too sure.  Looks like several folks have done that and may be familiar with some of the types of approaches that I’m talking about, and others are pretty new to the process.  That’s good.  That’s a good mix.

I wanted to talk a little bit about prior successes.  Not just to brag about the colleagues that I have here at the VA, but also at MUSC, but more importantly really to teach you what we’ve learned in the past.  Because PE-Web is something that’s a new training protocol that’s available now, but because we just recently made it available, we do not have a lot of evaluation data on that specifically.

I thought that we—it wouldn’t have much benefit to show you some of the evaluation data that we’ve collected for PE-Web at this early stage when I really can feature that we developed back in 2005, for which we’ve had eight years of data collection, and I can show you a lot more in terms of the number of completers, the number of people who register for the course, and what we’ve learned with regard to usability testing and knowledge change over time.

We’re pretty experienced in this; we’ve done this over a period of several years.  It started out with our development of trauma-focused cognitive behavioral therapy that—all the sites, you can see the web links there, and you’re welcome to use any of those sites.  All of them are freely accessible, and all of them provide CE credits, CE certificates at the end that you can turn in to your licensing board.

You may want to talk to your licensing board in advance if you’re not sure about whether or not they accept these.  In our experience, we haven’t had a lot of issues with that.  Because MUSC has the authority to provide these, has permission from APA and others to provide these online training courses, we haven’t had a lot of, encountered a lot of problems with people getting their certificates accepted.

Folks are welcome to go to those sites and use those sites, but Trauma Focused CBT was led by Ben Saunders here, and Dan Smith.  Cognitive Processing Therapy was led by Connie Best.  I was involved in several of these.  Helping Heroes, I led.  That was the development of a web training resource for providers who serve first responders and firefighters.

We featured in that approach therapeutic exposure and behavioral activation.  There are some critical differences between that and Prolonged Exposure Web, but you may also find Helping Heroes valuable.  It doesn’t feature any veteran scenarios, but it’s certainly relevant to the population in a number of ways.

Okay.  You can also see the name of the site—I can’t exactly remember how to use the pointer, so I better not do that.  *[Laughs]*  Actually, no, I think I might be able to do that.  You can see here, dark colored, I just circled the website if you would like to go to the website PE-Web, that’s how you get there.  It’s pe.musc.edu.

Okay.  I’ll go through some TF-CBT Web data.  It was funded by SAMHSA back in 2003, I believe.  It was, like PE-Web, and eight- to ten-hour, web-based training course for providers.  Just like PE-Web, it’s a free learning opportunity.  Nobody has to pay anything to use it.  You get a CE certificate at the end of the course.

It’s asynchronous learning, so there’s nobody on the other end.  You sort of learn at your own pace.  You could potentially complete it in one day, or you could complete it in two weeks.  In one case, you could see probably later, that it might take a thousand days to complete it!  There are some cases of that.

It was developed, took about two years to develop, and it was launched in October of 2005.  I have a few screen shots, and I’ll talk a little bit about some features that the site has which are similar to the features that PE-Web has, even though we use a different platform for that.  It is meant to train providers in trauma focused CBT for children.  It is—it provides ten hours of CE credits.  It’s, as I mentioned, a learn-at-your-own-pace course.

Our goal, essentially, is to make sure that all of our content, all of our explanations, all of our clinical challenge scenarios, are very concise, to the point, and not overly text-heavy.  We always do our best to try to limit the amount of text that we have on a page.  Sometimes that can get very difficult, because there are so many important components of several of the sites—of several of the interventions that we’ve been featuring on these sites.  That can often be a challenge, to make some of these explanations and pieces of content concise, but we do try to do our best to limit the amount of text and provide as many video demonstrations as we can so that people have a clear understanding of how different pieces of the protocol can be implemented.

We do try to provide a lot of video demonstrations in each component, and I believe that you’ll feel that way on PE-Web.  We have clinical scripts that you can use and you can print out before a session, if you think that that might be helpful in providing a rationale for providers—or, I’m sorry, for patients in addressing a certain challenge that you might be encountering in treatment.  We provide some of those scripts.  We also—in this case, for TF-CBT Web, one of our biggest features on the site was a cultural considerations component and a clinical challenge component.  We address a lot of the clinical challenges as well in PE-Web.

The evaluation data that I’m going to present to you is over a five-year period between October 2005 and September 2010.  It doesn’t cover the last three years, but I think you’ll find that it has a lot of data to really illustrate what we’ve learned from evaluation of the site.  Back in September of 2010, by that point, over that five year period, we had already accumulated 73,000 registered learners.

I should say at this point that we had no expectations whatsoever that we would have anywhere near that number of people.  We thought several thousand by that point would have gone onto the site and used it, but it really took off.  I would say that in the first six months of post-launch, this site probably had several hundred new learners every months.  That actually surprised us, that we had that many.  We expected, after the first six months—after that sort of initial excitement from the National Child Traumatic Stress Network wore off—we expected those numbers to decrease over time, but they’ve done nothing but increase over time.

On average, at that point about 40.4 learners had registered for the web site each day.  The last time I looked at the data, which would have been last year, we were up over 120,000 registered learners.  I know that that pace has actually continued to increase over time, which continues to be surprising to us.  A lot of people are coming to the site, a lot of people are finding value in the site.  I think a lot of people are using it as an ongoing resource.

Even ten percent of our audience by that point lived outside of the United States, with 111 countries represented, I think.  We’re up above 125 at this point.  Of course, most learners were from countries where English is prevalent.  On the other site, we had Spanish scripts and lots of resources that were in Spanish.  In this case, for PE-Web, we haven’t yet developed those pieces, at this point.

In terms of the degrees that we saw for learners, 75 percent roughly, three in four, had a master’s degree.  One in ten had a PhD.  About one in seven was a graduate student at the time that they accessed the site.  What were their professions?  About two in five were in social work.  About one in four were in counseling.  About one in five to six were in psychology.  Another nine percent in marital and family therapy.  You can see a diverse range, but mostly social work, psychology and counseling for that site.  Most folks had a small number of years of highly relevant clinical experience, less than five.  Many people were at all ends of the spectrum on that.

In terms of the registration versus completion, obviously not everybody who registers for the course will ultimately complete the course.  In our case, this used to be more around 30 percent, but it’s climbed over the years.  It is now up close to 50 percent.  About 47 percent of learners who had registered for the site actually completed it as of September 30, 2010.  Within the US, that number is a little bit higher in countries outside of the US.  It tends to be about one in five, at least at that point.  I don’t know how these numbers in particular have changed over time, but I can say, again, that the completion rate has certainly steadily climbed over time.

How long does it take the average person to complete?  On average, it is about 67 days, but that tends to be pretty rare.  I would say that actually you can see down a little bit further, 50 percent of people who complete the site do so within a two-week period.  That is kind of how it’s meant to be used.

Ideally, it should be completed within a two- to four-week period.  We see that three in every four completers completes within a 49 day period, so about a month and a half’s time, which is a reasonable amount of time.  It’s not so much time that a lot of forgetting is gonna happen between the time that you first log on and the time that you complete.  Several folks completed in one day or in one week.  One person apparently took five years or so to complete the site.  I don’t think that happens very often, fortunately.

When we look at knowledge change, we were very encouraged to see that on every module we were seeing significant knowledge increases from pre to post.  The only two cases where we didn’t see really clinically meaningful knowledge change was on Module Three and Module Six.  I can’t exactly remember what Modules those were, but I think it’s safe to say that the major reason why we weren’t seeing big, meaningful knowledge change there is because I think the pretest questions were set up in a way where people seemed to know the answers coming in.  As you can see, pre-knowledge levels are at 80 percent in both of those modules, and I think that’s why you’re not seeing a major increase.  In all other modules, you’re seeing about a 25 or so percentage increase, which is very meaningful, from pre to post.  Many of these folks already were familiar with TF-CBT before going into the site, which is why I’m—I was particularly impressed with those changes.

We’ve seen that with other courses as well.  I’m really only showing you some of the data from TF-CBT Web, but we’re seeing a lot of very similar patterns in the data from other sites as well.  The Helping Heroes site that I lead the development of, we’re seeing very similar patterns with the usability testing and also the knowledge change levels.  We’ve also—we only recently launched the Helping Heroes site, but we’ve already had several thousand providers complete the site at this point.

With regard to learner evaluations on the usability testing, we see that—maybe I should just point out real quickly that the, we’ve—agree is in yellow, strongly agree is in red, and we collapsed the strongly disagree and disagree into green, and that represents a small percentage for each of these items.  We asked folks whether they felt that the site was easy to navigate, and we saw that roughly 95 percent—to all of these questions, really—answered in a favorable way.

Whether it was easy to learn the content in the course; how they evaluated the look and feel of the site; whether they felt that the directions that we gave throughout the site were clearly outlined; whether the videos, particularly the demonstration videos, were helpful to them; whether the scripts were useful and practical things that could be used in the context of their sessions—most people felt very strongly that they were.

The clinical challenge scenarios, about 45 percent felt that those were especially helpful, and 95 percent overall felt that they were at least somewhat helpful.  The cultural challenges was something that people felt pretty good about in general.  That’s something that we’ve worked to improve since getting these evaluation data, because it probably performed more weakly than any other section in terms of the amount of people who strongly agreed with that statement.  That’s something that we’ll, we’re taking a look at.

Whether or not the techniques that we outlined were helpful; whether they felt—how well they felt they learned the pscyho ed component, the stress management component, the affective modulation component, the cognitive coping—you can see that we got very favorable feedback on each of these modules from folks, as well as cognitive processing, behavioral management, whether or not it helped them to engage parents in treatment, whether the content was appropriate to the clientele that they serve, whether it helped them appropriately identify and address problems and challenges that occurred in the context of treatment, and whether they will continue to use TF-CBT.  All across the board, we received some pretty favorable feedback on that.

Moving more specifically to our development process, for folks in the room who are interested in developing sites or interested in participating in the development process around other sites.  The primary goal for our project was to develop something that was user-friendly but still sophisticated and had a lot of video demonstrations.  A web course for providers who serve veterans and other adults, not just restricted to veterans, with PTSD.  It does—it is pretty, as I mentioned, pretty veteran-centric, and is designed for VA providers first and foremost, but I think it can be used by other providers who serve other populations with PTSD.  It’s designed to be fully sustainable and accessible.  It doesn’t cost anything for a user to access and complete it.  It also delivers a CE certificate at the end of the course that verifies the completion of the course.  That, as I mentioned, can be submitted to the licensing board by the learner.

We divided this particular course into 13 learning modules.  That was decided through a collaborative set of conversations.  Some of our provider training sites only have eight to ten modules, and others are a little bit more complicated and require a little bit more.  In this case, we have 13 learning modules.  We address all of the major components of prolonged exposure in the context of PE-Web.

The development process required a lot of levels of expertise.  Obviously, we needed expertise in PE.  I myself do not consider myself to be a fully trained expert in PE, so I needed a lot of expertise around that.  Dr. Foa, as I mentioned, played an integral role in every part of the project, as did Afsoon, Joe and Alyssa Rheingold.  Alyssa, as I mentioned, was our context expert locally in PE.

It was essential that we had staff that assisted with video production.  We already fortunately had several videos that had been created and that are used in the context of PE training standardly in the context of the PE training, the workshop training.  We were able to use some of those PE videos, demonstration videos, in the context of PE.  In fact, we were able to take some of the more lengthy videos and make them into smaller video segments, which I think will be easier on the learner who accesses PE-Web.  They had several existing videos, but we also filmed several videos here, as well, including Edna’s introductory videos.  We had some actors here at MUSC and at the VA, several staff members, assist with that process.  As I mentioned, we had existing DVD training videos that we used.

We needed web design and platform expertise, and we used MUSC for that.  They had the resources to provide that.  We spent a lot of time in content development, so it was essential that I had a team that could meet on site, locally with in Alyssa, in a research assistant, Kai Welsh, and also in Marty Strachan, who drove a lot of the process with regard to some of the heavy lifting on content development.

We needed content entry folks so once the content was developed we needed to plug it onto the site, figure out exactly how it would be divided up across components.  Of course, we needed an extensive alpha and beta testing process which, in some ways, is still ongoing, because we’ve had a lot of new users recently.  They’re finding new things.  We will always continue to find new things as people log into the site and go through the training course.  I’m always happy to receive feedback about bugs that are coming up.

As you know, people over time will start to use different browsers, different Firefox browsers will come out, different Internet Explorer browsers, so it’s important that we stay aware of any new glitches that occur as a result of new browsers coming out and new platforms that are being released, so that we know how to update the site and make sure that it’s as user friendly as possible.

I’ll spend a little time quickly just going through the modules.  I’m not going to spend a lot of time on this, because anybody, as I mentioned, can freely access the site at this point, and can go through and learn either quickly or can go through as a learner, going through this whole course in detail, each of these components.

We start off by just going through the basics.  What is the structure of posttraumatic stress disorder?  Some information about the theoretical framework guiding delivery of prolonged exposure.  Some example patient profiles to illustrate what makes a good PE candidate.  We give several case descriptions to help the learner’s understanding in terms of how to relate it to their own cases, and also how to deliver PE.

The second module goes through assessment, so it provides an overview of different assessment approaches for prolonged exposure.  It talks about different structured diagnostic interviews that can be used in the process, different symptom scales that can be used.  How to find them.  Some special considerations, for example, if people have patients with PTSD and also serious mental illness.  Some other special considerations can be suicidality, traumatic brain injury.  There are some examples of that in the assessment section.  We also have a special populations module later on that I’ll show you briefly.  Also, within session assessments such as SUDS ratings introduces that, as well, to the learner.

Each of the major treatment components are addressed in Module Three.  An overview is provided of psychoeducation, in-vivo exposure, imaginal exposure and processing, which are the four major components of PE.  We also talk about some session logistics like audio taping, the length of sessions, the structure of sessions, how to set the agenda for sessions.

In the fourth module, we talk about the trauma interview and talk about how to provide the rationale to patients of prolonged exposure.  We provide an overview on that, an overview on how to discuss with patients PTSD symptoms and the result of the assessment.  How to do more information gathering in the context of the trauma interview, and also introducing breathing retraining at that phase.

The fifth module introduces psychoeducation, talks about how to discuss common reactions to potentially traumatic events with your patients.  Talks about how to introduce the concept of the relation between traumatic events and folk’s symptoms that often can be—especially when there are multiple traumatic events, as is often evident in the patients that we serve.

That can often be tricky to try to tease that apart.  We don’t even necessarily encourage people to fully tease that apart, but that can be a source of, a challenging issue to discuss with patients.  Helping providers to articulate that a little bit better in the context of treatment.  To talk about common clinical challenges, what to do if suicidal ideations are reported, etcetera.

Module Six talks about in-vivo exposure.  How to set up the rationale for that.  That obviously can be a very challenging module if providers are not fully trained and don’t have a lot of experience implementing that.  We spend a lot of time in the rationale, in making sure the people are using SUDS ratings appropriately.  Sorry for that typo there, but constructing the hierarchy is one of the things that’s primary in that module.  Figuring out how to do that exactly, navigating that process.  Some examples are provided around that.  How to assign homework, and how to monitor homework over time.  How to address barriers in homework, and some trouble shooting around all those things.

Imaginal exposure—how to provide the rationale to provider, or to patients.  How to implement it, particularly some challenge scenarios.  How to implement it with patients who have multiple traumatic events.  How to trouble shoot around over engagement, under engagement.  When patients exhibit safety behavior and avoidance, how to address that.  Homework activities around that as well.

Also, the—it looks like I have a second typo there.  I’m sorry about that, too—with processing, implementation of the processing piece, do’s and don’ts.  Won’t go into a lot of detail there.  Same thing with hot spots; how to identify hot spots and how to make sure that you’re addressing certain hot spots within the context of imaginal exposure.  How to implement that and how to sort of repeatedly go through that process with patients and some clinical challenge scenarios on there.  For example, what if you identify multiple hot spots when you’re delivering prolonged exposure?  How do you prioritize them moving forward?

Homework, which tends to be a pretty significant challenge for providers.  How to address that, how to present the rationale to patients.  How to review homework, how to motivate patients to complete their homework exercises.  Special issues.  As I mention—I referred earlier to over-engagement and under-engaging.  How to address those issues with your patients.  How to address anger, guilt and shame.

Again, as I mentioned, special populations is Module 12, where we address patients who have traumatic brain injury, substance use, major depressive disorder, MST—military sexual trauma—Axis II diagnoses, and how to deliver treatment with active-duty service members.  You’ll see all the content about those issues in Module 12 on the site, if you go to the site.

Finally, Module 13 talks a little bit about telehealth delivery.  We’ve learned a lot and we’ve done a lot of the work here, actually, at the Ralph H. Johnson VA Medical Center, led by Ron Acierno through a couple projects DOD and VA funded on delivery of prolonged exposure and therapeutic exposure and behavioral activation with—through telehealth.  Their findings are very promising.

We’ve published a couple papers in *Journal of Traumatic Stress* and a white paper in, I believe, *Telemedicine and E-Health*, led by Peter Turk that you can read up on, where we talked a little bit about deliver of PE and other interventions through telehealth.  You might find that useful if that’s something that you do at your site.

Finally, I just wanted to show a few screen shots for folks who aren’t inclined to actually go to the site.  Those of you who will go to the site, obviously you’ll see a lot of this there.  This is just an example of how we just provide an overview to each module, how we provide some education—this is a little bit more text-heavy than we tend to like, but—where you just provide some education to folks when you’re introducing a module.

Some common questions—this is actually, there’s a lot of text on this screen, but you can see that you can show and hide your answer on the screen.  It’s actually pretty condensed just to the questions, so you can see some of the questions that are frequently asked questions.  In this case, it’s in the assessment module, but we have these for every module.

You can just—if there’s a question that you’re particularly interested in having an answer to, then you can simply click on and show—sorry about that—show the answer to that, and it’ll expand and you can see what the answer is to that question.

This is an example of a pretest set of questions; when you see circles like this, that just means that there’s just one answer to the question, and usually the stem will help to guide you toward just one answer.  Then when you see boxes, that means that often multiple answers are accepted with that.

This is just an example of, another example of a frequently asked question section.  This is in the psychoeducation module.  This is an example of a page where we have a demonstration video.  This is an example of how to introduce SUDS to your patients, and an example of how that’s done.  In this case, this is a little bit longer than we tend to like, but it’s still not overly long.  It’s a three-minute video.

This is an example of how we provide the basis steps for creating a fear hierarchy with your patient.  Another example of some common questions, frequently asked questions that providers can give.  Another example with one of Edna’s introductory videos.  In this case, this is the imaginal exposure module, where we provide a brief overview of imaginal exposure, and there’s an introductory video that normally is about two minutes—between one and two and a half minutes in length.

I very much encourage you to go, even if you don’t go through the entire site, I encourage folks to go to the site and watch her introductory videos, because I found them to be very educational.  When we were—when she came here to film those, I remember thinking, “I’m glad that I’m here, because I’m learning a lot in the process, as well, as we’re filming these.”

Another example of introducing the processing component.  This is just showing sort of why we do processing.  Finally, several steps on how to conduct hot spot exposures as sort of an introductory piece to that module.  I thought that was the last one, but maybe we have one or two more.  This is from the special issues module, where we address over-engagement.  You can see Edna has a video where she defines and talks about how to address over- and under-engagement.

Special populations, where we address several things.  You can see over here, co-morbid—you can also see that you can skip around on the site.  You don’t have to go through in a fully linear way; you can sort of skip around and take a look.  If you’re particularly interested in military sexual trauma, you can click directly on that after you’ve gone through the intro.  I believe you do have to complete the intro and pre-test before you have more flexibility with the other components, but you can see that you can jump around a little bit with those links.  This is just a section on military sexual trauma where we give some of that background.

A quick poll, and I think there’s just a couple more slides.  Where do you think that this type of training approach would be most valuable in the training process?

*[Pause 42:03—42:30]*

That’s great.  It looks like most people feel that it can be used as an ongoing resource that providers can refer to as needed.  We found that in our experience that a lot of people use it that way, and just keep coming back to the site, printing out a script here and there, getting a refresher as they’re preparing for a session, things of that nature.  That tends to be where it’s often used.  I would also agree that, I think that’s actually pretty consistent with using it as a refresher course as well, and that was the second most common answer.  Pre-workshop is a way that we often do this with folks, as well, with other training sites that we’ve developed.  Okay, great.  Thank you for that.

Couple—let me make sure—okay, yes.  Couple next steps: as I mentioned earlier, I wanted to spend just a minute talking about, there was a recently-funded project through DOD that Dr. Foa obtained not too long ago.  She has asked us, underneath that project, to develop some smartphone resources that can be used by providers and supervisors to help them to stay on track and monitor their own treatment progress, monitor their own fidelity, and also for supervisors to monitor fidelity of providers.

We’re in the process of developing that; we just started this subcontract, so it’ll probably be completed in about a year or so.  We plan on making these fully available in a web-based format, but also IOS compatible, so it would be able to be downloaded straight onto an iPhone and also to an Android.  Those are exciting, and I think those could also help to augment the training process.

Just in conclusion—this is the final slide—we feel, and we’ve experienced through feedback from learners, that if we can develop some engaging web-based training courses, then they, that they could be beneficial to providers in helping them to either get refreshed with regard to content or even, in some cases, prepare for workshops, so that they show up for workshops with a greater level of  knowledge, which then allows you and the workshop deliverers to move on more quickly through the process and get to a higher level of doing some demonstrations and role plays with the work shop trainees.

We feel that web training really can augment the training process, that if we—one of the things that we’ve learned through this process, we felt that free modular training approaches would be pretty popular.  We had no idea just how popular, and they’ve been incredibly popular.  We hope they continue to be that way.  We found that it is important to offer CE certificates in the context of doing that, because more people will come to it and more people will learn from it as a result.

We feel that it’s a cost-effective approach.  Obviously, there are costs associated with putting together a team, developing all this content, doing the video role plays and so on and so forth.  All of that is costly, obviously, but when you start to look at cost per learner over time, and you’ve had these things up and running for years and years, then the amount of investment per learner is actually incredibly small over time.

Online training is just one of many ways to improve the quality of mental health services, as we’re learning.  I believe that’s it.  I think if folks have any questions that I can answer, I’m happy to address them.

*Moderator:*Great.  Thank you so much!  We do have a couple of questions.  If attendees have further questions, please type them in the Q&A box in the lower right.  First question:  what are ESTs?

*Dr. Ruggiero:*Sorry about that.  That stands for empirically supported treatments.  Generally, those are treatments that have undergone a certain level of support.  People define ESTs differently; some folks refer more broadly to evidence-supported treatments, and that could simply mean that there, that the evidence available for that support the principles that are guiding the intervention development process.  In most cases, people generally refer to ESTs to mean interventions that have performed well in randomized controlled trial evaluations.  That’s how I was using it in this context.

*Moderator:*Great.  Thank you.  Can you repeat the site address for the training?

*Dr. Ruggiero:*Sure.  Good question.  You can find it at pe.musc.edu.  I can—let me make sure.  If you type in—I can see if I can make this available with a www link, but you don’t type in the www.  It’s just http://pe.musc.edu.  That will take you right there.  I can actually—if I can draw that—or I could actually go back.  Let me see if I can find it quickly.  It’s one of the first slides.  There it is right there.  It’s actually already circled here: http://pe.musc.edu.  

*Moderator:*Perfect.  Thank you for that.  Would you recommend this training for someone who has not been formally trained but has done PE?

*Dr. Ruggiero:*Absolutely.  I think what you will find, if you haven’t been formally trained in PE but have done PE—actually, I shouldn’t say this, because I don’t know for sure that such individuals will have already read the book fully and gone through the workshop materials, but you’ll find that it’s very consistent with a lot of the content that was provided in the workshop and in the books.

A lot of the information that’s available on the site is intended to augment that as well.  The main benefit, I think, from the site above and beyond the workshop and the man—not the workshop, the book and the manual, is that the demonstration videos, I think, just give folks a greater level of understanding.  You can find a lot of things to refresh your knowledge on pretty quickly through the site.  I absolutely would do that.

Edna has stressed many times, and I couldn’t agree with her more, that it’s generally—it’s specifically advised against to use the site as a stand-alone training approach.  I think she feels strongly against the idea of just reading the book and then going and delivering treatment, or just going to the site and then going out and delivering treatment.  She feels very strongly that there are certain challenges that present themselves in the context of treatment where formal training can really be beneficial in that process.

*Moderator:*Great.  Thank you for that.  Here’s another question: is there any empirical evidence that online trainings contribute to outcomes, both provider competency and patient outcome, above and beyond in-person training workshops?

*Dr. Ruggiero:*That’s a great question.  I do not—my answer is that we are not, we do not have that answer yet, but that there’s currently an NIMH—I believe it’s NIMH, I’m not a hundred percent sure of this, but I believe it’s an NIMH-funded project where they are—I showed you earlier trauma-focused CBT.  This is this one, over here.  That’s where I gave you a lot of screen shots and some early evaluation data.  They’re taking that website and doing an evaluation of the site to see, to get the answer to that exact question.

The lead on that is Judy Cohen, who is also the lead author of trauma-focused CBT and the person whom we collaborated with when developing TF-CBT Web.  She’s leading that project, but it is an ongoing project and we don’t have the data in just yet.  It’s definitely a very important question; we need to get a better sense for whether or not this is affecting outcomes.

We’re also—I have a project at NIMH that I’m leading right now, where we’ve developed tablet-based resources that providers can use in the context of treatment.  The two—this is an early pilot right now, so the two primary outcomes, instead of clinical outcomes, that we’re looking at in the context of that study are provider fidelity and child engagement in that treatment.

I think we need to look at patient outcomes, provider fidelity, patient engagement, adherence with homework exercises—a lot of those types outcomes, in the context of this type of research.  We’re just starting to scratch the surface on that.  It’s a very good question and something that we’re definitely prioritizing, moving forward.

*Moderator:*Thank you.  Here’s another question: among those providers for whom this is the first introduction to PE, do you have any data to show how often this training, quote-unquote, lights a fire for providers to go onto receive more intensive training, such as workshops, etcetera?

*Dr. Ruggiero:*That is exactly my hope.  My hope is that the—we know, because we know that 60 or so thousand providers haven’t been trained in TF-CBT through formal workshops in the last eight years, but we do know that about 60,000 providers have gone through TF-CBT Web.  We are hoping that yes, this gives people some fundamental knowledge that—and some excitement and enthusiasm, and a lot more familiarity than they otherwise might have—about what the treatment protocol consists of.

We’re hoping that that motivates people to go get additional, more formal training, so that they can be fully prepared to implement the protocol.  We hope that in a lot of cases supervisors are trained, so that even if they’re not going to get formal training, at least the supervisor can provide some hands-on training and do some role plays with folks so that they can get some of that everyday training that they might need to get started with their first few cases before they really fully understand the protocol and fully understand how to address the clinical challenges that come out in the context of treatment.

*Moderator:*Great.  We have time for one or two more questions, if attendees have more?  Okay.  I’d like to thank Dr. Ruggiero for taking the time to develop and present this talk.  Please forward remaining questions to VIReC help desk at virec@va.gov.  Our next session is scheduled for Tuesday, February 18th, and is entitled *Better Mental Health Scheduling* and will be presented by Dr. Peter Fore.  We hope that you can join us.  I hope that everyone has a great afternoon.

*Male Voice:*Thank you all very much.

*[End of Audio]*

Page 1 of 15