A Computer-Assisted CBT Tool to Enhance Fidelity in CBOCs
Phase: NA
Conditions: Depression
Sponsor: Central Arkansas Veterans Healthcare System

CALM_2 
v3; March 01, 2018 2017 
Page 1 of 26 
 
 
 
 
 
 
 
 
 
 
Project Title: A Computer-Assisted CBT Tool to Enhance Fidelity in CBOCs.  
Funding Agency: VA Health Services Research and Development Service 
Principal Investigator: Michael A. Cucciare, Ph.D.  
Date: v.3 March 01, 2018 
 
 
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 2 of 26 
 
Abstract 
Background and Significance: Randomized Controlled Trials (RCTs) have demonstrated that 
evidence-based psychotherapies (EBPs), particularly Cognitive Behavioral Therapy (CBT), are 
highly effective in treating anxiety and depression, the most common mental health disorders in 
primary care settings. Mental health (MH) providers in VA Community-Based-Out-Patient-
Clinics (CBOCs) are often located in rural areas and isolated from educational opportunities. 
Almost half of Veterans now use CBOCs. Studies have shown that the quality of delivery of 
EBPs (fidelity) impacts clinical outcomes. This study will test a computer-assisted tool (CALM 
Tools for Living) that increases fidelity to CBT in treating depression and four common anxiety 
disorders, including PTSD. Although results of a large RCT, the CALM study, suggested that the 
tool contributed to fidelity to the CBT protocol, this hypothesis has not been tested. This study 
will test the tool in rural CBOCs in VA VISN16.  
Objective: To evaluate the impact of a modified version of a computer-assisted CBT tool on 
providers’ fidelity to the CBT model and clinical outcomes, and to assess how best to support 
future implementation of this tool within VA CBOCs.   
Specific Aims/Hypothesis: This project has two aims. First, we plan to compare MH provider 
fidelity to CBT and clinical outcomes among providers who used the tool and those who did not. 
We hypothesize that clinicians who use the tool will have a higher fidelity to CBT and clinical 
outcomes among patients will be superior. Second, assess how best to support its future 
implementation in routine VA care.  
Methodology: This study will use a Type III hybrid effectiveness design. Thirty-four CBOC MH 
providers will be trained in CBT and randomized to use the tool or not. Both groups will receive 
external facilitation to encourage the full implementation of CBT into practice on the clinic level. 
MH providers will treat a total of 340 veterans. Patients will be assessed at baseline, three-, and 
six- months post treatment. Provider fidelity to the CBT protocol will be measured, and finally, a 
tool kit for future implementation of the tool will be disseminated.  
Impact: We expect the intervention to improve the technical quality of MH treatment in CBOCs 
and improve clinical outcomes among depressed and/or anxious Veterans receiving care in 
these settings.  
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 3 of 26 
 
List of Abbreviations 
 
 
CALM = Coordinated Anxiety Learning Management  
CBOC = Community-based Outpatient Clinic 
CBT = Cognitive Behavioral Therapy  
EBP = Evidence-based Practices 
EMR = Electronic Medical Record 
MH = Mental Health 
MHP= Mental Health Provider 
RCT = Randomized Controlled Trial 
VA = Veterans Health Administration 
VISN = Veterans Integrated Service Networks 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 4 of 26 
 
Contents 
Protocol Title: ................................................................................................................................ 5 
1.0 
Study Personnel ................................................................................................................. 5 
2.0 
Introduction ........................................................................................................................ 5 
3.0 
Objectives .......................................................................................................................... 7 
4.0 
Resources and Personnel .................................................................................................. 8 
5.0 
Study Procedures ............................................................................................................. 10 
5.1 Study Design ..................................................................................................................... 10 
5.2 Recruitment Methods ........................................................................................................ 13 
5.3 Informed Consent Procedures .......................................................................................... 15 
5.4 Inclusion/Exclusion Criteria ............................................................................................... 15 
5.5 Study Evaluations .............................................................................................................. 16 
5.6 Data Analysis .................................................................................................................... 17 
5.7 Withdrawal of Subjects ...................................................................................................... 20 
6.0 
Reporting .......................................................................................................................... 20 
7.0 
Privacy and Confidentiality ............................................................................................... 23 
8.0 
Communication Plan ........................................................................................................ 24 
9.0 
References ..................................................................................................................... 24 
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 5 of 26 
 
Protocol Title:  A Computer-Assisted CBT Tool to Enhance Fidelity in CBOCs 
1.0 
Study Personnel 
 
Principal Investigator 
Michael A. Cucciare, PhD 
Central Arkansas Veterans Healthcare System 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
2200 Fort Roots Drive 
North Little Rock, AR 72114 
501-257-1068 
Michael.Cucciare@va.gov 
 
Co-Investigators: 
Geoffrey Curran, Ph.D.  
Central Arkansas Veterans Healthcare System 
Center for Mental Healthcare and Outcomes Research (CeMHOR) 
2200 Fort Roots Drive 
North Little Rock, AR 72114 
501-257-1701 
 
 
Michael Kauth, Ph.D.  
Michael E. DeBakey VA Medical Center 
2002 Holcombe Blvd 
Houston, TX 77030 
713-794-8637 
 
Jeffrey Cully, Ph.D.  
 
Michael E. DeBakey VA Medical Center 
2002 Holcombe Blvd 
Houston, TX 77030 
713-794-8526 
 
2.0 
Introduction 
 
Multiple randomized controlled trials (RCTs) have demonstrated that evidence-based 
psychotherapies (EBPs), particularly Cognitive Behavioral Therapy (CBT) 1-3, are highly 
effective for anxiety and depression, the most common mental health (MH) disorders in primary 
care settings.4-6 The national VA Uniform Mental Health Services Handbook 7 clearly 
emphasizes the VA’s commitment to provide evidence-based treatments for Veterans with 
mental illness. However, broadly disseminating and implementing (a process often referred to 
as “scaling up”) these and other evidence-based practices has been a challenge in many health 
care systems 8, including the VA.9, 10 While training of clinicians is needed for scaling up, training 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 6 of 26 
 
alone is inadequate to assure that these EBPs are not only implemented in clinics but 
implemented with enough fidelity to assure their clinical effectiveness.11 Without adequate 
implementation and fidelity, patients receive few benefits and resources devoted to training 
clinicians and do not achieve their ultimate goal of improving patient outcomes.  
 
Implementing psychotherapies in small or remote primary care clinics, such as many VA 
community-based outpatient clinics (CBOCs), can be especially challenging. [A 2008 study 
showed that only 22% of Veterans with depression, PTSD, or anxiety received at least one 
session of psychotherapy, and that rural Veterans were even less likely than urban Veterans to 
receive any psychotherapy 12 and the quality of the psychotherapy received is unknown]. MH 
providers in these clinics, many of whom are located in rural and highly rural areas, are 
sometimes the only MH provider at the clinic and may be isolated from their peers and from 
educational resources. Some of these MH providers, especially those with social work or 
nursing backgrounds, have no previous training in evidence-based psychotherapies; yet, as 
“general practitioners” of MH treatment, they are expected to be able to respond to a wide range 
of MH problems using evidence-based techniques, even though some of these techniques 
require considerable time to fully master. Approaches or tools to support and assist CBOC-
based MH providers to deliver evidence-based psychotherapies for a range of mental disorders 
are needed with high fidelity.  
 
The NIMH-funded Coordinated Anxiety Learning and Management (CALM) study 13 faced a 
similar challenge in implementing CBT in 17 non-VA primary care clinics across the US with 
providers, usually nurses and social workers, most of whom had no previous training or 
experience in CBT. To meet this challenge, Michelle Craske, PhD, and her UCLA-based team 
of experts, developed a tool to support primary care-based MH providers to deliver CBT for 
patients with a range of disorders, including post-traumatic stress disorder (PTSD), panic 
disorder (PD), generalized anxiety disorder (GAD), and social anxiety disorder (SAD), and 
depression. The CALM CBT program is unique in that it can be used for any of these disorders. 
This is possible because the program includes some modules that are common across 
disorders but additional modules that are disorder-specific.14 Computer-assisted programs have 
been used in training in CBT, and in self-directed CBT 15-17, but prior to CALM had not been 
used for ongoing assistance of the MH provider in the delivery of CBT. Designed to be used as 
a tool by the provider, the CALM program involves the patient and provider looking at the 
computer screen together and proceeding through the modules at their own pace.18 The CALM 
CBT program was used by 14 providers in treating more than 500 patients in a variety of 
primary care clinics, was considered acceptable to providers and patients, and achieved a high 
degree of fidelity. Most importantly, the CALM intervention as a whole was clinically effective 
(relative to treatment as usual) 19 for all disorders (PTSD, panic, GAD, SAD with or without 
depression).14 Further, the improved clinical outcomes achieved in CALM appear to be due to 
receipt of CBT delivered with this tool.19 A tool providing fidelity support to MH providers who are 
social workers, nurses, and masters’ level psychologists, in primary care could be invaluable in 
VA CBOCs.  
 
Our research partners are very interested in adapting the CALM computer-assisted CBT tool for 
use in VA CBOCs. The tool promotes fidelity by having the provider and the patient use the tool 
together during the session and by generating homework assignments tailored to each patient. 
Although we believe that the tool contributed to the high fidelity to the CBT protocol in the CALM 
study, this hypothesis has not been tested. This project will focus on testing whether computer-
assisted CBT can improve technical quality of care within VA CBOCs. 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 7 of 26 
 
3.0 
Objectives 
 
Using a Type III 20 hybrid effectiveness-implementation study design, the objectives of this 
project are to modify the CALM Tool, computer-assisted CBT tool, and its impact on 
providers' fidelity to the CBT model, and to assess how best to support its future 
implementation in routine care. The study meets the criteria for a Hybrid Type III because the 
primary focus of the study is to test an implementation intervention at the provider-level. We will 
provide (1) face to face training in CBT (including three months of clinical supervision to assure 
clinical competence) coupled with (2) external facilitation to 34 CBOC MH providers; half will 
receive the computerized CBT tool and half will not. We expect the computerized tool to 
improve fidelity (quality) of CBT. A secondary focus of the project is to collect patient-level data 
on clinical impact. 
 
Previous VA training in CBT. The VA has sponsored at least two large-scale initiatives for 
clinician training in CBT. Both of these therapies are relatively long (12 – 16 sessions), have 
been intended for treatment in specialty mental health settings (not primary care settings) and 
each has targeted only one disorder. We are unable to confirm the extent to which CBOC 
mental health clinicians participated in these training efforts. However, the number of CBOCs 
has expanded considerably since 2000, going from about 600 to more than 800 by 2010, and 
there has also been a corresponding expansion in mental health providers located at CBOCs. 
We suspect that the majority of mental health providers in CBOC have not received training. 
 
The first VA training initiative, started in 2004, trained licensed mental health professionals in 
Cognitive Behavioral Therapy (CBT) for depression. The CBT protocol was designed to be 
delivered using 16 individual 50-75-minute weekly sessions over the course of 4-6 months of 
treatment. In this training program, providers could become certified in CBT for depression by 
attending a multi-day didactic workshop and participating in follow-up consultation. 21 Providers 
could also use support materials such as treatment manuals, web-based treatment resources, 
and phone consultation with treatment experts. We were unable to determine the total number 
of persons who received training in this program but we do know that use of follow-up 
supervision to assure clinical competence was disappointingly low.  
 
Drawing from lessons learned in this first effort, in 2006 VA began a second national initiative to 
disseminate Cognitive Processing Therapy (CPT) (a type of CBT specifically for PTSD) as an 
evidence-based treatment for PTSD in the VA. CPT is a 12-session treatment based on social 
cognitive theory that focuses on the thoughts and coping behaviors that individuals have in 
response to a traumatic event. This dissemination initiative has evolved over the six years and 
currently involves a 3-day face-to-face meeting, followed by 6 months of weekly remote 
supervision in a small group by an assigned training consultant and the submission of work 
samples to demonstrate successful completion of the training program. Mental health providers 
are not followed after they complete the 6 months supervision. As of March 31, 2010, nationally 
about 2,300 mental health staff have been trained in CPT (There are an estimated 2500 mental 
health providers in VISN 16 alone) and numerous hard-copy manuals have been distributed. 
 
While these training efforts have reached many VA clinicians, we do not know, as stated earlier, 
the extent to which CBOC mental health providers have participated in these trainings. All of 
these trainings have been focused on a single disorder, are relatively long (e.g., 12 – 16 
session), and were designed for intensive treatment in specialty (not primary care) treatment 
settings. (See preliminary studies section below on CBOC interviews). Our research partners at 
OMHS, OMHO, and the VISN16 MHPL are very concerned that fidelity to the CBT protocol 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 8 of 26 
 
following national training is considerably less than optimal. Given the time and preparation that 
is involved in the provision of evidence-based treatments such as CPT, it is often difficult for 
mental health providers, especially ones that are geographically isolated or practice in fast-
paced environments, to maintain fidelity to a hard-copy manualized protocol and to use a 
different protocol for each disorder. Further, research has shown that after receiving training 
clinicians typically do not remain true to a model but, instead, pick and choose the elements of 
the practice that they incorporate into their usual work 8 
 
Specific Aims: 
 
Specific Aim 1: To compare two different approaches to implement CBT in CBOCs  
 
Hypothesis 1a (primary): Clinicians randomized to receive CBT training and the computer-
assisted CBT tool will have higher fidelity than providers randomized to receive CBT training but 
not tool but receiving a hardcopy treatment manual.  
 
Hypothesis 1b (secondary): Patients of clinicians randomized to receive CBT training and the 
computer-assisted CBT tool will have clinical outcomes that are superior to patients of clinicians 
utilizing a manual only.  
 
Specific Aim 2: Prepare for future implementation in routine care by performing an 
implementation needs assessment with key stakeholders (participating CBOC MH providers 
and participating CBOC directors). 
4.0 
Resources and Personnel 
 
This 3.5-year study will be based at the Central Arkansas Veterans Healthcare System 
(CAVHS). The PI, Dr. Michael Cucciare, and study coordinators will have access to protected 
health information. Study coordinators will be involved in recruiting subjects and obtaining 
informed consent for Aims 1 and 2 of the study. For Aim 1, study coordinators will recruit and 
consent CBOC MH providers to participate in a three-day training to be held in Little Rock, AR, 
at CAVHS. They will also be responsible for screening, recruiting, and consenting Veteran 
participants receiving care at VA CBOCs to participate in the randomized trial. Study 
coordinators will be responsible for administering baseline and follow-up interviews to Veteran 
participants which will be conducted via using computer-assisted telephone interviewing (CATI).  
 
For Aim 2, the study coordinators at CAVHS will consent CBOC directors from the CBOCs 
participating in Aim 1 to a short telephone interview (CBOC MH providers will be consented for 
Aim 1 and Aim 2 simultaneously). Data analysis will be performed by the PI and Co-Is, and 
supported by the consultants and research staff as detailed below.  
 
Key Personnel – Little Rock, AR 
 
Michael Cucciare, Ph.D (Principal Investigator: 25% (3.0 calendar months) in all years). 
Dr. Cucciare is a clinical psychologist and Research Health Scientist at the VA HSR&D Center 
for Mental Healthcare and Outcomes Research (CeMHOR). He is also an Assistant Professor 
(tenure track) in the Division of Health Services Research at the University of Arkansas for 
Medical Sciences. Dr. Cucciare has extensive knowledge of cognitive behavioral therapy (CBT) 
and the use of computerized aids in support of CBT. His career development award, now 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 9 of 26 
 
completed, had a similar focus (evidence-based practices in medical settings). He is currently a 
Co-PI on a CREATE project out of Palo Alto that involves implementing and evaluating a 
computerized brief alcohol intervention for Veterans with hepatitis C and who consume alcohol. 
Dr. Cucciare will assume overall responsibility for the scientific integrity of the project.  He will 
lead weekly project meetings, manage the budget, and supervise the project coordinator as she 
oversees day-to-day activities and IRB submissions. He will also oversee recruitment and data 
collection for Aim 1 and Aim 2 and co-lead implementation needs assessment with Dr. Curran.    
 
Geoffrey Curran, PhD (Co-Investigator: 10% (1.2 calendar months) in all years).  Dr. 
Curran is a sociologist and Research Health Scientist at the VA HSR&D Center for Mental 
Healthcare and Outcomes Research (CeHMOR).  He is also Director of the Center for 
Implementation Research in the Colleges of Pharmacy and Medicine at the University of 
Arkansas for Medical Sciences. He is also a Research Health Scientist at the VA HSR&D 
Center for Mental Healthcare and Outcomes Research (CeMHOR). Dr. Curran developed an 
online CBT training tool using the ADDIE developmental methodology. He recently presented a 
model for hybrid clinical effectiveness-implementation research trials which has guided the 
design of this study.  Dr. Curran will work closely with Dr. Cucciare to help develop the 
computer-based CBT tool and co-lead implementation needs assessment.   
 
 
Study Staff – Little Rock, AR 
 
 
Kathy Marchant (Project Director: 100% (6 calendar months for 3.5 years). Ms. Marchant 
will be responsible for all IRB submission and correspondence; conduct provider and patient 
recruitment and consenting; be involved in planning the provider training; assist in the 
management of qualitative data; oversee the collection of outcome assessments; conduct 
outcomes assessments; and oversee IRB requirements such as the documentation of informed 
consent. 
 
Mitzi Mosier (Project Coordinator: 50% (6 calendar months for 3.5 years). Ms. Mosier will 
be responsible for supporting recruitment efforts for Aims 1 and 2, conducting informed consent 
of providers, Veterans, and CBOC directors.  
 
Rebecca Doan (Project Coordinator: 20% (2.4 calendar months for 3.5 years). Ms. Doan 
will provide IRB consultation to the study team; support the team in the development of consent 
forms, and continuing review documentation.  
 
Xiaotong Han, MS (Biostatistician, 5% -20% (.6 calendar months for year 1, 3 calendar 
months for years 2-4.) Ms. Han will be the study biostatistician responsible for data analysis to 
examine our primary and secondary outcomes. She will be also be responsible for the overall 
management of the database.  
 
Traci Abraham, PhD (Qualitative Interviewer). Dr. Abraham is an anthropologist and health 
services researcher with expertise in qualitative interviewing. Dr. Abraham will lead the 
qualitative interviewing efforts for Aim 2.   
 
Key Personnel – Houston, TX  
 
Michael Kauth, PhD (Co-Investigator, 10% (1.2 calendar months for 3.5 years). Dr. Kauth is 
a clinical psychologist, Co-Director and Associate Director for Education in the SC MIRECC, 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 10 of 26 
 
overseeing clinical training programs for mental health clinicians.  Increasingly, he has focused 
on strategies for dissemination of training and implementation of training – including the use of 
technology-enhanced tools for clinical training to reach a wider audience. Dr. Kauth has a strong 
background in implementation research, specifically external facilitation, to increase the 
adoption of evidence-based psychotherapy. He has collaborated with VISN 16 Mental Health 
Product Line to implement evidence-based PTSD psychotherapy in CBOCs, using telehealth 
technology. Dr. Kauth will assist with provider recruitment, provide guidance and leadership on 
the implementation strategy and supervise the efforts of the external facilitator. He will also 
provide consultation on all training activities and assist with the implementation toolkit and 
manuscripts. 
 
Jeffrey Cully, PhD (Co-Investigator 10% (1.2 calendar months for 3.5 years). Dr. Cully is 
a clinical psychologist and health services researcher with expertise in the study and 
implementation of mental health interventions, cognitive behavioral therapy, and the 
application of psychological interventions outside of traditional mental health specialty care 
settings (e.g. primary care).  Dr. Cully directs the SC MIRECC fellowship program in Houston.  
Dr. Cully will provide CBT training and on-going supervision to the 34 participating providers.  
He will assist with the implementation toolkit and manuscripts.     
 
Study Staff – Houston, TX  
 
Jan Lindsay, PhD (External Facilitator: 10% (1.2 calendar months) in year 1; 25% (3 
calendar months) in years 2 and 3. Dr. Lindsay will attend the CBT training and build rapport 
with the 34 participating providers and speak with them at least once a month for the duration of 
the intervention to problem solve barriers to implementing the CBT protocol and/or the 
computer-assisted CBT tool.  She will also maintain a detailed time-log of all facilitation 
activities. 
 
5.0 
Study Procedures 
 
5.1 Study Design 
 
Specific Aim 1: Randomized Trial  
 
MH provider participants; Mental Health Providers (MH providers) will be asked about their 
previous training and clinical experience with CBT. Because our own VA work and the CALM 
research both showed that previous training in CBT was moderately related to performance 
after CBT training, we will attempt to match providers on previous CBT experience. We will 
randomize 17 providers to receive the tool at the end of training. (Not receiving the 
computerized tool is in effect “training as usual”). To randomize we will first separate providers 
empirically into three groups, depending on their previous CBT experience (no, some, most 
experience). We will then create three pots containing assignment numbers (1 = receives tool; 2 
= no tool) for each of the three groups. We will draw a number for each provider from the 
appropriate pot. Pots will have equal amounts of assignment numbers in each (i.e. equal 
numbers of 1’s and 2’s).  Only the Mental Health Providers will be randomized. 
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 11 of 26 
 
Veteran patients may be existing or new patients to the provider. Patients will be considered 
eligible if they: (1) are a veteran, (2) their Mental Health provider is a participant in this study; (3) 
their Mental Health provider has made a clinical determination that CBT is an indicated 
treatment; (4) the patients plan to continue to receive mental health care at the CBOC, (5) have 
depression, PTSD, or other anxiety disorder (PD, GAD, SAD), (6) are willing to  receive CBT 
specifically, (7) and are willing and able to participate in clinical assessments (baseline, 3, and 6 
months) by phone. Patients will be considered ineligible if they (1) have evidence of significant 
cognitive impairment, are in crisis (e.g., suicidal), (2) are dependent on alcohol or drugs 
(substance abuse is allowed), (3) or have a comorbid diagnosis of schizophrenia or bipolar 
disorder.  We will expect the MH providers to recruit a total of 340 veterans for the first two 
years of the project. For each patient, providers will be asked to proceed through treatment 
modules in a step-wise fashion; however, we recognize that patients may not complete all 
modules. 
  
We are adding an additional option of CALM delivery for MHPs randomized to the Computer 
arm of the study.  This would include the option of Video-to-Home delivery.  This option would 
require the following Veteran pre-requisites:   
 
o Computer with reliable internet connection, with a video camera, microphone, 
and speakers.  
o Access to MyHealtheVet to send/receive handouts through secure messaging.  
o Active email address.  
o Phone availability.  
o Consent to telemental health and treatment of ____ (anxiety, depression, etc.) 
using CALM (standard of care) 
 
On the day of the appointment: 
o Veterans receive an email shortly before the scheduled start of the session. The 
email contains a hyperlink to navigate to the secure webpage/virtual medical 
room (VMR).  
o Veterans access VMR at the time of the appointment, and contact the MHP by 
phone if they experience technical difficulties.  
o If the provider and Veteran are unable to complete the session by video, we may 
use telephone if appropriate. 
 
Additional logistics: 
 
MHP logs into CALM online program (Veteran does not have program access) 
 
MHP shares screen with Veteran over video telehealth connection, walks Veteran 
through content 
 
Veteran and MHP able to either switch between viewing each other/the CALM program 
or view a thumbnail of the alternate screen (i.e., large view of person with thumbnail of 
program or large view of program with thumbnail of person) 
 
 
Training and supervision, all 34 MH providers will receive the same training in CBT 
led by Dr. Michelle Craske (CALM tool consultant) with assistance from our study team. 
The training will be of the same length and format (3-day, face to face) this structure is 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 12 of 26 
 
the same as current VA training programs. The training will differ across the two MH 
provider groups in that the final two hours of each day will involve either practice with 
the tool or practice with the manual.  All providers who complete the training (regardless 
of randomization) will receive supervision for 3 months. This supervision program will be 
based on the previous post-training supervision provided by VA for CBT 12, 22 and will 
consist of small group supervision with the first 4 weeks of clinical supervision to include 
roleplays with the supervisor giving feedback to each clinician, and all clinicians 
benefiting vicariously from listening to each other’s’ role plays and feedback.  The 
following 8 weeks will allow the providers to prepare and discuss actual veteran 
scenarios.  Site facilitation our experience has shown that coupling training with a site 
implementation strategy optimizes the benefits of the training. External facilitation increases 
providers’ success in actually bringing their new skills into their particular clinical setting.9, 23 All 
34 MH providers will receive external facilitation. The external facilitator, Dr. Lindsay, will help 
each provider develop an implementation plan, including setting individual goals for CBT 
implementation and negotiate a clinic implementation plan with clinic leadership. She will be 
present at the training sessions in order to meet the MH provider face to face. Thereafter, the 
facilitator will meet by phone with each provider at least once following the supervision and as 
needed during the implementation phase.  The calls between the facilitator and MH provider will 
typically focus on quickly generating solutions to barriers to implementation (e.g., scheduling, 
time management) and maintaining motivation to practice CBT in the face of multiple barriers. 
Michael Kauth, PhD, who has facilitated numerous educational interventions in VISN 16 (see 
preliminary studies) will supervise Dr. Lindsay for this project.  
 
Training and technical assistance for computerized tool The 17 MH providers who receive 
the computerized tool will receive ongoing technical support by phone for use of the 
computerized tool. A technical support specialist (to be named) will check weekly with each of 
the 17 MH providers using the computerized tool during the first month and will trouble-shoot 
any problems that emerge. Thereafter, the technical support specialist will check in monthly but 
will also be “on call’ to address any problems with implementation and use of the computerized 
tool. 
 
Risks and Benefits: There are minimal risks anticipated for MH providers and Veterans 
participating in Aim 1. It is possible that both participant groups will feel some emotional 
discomfort while being audiotaped during session. However, audiotaping therapy sessions is a 
well-studied method for examining treatment fidelity and provider activity during a treatment 
session. Any emotional discomfort typically dissipates once the session begins. Participants 
may also feel some concern about who will view and have access to audiotaped treatment 
sessions. We will make a point of emphasizing to both providers and patients that audiotaped 
sessions will be kept strictly confidential with only authorized research staff having access.  
 Audiotaped sessions will be stored electronically on a secure service drive in password 
protected folders behind the VA firewall.  All hard copy source documents will be kept in a 
locked filing cabinet at CAVHS, building 58, room 251.   
 
 
Specific Aim 2: Implementation Needs Assessment  
 
All MH providers who participated in the implementation trial will be invited to take part in a 
focus group at the end of the trial. In addition, we will recruit 8-10 CBOC directors from those 
sites who participated in the trial and invite them to take part in a brief telephone interview (up to 
15 minutes).  

Protocol CALM _ 2 
v3; March 01, 2018   
Page 13 of 26 
 
 
We will conduct focus groups with the MH providers to assess their perspectives about what 
worked and what did not in terms of the training they received, the use of the tool (or not) in 
providing CBT, the supervision they received, and the additional facilitation/support provided to 
them and their CBOCs to provide the CBT. In addition, we will ask them to comment on the 
likely barriers and facilitators to providing the CBT and the use of the tool in routine care. After 
the focus groups with MH providers are completed, we will conduct interviews with a sample of 
CBOC directors to assess their perspectives on implementation of the tool under routine care 
circumstances, and we will get their reaction to the barriers/facilitators raised by the MH 
providers. The focus groups will take place via telephone. We will attempt to schedule 3 focus 
groups of up to 10 MH providers each. Given the difficulty of scheduling provider interviews or 
focus groups during the workday, we will attempt to schedule the focus groups either 
before/after the usual clinic hours, or during the lunch hour. The CBOC Director interviews will 
be scheduled at their convenience. The Directors of all participating CBOCS will be invited to 
participate. All focus groups and interviews will be audiotaped to allow for analyses.  
 
Risks and Benefits: There are no anticipated risks associated with participation in Aim 2. It is 
possible that CBOC MH providers and/or directors will be uncomfortable providing their opinions 
about the CBT training and/or use of CALM (or not) at their respective clinic. Again, we will 
make every effort to address these concerns and make it a point to explain that feedback will be 
kept strictly confidential and will not be shared with anyone outside of the study environment.  
 
5.2 Recruitment Methods 
 
Aim 1: Provider Participants 
We have had extensive experience in training VA providers 9 through the South Central 
MIRECC Education program. We have learned that it is vital to invite only those who are truly 
motivated and eager to participate in trainings. We will let all CBOC MH providers in VISN 16 
know about this training and research opportunity in three ways. First, we will announce the 
opportunity on the monthly meeting of MH VISN-wide leaders. Second, we will publicize the 
opportunity on the SC MIRECC professional networking site designed specifically for CBOC 
clinicians, called the Mental Health PREP. Third, we will send out study information sheets to 
providers on the email list-serve of CBOC clinicians maintained by the VISN 16 Mental Health 
Product Line office. Fourth, we will coordinate via email and telephone conversation with CBOC 
directors to identify MH providers who might be interested in participating in this project, and 
approach them via email or telephone, depending on their preference.  We will also request that 
each mental health provider interested in participating obtain permission from their respective 
supervisors to receive training in CBT and participate.  This will include some indication (i.e., 
email or memo), from their supervisor to the provider, that they have permission to participate in 
the CBT training in Little Rock. This will ensure that participant providers have discussed the 
project with their supervisor and have received appropriate permission to leave their work 
station for the training.  
 
Training will be contingent on willingness to participate in the research project but it will be made 
very clear that participation is entirely voluntary. Providers who desire training but do not want to 
participate in the research project will be linked with the VA’s national training program in CBT 
through which they may obtain training. To be eligible, clinicians must (1) currently providing 
CBT; (2) be located at the CBOC (all disciplines will be allowed); (3) have an office at the CBOC 
with a computer, (4) be willing to be randomized to a group receiving the computerized tool at 
the completion of training vs. a group receiving a hardcopy manualized protocol only; and (5) 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 14 of 26 
 
must be willing to audiotape patients and participate in supervision. There are approximately 
180 CBOC-based MH providers in VISN 16. We will need to recruit a total of 34 MH 
providers (or 19% of 180) for the training. We are confident that we can recruit 34 based on our 
previous experience and interviews with VISN 16 CBOC MH providers. (See preliminary studies 
above). For any provider interested in participating, we will offer a phone consult to describe the 
study in detail. Those who are interested in participating will be consented over the phone.    
 
Aim 1: Veteran Participants 
 
A total of 340 patients will also be recruited for Aim 1. Patients may be existing or new 
patients to the provider. Patients will be considered eligible if they: (1) are a veteran, (2) their 
Mental Health provider is a participant in this study; (3) their Mental Health provider has made a 
clinical determination that CBT is an indicated treatment; (4) the patients plan to continue to 
receive mental health care at the CBOC, (5) have depression, PTSD, or other anxiety disorder 
(PD, GAD, SAD), (6) are willing to receive CBT specifically, and (4) are willing and able to 
participate in clinical assessments (baseline, 3, and 6 months) by phone. Patients will be 
considered ineligible if they (1) have significant cognitive impairment, are in crisis (e.g., 
suicidal), or have a comorbid diagnosis of schizophrenia or psychosis.  
 
Patients will be recruited one of two ways. First, MH CBOC providers participating in Aim 1 
will provide each patient with a study information sheet inviting them to participate. This 
information sheet will contain research team contact information.  Veterans who express an 
interest in participating and/or learning more about the study will be encouraged to contact the 
study team for more information or the veteran may provide contact information and a preferred 
day/time to their provider and the provider will give the information to the study team.  The 
mental health providers will not be recruiting participants but rather providing information about 
the study and study team contact information.  Only the research team will be involved in the 
recruitment of eligible Veterans.  Veterans, who give their permission, will be contacted by a 
study team member (qualified research personnel) who will answer any questions about the 
study, and if interested, conduct the informed consent via the telephone and enroll the veteran 
into the study.  Second, we will obtain a list of Veterans with diagnosis of anxiety or depression 
for each of the participating CBOC MH providers (through the use of VA administrative records).  
Veterans with diagnosis of depression, PTSD or anxiety disorders under the care of the 
Mental Health provider participant and for whom their Mental Health provider has made a 
clinical determination that CBT is an indicated treatment will be presented to the participating 
Mental Health Provider for review and confirmation that veteran meets eligibility criteria.  After 
the MHP participant has confirmed the veteran is under their care and a good candidate, the 
study team will send the veteran an opt-out letter inviting them to participate. Veterans 
interested in participating in the study will be asked to contact study staff and if the veterans do 
not opt-out; the research staff will contact the veterans within 10 days of mailing the opt-out 
letter. If veteran meets inclusion criteria and agrees to participate, informed consent will be 
obtained via telephone by trained study staff.     Both recruitment methods are needed to ensure 
we can meet recruitment goals. Veterans will be compensated $25 for the first assessment, $25 
for the 3-month assessment, and $30 for the 6-month assessment (for a total possible of $80).   
 
Aim 1: Screening Veterans for eligibility 
 
In Aim 1, we have requested a waiver of informed consent for recruitment purposes to allow 
research staff to review potential participants’ electronic medical record to determine eligibility 
prior to contacting participant and/or mailing opt-out letters. Screening will consist of reviewing 
(1) whether they currently have a diagnosis of depression, PTSD, or other anxiety disorder, (2) 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 15 of 26 
 
review to identify if they have received CBT in the past (3) and/or does their MHP recommend 
CBT.   
Patients will be considered ineligible if they (1) have evidence of significant cognitive 
impairment, are in crisis (e.g., suicidal plus current intent to harm oneself), (2) have previously 
completed a course of CBT or CPT treatment (patients who have previously had only one or two 
sessions of CBT or CPT will be allowed), or (3) have a comorbid diagnosis of schizophrenia or 
psychosis 
 
Aim 2: CBOC directors 
 
All MH providers who participated in the implementation trial will be invited to take part in a 
focus group at the end of the trial. In addition, we will recruit f 8-10 CBOC directors from those 
sites whose Mental Health providers participated in the trial and invite them to participate in a 
brief telephone interview (up to 15 minutes). CBOC directors will be identified via VA Mental 
Health Leadership, and invited to participate in Aim 2 via email and follow up phone call. We will 
also attend a CBOC director monthly call to discuss the project and invite directors to 
participate.   
 
5.3 Informed Consent Procedures 
 
Aim 1: provider and Veteran participants: 
Mental Health Providers who express interest in this project will be contacted by study staff and 
after confirming eligibility and obtaining verbal consent to discuss this project, the study 
coordinator will describe the study purpose and interventions with the potential participant.  
Informed consent will be obtained verbally via telephone.   We have requested a waiver of 
documentation of informed consent for this purpose.  Likewise, Veterans who express interest in 
this project and meet eligibility requirements will be contacted by the study coordinator and (or 
the veteran may contact the study staff).  Some potential participants will be referred to the 
Study Coordinator from their CBOC provider while some potential participants may be 
responding to an opt-out letter. We will access the veteran EMR via CPRS / CAPRI to screen 
for eligibility, the study staff will obtain verbal permission to discuss this project; the study 
coordinator (SC) will describe the study purpose and interventions with the potential participant.  
Again, we are requesting a waiver of documentation of informed consent as trained study staff 
will conduct the informed consent process via telephone.    
Informed consent will be obtained before data collection occurs for either participant group. 
Patients with evidence of psychosis, dementia, and/or terminal illness will be ineligible to 
participate.  
 
Aim 2: provider participants: 
CBOC directors who express interest in participating in the interview will be provided an 
additional informed consent information sheet and informed consent will be obtained via 
telephone with qualified study staff.   We are requesting a waiver of documentation of informed 
consent for the CBOC directors as we will not be conducting informed consent in person. 
 
5.4 Inclusion/Exclusion Criteria 
 
Patients will be considered eligible if they: (1) are a veteran, (2) their Mental Health provider is 
a participant in this study; (3) their Mental Health provider has made a clinical determination that 
CBT is an indicated treatment; (4) the patients plan to continue to receive mental health care at 
the CBOC or have a computer with a reliable internet connection, with a video camera, 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 16 of 26 
 
microphone and speakers, (5) have depression, PTSD, or other anxiety disorder (PD, GAD, 
SAD), (6) are willing to  receive CBT specifically, () are willing and able to participate in clinical 
assessments (baseline, 3, and 6 months) by phone. Patients will be considered ineligible if 
they (1) have significant cognitive impairment, are in crisis (e.g., suicidal), (2) are dependent on 
alcohol or drugs (substance abuse is allowed or (3) have a comorbid diagnosis of schizophrenia 
or psychosis. 
 
All MH CBOC providers within VISN 16 will be invited to participate. All directors of CBOC sites 
with a MH provider participant will be eligible to participate.  
5.5 Study Evaluations 
 
 Aim 1: Veteran participants will be asked to complete following questionnaires at each of three 
assessment points; Baseline, 3 month and 6 months: 
 
Table 1 – Veteran completed research assessments (see Appendix C) 
Instrument  
Construct  
 
Socio-Demographics  
18 items that measure socio-economic and military 
characteristics  
 
 
 
Readiness Ruler  
3 items that assess perceived readiness to seek 
treatment  
 
SF-12 Patient Questionnaire  
12 items addressing overall physical and mental 
health functioning  
 
BAM 
 
Brief Addiction Monitor (BAM)  
 
PHQ9  
9 item inventory that yields a continuous and 
dichotomous assessment of depression  
 
GAD7  
7 item inventory that yields a continuous and 
dichotomous assessment of generalized anxiety 
disorder  
 
APA_DSM5  
Severity Measure for Panic Disorder 
 
PCL-5  
17 item inventory that yields a continuous and 
dichotomous assessment of PTSD  
 
Perceived Access Inventory*  
Perceived access instrument developed by 
CREATE Project 1  
 
BSI 18  
18 items gather patient-reported data used to 
measure psychological distress and psychiatric 
disorders  
 
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 17 of 26 
 
 
The Veteran assessments will be conducted via telephone; however, for those Veterans who 
are hard to reach for the follow up assessments we will send the follow up questionnaires via 
US mail and ask them to return via US mail.  We will provide the Veterans postage paid 
envelopes with instruction to complete the survey and not include their name or any identifying 
information on the assessment form or the postage-paid envelope.   
Aim 1: Fidelity Assessments: Implementation fidelity of CBT is our primary outcome. 
Consistent with the recommendations of Sharpless and Barber (2009),24 a 
multidimensional approach to measuring fidelity is taken. We will use the fidelity rating 
scales employed in the CALM study 49 in which fidelity is rated on a 1 to 7 scale using 
multiple items spanning three domains: protocol adherence, protocol competence, and 
global competence. Although separate ratings will be completed for each of these three 
domains, they will ultimately be combined to create a single measure of fidelity. We will 
add a fourth domain (non-adherence) developed just for this study. We felt this 
additional measure would be useful in identifying incidents of serious non-adherence to 
the model. This measure will be analyzed separately as a possible covariate for patient 
outcomes (see Appendix D).  
 
Rater Selection, Training, and Procedures. Raters will be doctoral level CBT experts 
who either attend or review an audio recording of the training workshop for study 
therapists (to be named). They will also be provided with the treatment manual. To 
establish initial inter-rater reliability each of two raters will independently rate an 
audiotape of a session and share their item-by-item ratings to determine a consensus 
rating for each item when discrepancies are found. This procedure will be repeated for a 
minimum of two sessions and until a percentage agreement on items ≥80% for 
achieved. 
 
Ratings. Fidelity assessments will focus on therapy sessions delivered early and late in 
the project. Fidelity for the first patient (of 3) will be assessed and used to control for 
initial group differences in multivariate analyses. We expect each  patient to complete up 
to 8 sessions. Fidelity will be assessed only for patients who have attended at least four 
sessions. The first and last two patients receiving at least four treatment sessions from 
each provider will be selected to have all of their treatment sessions rated for fidelity. 
Fidelity for the last 2 (of 3) patients for each provider is our primary outcome measure. A 
single overall fidelity rating will be completed for each of these patients’ last 3 sessions. 
A minimum of 5% of the total sessions being analyzed will be randomly selected to be 
rated by both raters to characterize inter-rater reliability. Average absolute agreement 
will be established using 2-way, mixed model intraclass correlation coefficients (ICC) on 
individual items and summary (i.e., mean) scores. Inter-rater reliability for the non-
adherence items will be calculated separately. 
 
Aim 2: All MH providers who participated in the randomized implementation trial will be 
invited to take part in a focus group at the end of the trial. In addition, we will invite a total 
of 8-10 CBOC directors from those sites who participated in the trial to participate in a 
brief telephone interviewSee draft interview guide. 
 
5.6 Data Analysis 
 
Aim 1: Analysis Plan 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 18 of 26 
 
 
Analysis plan for hypothesis 1a. MH providers will be the unit of the intent-to-treat analysis for 
hypothesis 1a. The dependent variable will be the composite fidelity score from each provider’s 
last three patients. The composite score will be calculated as the mean of the three scores. A 
dummy variable representing treatment group assignment will be specified as the explanatory 
variable of interest. An alpha significance level of 0.05 will be used to reject/accept the null 
hypothesis. The hypotheses will be tested using a standard logistic regression model. Three 
covariates will be specified to control for differences across providers. To choose the covariates, 
we will first examine differences in the characteristics of providers across the intervention and 
control group and include those characteristics that differed with a p<0.2. If no group differences 
are observed in the characteristics of the providers, we will control for: 1) previous CBT 
experience and 2) length of time in clinical practice. 
 
Power Calculation: The power calculation takes into consideration analyses involving 
comparisons of means and proportions. In making these power calculations we assume that 2 
providers in each group (4 total) will either “drop out” of the project, change jobs, or leave VA 
employment, leaving us with a sample size of 30. With a sample size of n=30 providers, we will 
have an 84% power to detect a large effect size (Cohen’s d=1.0) in mean treatment fidelity 
based on all sessions from each provider’s last three patients. This calculation assumed a 1.2 
point difference in mean fidelity between the intervention and control groups, a pooled standard 
deviation of 1.2, alpha=0.5, and a two-tailed test of significance. This calculation assumed a 
conservative (i.e., lower) estimate of fidelity for the intervention group (5.2 of 7) compared to 
previous data 27, places the mean of the control group at the minimally adequate rating of 4, and 
assumes larger standard deviations in the fidelity of providers not using the computer-based 
fidelity assistant (s.d. = 2.0 versus 1.0). With a sample size of n=30 providers, we will also have 
88% power to detect a 35% difference (85% versus 50%) in the proportion of providers 
maintaining minimally acceptable treatment fidelity ( ≥4) between the intervention and control 
groups based on all sessions from each provider’s last three patients. This calculation assumed 
0% attrition rate, alpha=0.05, and a two-tailed test of significance. 
 
Analysis plan for hypothesis 2b. Veterans will be the unit of intent-to-treat analysis for 
hypothesis 2b. The primary outcomes will be the continuously measured BSI and SF12V scores 
at baseline, 3, and 6 months. These are generic measures that are appropriate for all 
participants.  
 
Primary outcome analyses. A dummy variable representing treatment group assignment will 
be specified as the explanatory variable of interest. An alpha significance level of 0.05 will be 
used to reject/accept the null hypothesis. We will use mixed models and included data from all 
completed research assessments.25 Casemix variables will be selected using the method of 
purposeful selection.26, 27 Casemix variables with missing values were imputed using the PROC 
MI procedure in SAS9.3. PROC GLIMMIX will be used with the PROC MIANALYZE procedure 
to model the continuous outcomes measures (i.e., BSI and SF12V MCS). Time will be included 
as a fixed effect with baseline specified as the reference group. The model specifications will 
include main effects for group and time (with the control group assigned as the reference 
group), and interaction effects for group by time. The group by time interaction effects will be 
used to test the hypotheses that the rate of improvement differed across the two groups. 
Because there will be multiple group by time interaction terms, an omnibus test will be used to 
determine whether these variables collectively explained a significant amount of the variance in 
the dependent variable.28 If the omnibus test is significant at the conservative alpha <0.10 level, 
we will report group differences and significance tests for each time period. Otherwise, we will 
report group differences averaged across the time periods and one significance test. The 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 19 of 26 
 
clustering of patients within providers may require that the regression error term be correlated 
across patients within the same cluster. Significant intra-class correlation violates the 
independence assumption of standard regression models and may cause underestimation of 
coefficient standard errors, possibly leading to incorrect inferences concerning the rejection of 
the null hypotheses. Therefore, the first step of the statistical analysis will be to test for lack of 
independence among observations within clusters using intra-class correlation coefficients at 
provider level. Specifically, using a likelihood ratio test, we will compare the -2log likelihoods for 
an unrestricted model to a model that restricts the intra-class correlation to be zero. 
Raudenbush recommends that unconditional models (i.e. without explanatory variables) should 
be estimated prior to considering conditional models (i.e., with explanatory variables).29 If it turns 
out that the -2log likelihoods are not significantly different, the mixed models will be specified to 
account for nesting at the provider level. The variance-covariance matrix will be specified to be 
unstructured. 
 
Power Calculation for primary outcome analysis: It will not be necessary for patients to 
complete a full course (8 sessions) of CBT in order to be included in the study so we expect 
minimal “drop out” of patients. The power calculation must take into account the clustering of 
patients with providers. Prior psychotherapy RCTs have found intraclass correlation coefficients 
(ICC) to vary widely (range 0 to 0.73) with a mean of about 0.08.30 We based our power analysis 
on a slightly conservative ICC of 0.10. With a sample size of n=300, we will have 85% power to 
detect a medium effect size (Cohen's d=0.5) for our primary outcome (BSI).31 This calculation 
assumed a 5-point difference in MCS scores between the two groups, a pooled standard 
deviation of 10, 15% attrition rate, ICC=0.10, alpha=0.05, and a two-tailed test of significance.  
 
Aim 2: Data Analysis plan 
 
Data Analysis: The focus groups will be audio-taped, transcribed, and co-analyzed by Drs. 
Curran, Cucciare and Abraham using "traditional content analysis" techniques which are largely 
inductive.32Drs. Curran, Cucciare and Abraham will code the qualitative data together. Coding is 
analogous to developing a book index, with coders linking blocks of text to a code or group of 
codes.33, 34 In general, we will note patterns that seem salient due to their recurring nature (both 
within and across interviews) or the extent of responses devoted to them. We will divide the 
data-coding process into 4 steps:  
 
1. Data Management: We will enter verbatim interview transcripts into a qualitative data analysis 
software package (MAXQDA) that enables researchers to mark blocks of text with thematic 
codes and explore relationships among codes and between codes/participant groups.  
 
2. Open Coding: Drs. Curran, Cucciare and Abraham will review the interview guides and 
transcripts line-by-line and begin to identify key emerging themes. This open-coding approach 
allows us to discover themes that appear regularly, leading to development of initial “top-level” 
codes.35 
 
3. Top-Level Coding: After coming to consensus on the top-level codes (key 
themes/categories); the investigators will then apply these codes to the focus group and 
interview transcripts (2-3 transcripts each depending on the number of focus groups completed). 
To ensure coding reliability and consistency, Drs. Curran and Abraham will review one another’s 
work and discuss and resolve differences.  
 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 20 of 26 
 
4. Subcoding: After completing top-level coding, Drs. Curran, Cucciare and Abraham will 
subcode those top-level codes that are “grounded” (i.e., associated with repeated 
quotations/discussions), using the same methods described above for top-level coding. 
Subcoding entails further refinement of the broad constructs represented by the top-level codes 
into subcategories.  
 
5.7 Withdrawal of Subjects 
 
Although we do not foresee any circumstances under which subjects with be withdrawn from the 
research without their consent, there is a possibility that participant may be deemed cognitively 
impaired, at which point we will withdraw them from the study. Withdrawing from the study at 
any point in time will not impact care or treatment at the VA.  
6.0 
Reporting 
 
The PI or designee will be responsible for evaluating any adverse events or serious 
adverse events and submitting the appropriate reports to the CIRB within 5 working days 
after being made aware of the occurrence. The PI will evaluate whether a participant in the 
study should be discontinued from further participation in the study, for safety reasons.  The PI 
and study staff will respond immediately to all directives from the CIRB or VA concerning the 
protocol and the continuation of the study.   
We will consider unanticipated problems or events that place participants at a greater risk of 
harm or discomfort than was previously known or recognized as adverse events. Stable, chronic 
conditions that are present prior to the study and do not worsen will not be considered. Given 
the nature of the study, the following adverse events are possible: discomfort or distress during 
primary data collection and breaches of confidentiality for data collected from participants and 
from databases. Should such adverse events be observed by or reported to members of the 
study team, research staff will take appropriate action, reporting the events to Dr. Cucciare, who 
will then report to the VA CIRB and CAVHS R&D Committee according to VHA policies.  
If a participant experiences any distress during data collection, study staff will inform the 
participant that they can take their time, ignore questions they do not wish to answer, and that 
data collection can be stopped at any time. Breaches of confidentiality are not anticipated given 
the commitment and experience of the PI and study staff/investigators; however, if this occurs 
the PI will follow VHA, federal and institutional requirements for reporting.  All reports will be 
submitted to the VACIRB.  
Primary responsibility for regulatory compliance and monitoring adherence to the protocol 
resides with the Principal Investigator, Dr. Michael Cucciare.  He will ensure that all members of 
the research team involved in human subjects research have successfully completed mandatory 
VA human subjects protection, good clinical practice and privacy policy training and understand 
their responsibility for reporting 1) adverse events or unanticipated problems to the PI 
immediately and 2) any incident involving the potential for lost, stolen or compromised research 
data to the PI and to CAVHS’s Information Security Officer/Privacy Officer within one hour after 
the incident has been identified. 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 21 of 26 
 
Data and Safety Monitoring Plan 
The PI will be responsible for assuring that all elements of the DSMP are followed.  He will also 
be responsible for notifying HSR&D should the Central IRB suspend the study for any reason or 
length of time.  During regularly scheduled weekly meetings, project personnel will discuss 
study progress, including recruitment; conduct of interviews; protocol adherence problems, if 
any; and any adverse events that may occur. 
 
Safety of participants:  Veterans who receive the baseline, 3-month and 6-month assessments 
will complete a 20-40-minute individual telephone interview.  For those hard to reach Veterans, 
we will send the Veteran Assessments via US mail using only their Study ID on the actual 
assessment forms.    For Veteran participants, the intervention activities pose little, if any, risk to 
participants, as described above.  All interviews will be conducted via telephone. If participants 
feel tired during the interview, they can take a break or discontinue study participation at any 
time.  We will take every precaution to ensure Veteran participant confidentiality and especially, 
to avoid disclosing Veterans’ identity as individuals who have used VA mental health services.   
 
Accuracy and integrity of the data: 
MHP/CBOC participant qualitative interviews:  All interviews will be audio-recorded.  Hard 
copies of documents will be stored in a locked file cabinet in a locked location at CAVHS 
HSR&D offices.  Digital recordings will be uploaded to a restricted-access folder on the 
password-protected CAVHS HSR&D server.  Before erasing the interview from the digital 
recorder, Dr. Abraham will confirm the completeness and integrity of the saved audio-recording. 
The audio-recordings for each interview will be transcribed verbatim.  Dr. Abraham will 
systematically review the transcript for completeness and accuracy.  She will correct any errors 
or omissions in the transcript before it is entered into the qualitative data management system. 
 
Veteran Quantitative data collection:  Data will be collected using a Microsoft ACCESS 
database.  Direct data entry via Microsoft Access and inclusion of “not applicable/doesn’t know” 
and “refused to answer” response categories will minimize the likelihood of unintentional 
missing data.  Site RAs will collect interviewer-administered questionnaire data using these 
programs on a desktop computer connected to the VA network. 
 
The Data Safety and Monitoring Plan will capture Serious Adverse Events (SAEs), i.e., death, 
hospitalization.  It will also capture Adverse Events (AEs), i.e., suicidal or homicidal ideation, 
expression of severe distress, other serious psychiatric or medical symptoms.  The Data Safety 
and Monitoring Plan will utilize the following categories in determining the severity of an event: 
 
 A. Classification of Event Severity: Events will be labeled according to severity which is based 
on their impact on the patient.  An Event will be termed 'mild' if it does not have a major impact 
on the patient, 'moderate' if it causes the patient some minor inconvenience and 'severe' if it 
causes a substantial disruption to the patient's well-being. 
 
B. Event Attribution Scale: Events will be categorized according to the likelihood that they are 
related to the study intervention. Specifically, they will be labeled either probably, 
possibly, or unrelated to the study intervention. 
 
C. Expected Risks: This study includes no invasive procedures or medications. All changes to 
patients' treatment will be known to their MH care team.  

Protocol CALM _ 2 
v3; March 01, 2018   
Page 22 of 26 
 
 
(1) Breach of confidentiality associated with reporting suicidal or homicidal intent to appropriate 
authorities or health personnel 
 
(2) Severe distress as indicated by: (a) an increase of more than one standard deviation from 
their mean score on the PHQ-9 during follow-up assessments, (b) patient verbal report of a 
serious psychiatric distress at follow-up, (c) patients reporting significant distress due to the 
content of the self-report measures, or (d) patients reporting significant distress due to the 
content of CBT.  In each case, the research personnel will consult with the PI to determine 
whether the participant's distress is significant enough to warrant an AE. 
 
(3) Suicidal ideation will be indicated by: (a) self-report of significant suicidal ideation on the 
PHQ-9, or (b) verbal report of thoughts of suicide.  Even for those who report suicidal ideation 
on the PHQ-9, determination of suicidality will be based on a combination of the self-report 
information and the interaction between the research personnel and the participant.  The 
research personnel will be trained to identify and address suicidal ideation in patients.  If the 
patient articulates thoughts of death or suicidal ideation, the study team member will ask the 
patient to elaborate on recent suicidal thoughts/behavior (e.g., does the patient have a suicide 
plan).  The situation will be based on the patient's responses and whether the study team 
member perceives that the patient is in immediate danger (e.g., an active plan 
verbalized).  Study personnel will follow established safety protocols and contact local licensed 
clinical staff when appropriate. Also, the study PI Dr. Cucciare is well-suited to manage clinical 
situations involving the expression of intent to harm oneself as he is a licensed clinical 
psychologist in the State of Arkansas.   
 
(4) Homicidal ideation will be indicated by verbal report of thoughts of hurting others in general 
or harming a specific individual.  The study does not directly ask about homicidal ideation but 
this information may be provided by a participant during an interaction with study team 
members.  This protocol will be similar to that for suicidal ideation.  Specifically, the research 
personnel will be trained to identify and address homicidal ideation in patients.  If the patient 
articulates thoughts of hurting others, the study team member will ask the patient to elaborate 
on his/her thoughts/behaviors (e.g., does the patient have a plan or specific target?).  The 
situation will be based on the patient's responses and whether the study team member 
perceives that the patient or someone else is in immediate danger.  Study personnel will follow 
established safety protocols and contact local licensed clinical staff when appropriate. Also, the 
study PI Dr. Cucciare is well-suited to manage clinical situations involving the expression of 
intent to harm another person as he is a licensed clinical psychologist in the State of Arkansas.  
 
(5) Given the study population, it is possible that participants will report other serious psychiatric 
or medical symptoms.  Research staff will be trained to monitor significant abnormal behavior 
and/or report that the patient perceives him/herself to be in imminent need of medical 
treatment.  If it is determined that the patient requires immediate care, the research staff will 
coordinate access to appropriate services.   

Protocol CALM _ 2 
v3; March 01, 2018   
Page 23 of 26 
 
Triggers to action will be the participant's expression of severe distress, suicidal or homicidal 
ideation, and/or other serious psychiatric or medical symptoms.  Research staff will be trained 
and prepared for situations in which a participant expresses severe distress. Specifically, all 
project staff will have been trained on VA's suicide risk assessment and response guidelines, 
including the capability of directly connecting suicidal participants to the 24-hour VA suicide 
hotline, as well as arranging with local psychiatry crisis management staff to assess and 
potentially hospitalize the patient.  Subsequent to any participant's expression of severe 
distress, ideation, or other serious symptoms, project staff will make all reasonable attempts to 
re-contact the participant to monitor his or her well-being until the acute situation is resolved.   
 
Research staff will also be trained and prepared for situations in which a participant not currently 
in addiction treatment reports severe alcohol misuse, e.g., A blackout, delirium tremens, 
alcohol-related injury).  Such individuals will be protected by having research staff provide a 
referral to substance use disorder treatment at the VA closest to where they are residing.  If the 
VA is rejected by the participant for any reason, project staff will utilize SAMHSA's (the federal 
agency within HHS that funds most non-VA substance use disorder treatment) online substance 
abuse treatment facility locator to provide a referral. 
7.0 
Privacy and Confidentiality 
 
We will use Protected Health Information (PHI) for this study.   PHI will not be disclosed to 
anyone outside of the PI and study Co-I’s and research team members involved with data 
collection and management.   
Digital Voice recordings, Mental Health Providers will audiotape their sessions with veteran 
participants with a digital recorder and upload these recordings directly to a secure folder on the 
HSR&D drive located in Little Rock.  This drive is behind the VA firewall.  Upon verification of a 
successful upload, the recording will be erased from the recorder.  We will use only government 
issued, ISO approved devices that are FIPS compliant per VHA policy.   
All PHI and final data collected for this study will be managed and housed at the Center for 
Mental Healthcare and Outcomes Research (CeMHOR), located at CAVHS in North Little Rock. 
CeMHOR has a centralized server and mass data storage device that is fire-walled, encrypted, 
and backed-up daily; two information technology specialists to ensure data security and 
integrity; networked personal computers for all personnel; and statistical, publishing and 
database software for data management and analysis. Relevant to data storage and safety, 
access to local Windows Active Directory Domain Controllers, a UNIX NIS Server and Network 
Appliance storage facilities are available. These resources allow large amounts of research data 
to be stored and analyzed within a secure environment, while still connected to VA computing 
resources. Network infrastructure and connectivity are maintained by the Shared Computer and 
Resources Facility (SCARF) that serves CeMHOR.  
All source documents and hard copy records will be kept in a locked cabinet in a locked room in 
building 58. Only authorized research staff will have the key to the locked cabinet. All electronic 
records will be kept on maintained, encrypted VA servers housed in building 58. All files and 
folders will be password protected. Research Investigator files will be destroyed six years after 
the end of the fiscal year when the research project has been completed per Records Schedule 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 24 of 26 
 
DAA-0015-2015-004. The ISO/PO will be notified within one hour if data loss or misuse is 
discovered. 
 
Ethics: 
The protocol and all related documents shall be approved by the IRB and R&D Committee prior 
to enrollment of any subjects. The informed consent process shall be conducted only by 
appropriately qualified study personnel. Subject privacy and confidentiality shall be respected by 
all study personnel.   
 
8.0 
Communication Plan 
 
The research team will regularly meet to discuss the study’s progress and address any 
adverse events or unanticipated issues. The study coordinators will be in continuous 
communication as they coordinate screening, data collection and management.   The Study 
coordinators will be in direct communication with the PI, Dr. Michael Cucciare, to remain up to 
date on the study protocol and to ensure that any serious adverse events or unanticipated 
problems are reviewed and reported appropriately.  
9.0 
References 
 
 (1)  Cape J, Whittington C, Buszewicz M, Wallace P, Underwood L. Brief Psychological 
Therapies for Anxiety and Depression in Primary Care:  Meta-Analysis and Meta-
Regression. BMC Medicine 2010;8:38. 
 (2)  Høifødt RS, Strøm C, Kolstrup N, Eisemann M, Waterloo K. Effectiveness of Cognitive 
Behavioural Therapy in Primary Health Care: A Review. Fam Pract 2011 
October;28(5):489-504. 
 (3)  Kazdin AE, Bass D, Siegel T, Thomas C. Cognitive-behavioral therapy and relationship 
therapy in the treatment of children referred for antisocial behavior. J Consult Clin Psychol 
1989;57(4):522-35. 
 (4)  Leon AC, Olfson M, Broadhead WE et al. Prevalence of mental disorders in primary care: 
Implications for screening. Arch Fam Med 1995 October;4(10):857-61. 
 (5)  Ansseau M, Dierick M, Buntinkx F et al. High Prevalence of Mental Disorders in Primary 
Care. J Affect Disord 2004 January;78(1):49-55. 
 (6)  Kroenke K, Spitzer RL, Williams JBW, Monahan PO, Löwe B. Anxiety Disorders in Primary 
Care:  Prevalence, Impairment, Comorbidity, and Detection. Ann Intern Med 2007 March 
6;146:317-25. 
 (7)  Department of Veterans Affairs: Veterans Health Administration. VHA Handbook 1160.01:  
Uniform Mental Health Services in VA Medical Centers and Clinics. Washington, D.C.: 
2008. 
 (8)  Hepner KA, Azocar F, Greenwood GL, Miranda J, Burnam MA. Development of a clinician 
report measure to assess psychotherapy for depression in usual care settings. 
Adminstration and Policy in Mental Health 2010 May;37(3):221-9. 
 (9)  Sullivan G, Blevins D, Kauth MR. Translating Clinical Training into Practice in Complex 
Mental Health Systems: Toward Opening the 'Black Box' of Implementation. 
Implementation Science 2008 June 3;3:33. 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 25 of 26 
 
(10)  Kauth MR, Sullivan G, Cully J, Blevins D. Facilitating practice changes in mental health 
clinics:  A guide for implementation development in health care systems. Psychological 
Services 2011 February;8(1):36-47. 
(11)  Stetler CB, Legro MW, Wallace CM et al. The role of formative evaluation in 
implementation research and the QUERI experience. J Gen Intern Med 2006 
February;21(Suppl 2):S1-S8. 
(12)  Cully JA, Jameson JP, Phillips LL, Kunik ME, Fortney JC. Use of Psychotherapy by Rural 
and Urban Veterans. The Journal of Rural Health 2010 June;26(3):225-33. 
(13)  Sullivan G, Craske MG, Sherbourne C et al. Design of the Coordinated Anxiety Learning 
and Management (CALM) Study: Innovations in Collaborative Care for Anxiety Disorders. 
Gen Hosp Psychiatry 2007 September;29(5):379-87. 
(14)  Craske MG, Stein MB, Sullivan G et al. Disorder-Specific Impact of Coordinated Anxiety 
Learning and Management Treatment for Anxiety Disorders in Primary Care. Arch Gen 
Psychiatry 2011 April 1;68(4):378-88. 
(15)  Carroll KM, Rounsaville BJ. A Vision of the Next Generation of Behavioral Therapies 
Research in the Addictions. Addiction 2007 June;102(6):850-69. 
(16)  Gega L, Norman IJ, Marks IM. Computer-Aided vs. Tutor-Delivered Teaching of Exposure 
Therapy for Phobia/Panic: Randomized Controlled Trial with Pre-Registration Nursing 
Students. Int J Nurs Stud 2007 March;44(3):397-405. 
(17)  Spek V, Cuijpers P, Nyklicek I, Riper H, Keyzer J, Pop V. Internet-Based Cognitive 
Behaviour Therapy for Symptoms of Depression and Anxiety: A Meta-Analysis. Psychol 
Med 2007 March;37(3):319-28. 
(18)  Craske MG, Rose RD, Lang A et al. Computer-assisted delivery of cognitive behavioral 
therapy for anxiety disorders in primary-care settings. Depression & Anxiety 
2009;26(3):235-42. 
(19)  Roy-Byrne P, Craske MG, Sullivan G et al. Delivery of Evidence-Based Treatment for 
Multiple Anxiety Disorders in Primary Care: A Randomized Controlled Trial. J Am Med 
Assoc 2010 May 19;303(19):1921-8. 
(20)  Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid 
designs: Combining elements of clinical effectiveness and implementation research to 
enhance public health impact. Med Care 2012 March;50(3):217-26. 
(21)  McHugh RK, Barlow DH. The Dissemination and Implementation of Evidence-Based 
Psychological           Treatments. A Review of Current Efforts. The American Psychologist 
2010 February;65(2):73-84. 
(22)  Kauth MR, Sullivan G, Blevins D et al. Employing external facilitation to implement 
cognitive behavioral therapy in VA clinics: a pilot study. Implementation Science 2010 
October 13;5:75. 
(23)  Karlin BE, Ruzek JI, Chard KM et al. Dissemination of evidence-based psychological 
treatments for posttraumatic stress disorder in the Veterans Health Administration. J 
Trauma Stress 2010 December;23(6):663-73. 
 
(24)  Sharpless BA, Barber JP. A conceptual and empirical review of the meaning, 
measurement, development, and teaching of intervention competence in clinical 
psychology. Clin Psyhcol Rev 2009 February;29(1):47-56. 
(25)  Little RC, Milliken GA, Stroup WW, Wolfinger RD. SAS System for Mixed Models. Cary, 
NC: SAS Institute Inc.; 1996. 
(26)  Bursac Z, Gauss CH, Williams DK, Hosmer DW. Purposeful selection of variables in 
logistic regression. Source Code for Biology and Medicine 2008;3(17). 
(27)  Bursac Z, Gauss H, Williams DK, Hosmer D. A purposeful selection of variables macro for 
logistic regression. Proceedings at the SAS Global Forum 2007 Conference. Cary, NC 
2007. 

Protocol CALM _ 2 
v3; March 01, 2018   
Page 26 of 26 
 
(28)  Hogan JW, Laird NM. Intention-to-treat analyses for incomplete repeated measures data. 
Biometrics 1996;52:1002-17. 
(29)  Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis 
Methods. Second Edition ed. London: Sage Publications; 2002. 
(30)  Crits-Christoph P, Baranackie K, Kurcias J et al. Meta-Analysis of Therapist Effects in 
Psychotherapy Outcome Studies. Psychotherapy Research 1991;1(2):81-91. 
(31)  Donner A, Brikett N, Buck C. Randomization by cluster: Sample size requirements and 
analysis. Am J Epidemiol 1981;114(6):906-14. 
(32)  Hsieh HF, Shannon SE. Three Approaches to Qualitative Content Analysis. Qualitative 
Health Research 2005 November;15(9):1277-88. 
(33)  Patton MQ. Qualitative Research and Evaluation Methods. 3rd ed. Thousand Oaks, CA: 
Sage Publications; 2002. 
(34)  Parker LE, Kirchner JE, Bonner LM et al. Creating a quality-improvement dialogue: 
Utilizing knowledge from frontline staff, managers, and experts to foster health care quality 
improvement. Qualitative Health Research 2009 February;19(2):229-42. 
(35)  Strauss A, Corbin J. Grounded Theory Methodology: An Overview. In: Denzin NK, Lincoln 
YS, editors. Strategies of Qualitative Inquiry.Thousand Oaks, CA: Sage Publications, Ltd; 
1988.