Adapting and Implementing the Blended Collaborative Care Model in CBOCs
Conditions: Depressive Disorders, Alcohol Use Disorders, Post-traumatic Stress Disorder (PTSD)
Sponsor: VA Office of Research and Development

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
1 
Study Protocol 
Adapting and Implementing the Blended Collaborative Care Model in CBOCs 
 
PI:  Richard R. Owen, MD 
Co-PI: JoAnn E. Kirchner, MD 
Central Arkansas Veterans Healthcare System 
1. Protocol title:  Adapting and Implementing the Blended Collaborative Care Model in CBOCs   
 
2. IRB Protocol # ; Version 9:  Date:  4/17/2018 
 
3. Name/address of the sponsor:  VA Health Services Research & Development Service 
 
4. Name/title of the Principal Investigator and Co-Investigators: 
Richard R. Owen, MD, PI 
JoAnn E. Kirchner, MD, Co PI 
Karen Drummond, PhD 
Tisha Deen, PhD 
Eva Woodward, PhD 
Scott Meit, PhD 
 
 
5. Address/phone number of site: 
Central Arkansas Veterans Healthcare System, North Little Rock Campus 
2200 Fort Roots Dr., North Little Rock, AR  72114 
(501) 257-1000 
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
2 
TABLE OF CONTENTS 
 
BACKGROUND ………………………………………………………………………… page 3 
Known and potential risks and benefits ……………………………………………… page 6 
Population to be studied ………………………………………………………………. 
page 6 
Statement of compliance ……………………………………………………………… 
page 7 
Purpose and specific aims ……………………………………………………….…… 
page 7 
Study design …………………………………………………………………………… 
page 9 
Procedures 
Specific Aim 1 ………………………………………………………………… 
page 10 
Specific Aim 2 ………………………………………………………………… 
page 12 
Interview Assessment Data…………………………………………………………… 
page 17 
Measures taken to minimize/avoid bias ……………………………………………. 
page 18 
Expected Duration ……………………………………………………………………. 
page 21 
Stopping or discontinuation criteria for individual participants …………………… 
page 21 
Selection and Withdrawal of participants ………………………………………….. 
page 21 
Assessment of safety ………………………………………………………………… 
page 22 
Statistical Analysis ……………………………………………………………………… page 22 
Direct access to source data/documents …………………………………………... 
page 24 
Data Safety and Monitoring Plan …………………………………………………… 
page 24 
Ethics …………………………………………………………………………………… 
page 25 
Data handling and recording ………………………………………………………… 
page 25 
References……………………………………………………………………………… 
page 26 
 
 
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
3 
Background: 
Integrated Care - Our analyses indicate that 40% of all VHA enrollees were diagnosed with a 
MH disorder in FY11. The integration of MH treatments into PC settings is an evidence-based 
approach to managing common depression and anxiety disorders.1,2 True integrated care is 
based on the biopsychosocial model of illness in which both physical and mental illnesses are 
seen as the result of biological, psychological and social influences. With VHA's adoption of 
Patient Aligned Care Teams (PACT), the PC setting represents a health home, where all 
aspects of health care are delivered in a continuous, coordinated and patient-centered manner. 
Integration increases: 1) temporal access to MH care by making it available during PC 
encounters, 2) geographic access to MH care by making it available on-site in the PC clinic, and 
3) cultural access to MH care by making it available in the less stigmatizing PC setting. In VA 
clinics where PC MH integration (PC-MHI) was deployed in FY07 and FY08, the detection, 
diagnosis and treatment of depression, anxiety, post-traumatic stress and substance use 
disorders increased significantly more than in clinics were PC-MHI was not deployed.3 Because 
of the greater geographic and cultural barriers to MH treatment in rural areas, the benefits of 
PC-MHI are even greater for the 41%4 of VHA enrollees who live in rural or highly rural areas. In 
fact, Veterans treated in CBOCs have significantly fewer MH encounters than Veterans treated 
at VA Medical Centers.5 Veterans treated in the smaller contract CBOCs have even fewer MH 
encounters than Veterans treated in VA staffed (i.e., owned and operated) CBOCs.6 Lack of 
engagement in MH care among rural Veterans may contribute to the higher MH severity levels7-
9 and suicide rates10 observed among rural compared to urban VHA enrollees. Therefore, PC-
MHI is critical to the well being of rural Veterans.  
Integrating MH care into PC settings is a major priority of the OMHS.11 The Uniform Mental 
Health Services Handbook mandates the blending of the two predominant, evidence-based 
models of integrated care (the Care Management model and the Co-Located model) at VAMCs, 
very large CBOCs, (>10,000 uniques) and large CBOCs (5,000-10,000 uniques).12 Because 
there is no scientific evidence to support its implementation, the Blended model is not mandated 
at medium CBOCs (1,500-5,000 uniques) or small CBOCs (<1,500 uniques). Based on 
numerous experiences facilitating the deployment of integrated care models in CBOCs, our PC-
MHI and VISN 16 Mental Health Product Line (MHPL) partners report that implementation of 
integrated care models is extremely challenging for CBOCs that lack a full range of MH staff 
(which also include many large CBOCs as well). These challenges are highlighted by findings of 
several preliminary studies described in the original grant application.  
Briefly, we found that, while CBOCs are very important to the VHA healthcare system (half of 
primary care users receive care in CBOCs), the implementation of PC-MHI is lagging in these 
settings. While 6.2% of the PC encounters at VAMCs were PC-MHI encounters, only 2.9% of 
the PC encounters at CBOCs were PC-MHI encounters. Small and medium sized CBOCs had 
an even lower proportion of PC-MHI encounters. We have also shown that the co-located 
referral model, in which traditional "silo" MH services are physically co-located in a PC clinic, 
produces clinical outcomes that are inferior to outcomes of patients receiving enhanced referral 
to off-site mental health specialty care. Further, Dr. Fortney (Co-I) and colleagues found that 
usual MH care (i.e., Co-Located Referral Model) results in sub-optimal outcomes for rural 
Veterans treated in small and medium CBOCs. However, a Care Management Model, 
developed by depression researchers based on the chronic care model,13,14 and specifically 
tailored by Drs. Fortney and Pyne for CBOCs that lack on-site psychiatrists and PhD 
psychologists, found significant improvements in adherence to medication, treatment response, 
and remission of depression for patients assigned to the care management condition as 
compared with those in usual care.15,16  
Preliminary Studies: Drs. Fortney and Pyne, in collaboration with the Medical Directors of 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
4 
VISN 16 (Dr. Enderle) and VISN 22 (Dr. McDougall), conducted a HSR&D QUERI-funded study 
(IMV 04-360 ) to test the Evidence Based Quality Improvement (EBQI) implementation strategy, 
similar to the one proposed in this application. This non-randomized implementation trial of 
EBQI was conducted at 11 CBOCs (associated with 3 VAMCs) lacking psychiatrists and PhD 
psychologists. The Plan phase of the EBQI process involved the localized tailoring and 
adaptation of the Telemedicine Based Care Management model to each VAMC and their 
affiliated CBOCs. Researchers ensured that the adaptations were evidence based. Clinical and 
administrative staff were responsible for adapting the care management model for local needs, 
priorities, preferences and resources. Planning was based on the Steps and Decisions Guide 
for Implementing Depression Care Management Models which was developed specifically for 
the study.17 Depression care managers were trained using the VA Mental Health QUERI 
Depression Care Manager Training Manual which was specifically developed for the study. In 
addition, a web-based decision support system (NetDSS - Net Decision Support System), and a 
NetDSS User’s Guide were specifically developed for the study to promote care manager 
fidelity.18 Do-Study-Act cycles were used to refine the program over time. Outcomes were based 
on the RE-AIM Framework (described later in the Research Design and Methods section) and 
used data from multiple sources: administrative records, web-based decision support system, 
surveys and key-informant interviews.19 Adoption: 69.0% (58/84) of PCPs referred patients to 
Telemedicine Based Care Management. Reach: 9.0% (298/3,296) of PC patients diagnosed 
with depression who were not already receiving specialty MH care were enrolled in the program. 
Fidelity: During baseline care manager encounters, education/activation was provided to 100% 
(298/298) of patients, barriers were assessed and addressed for 100% (298/298) of patients 
and depression severity was monitored for 100% (298/298) of patients. During follow-up, 
symptoms were monitored using the PHQ9 at 100% of encounters, medication adherence was 
assessed at 99.1% of encounters and side effects were assessed at 92.4% of encounters. 
Clinical Effectiveness: 40.9% (122/298) of patients had positive outcomes, including 18.8% 
(56/298) who remitted and another 22.1% (66/298) who responded. Maintenance: 91.9% 
(10/11) of the CBOCs chose to sustain the program after research funds were withdrawn. 
Despite the organizational barriers to implementation at smaller contract CBOCs, EBQI was 
found to be an effective facilitation strategy for disseminating Telemedicine Based Care 
Management.   
Finally, collaborator Dr. Andrew Pomerantz has studied the Co-Located Collaborative Care 
Model, in which MH providers are embedded in the PC team,20,21 and there is true integration of 
the biological and psychosocial care models. In contrast to the disease-specific Care 
Management model, the Co-Located Collaborative Care model addresses a wide array of MH 
disorders commonly seen in PC, as well as behavioral factors that negatively impact chronic 
medical conditions (e.g., smoking, medication non-adherence). Co-located MHPs include both 
prescribers such as a psychiatrist or advanced practice nurse (APN) and therapists such as 
PhD psychologists. These MHPs see patients in conjunction with PCPs and provide “curb side” 
consultation to PCPs in real time, which allows for the recognition and treatment of biological, 
psychological and social determinants of illness during a single encounter. The MHPs staff an 
open access clinic and PCPs conduct warm handoffs to MHPs in real time, virtually eliminating 
no shows (a chronic problem with the referral model). The Co-Located Collaborative Care 
Model was developed independently at Cherokee Health Systems (a Federally Qualified Health 
Center in Tennessee) and the White River Junction VAMC (which earned it the American 
Psychiatric Association's Gold Achievement Award and the Secretary of VA's Advanced Clinical 
Access National Champion Award). The Co-Located Collaborative Care Model was developed 
by providers rather than researchers, and rigorous scientific evidence is lacking regarding its 
impact on clinical outcomes because of the difficulties associated with randomizing patients 
(discussed in detail in the Research Design and Methods section). One non-experimental study 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
5 
in the VA demonstrated that patients completing a Co-Located Collaborative Care protocol had 
greater improvements in depression severity than non-completers.22  Other non-VA quasi-
experimental pre-post studies have found that it improves patient access to MH care by 
decreasing wait times and no-show rates,23-26 and increases patient functioning.27,28 Another 
non-VA study found high rates of patient and provider satisfaction.29 However, none of these 
studies had a control group and there was no random assignment to treatment conditions, 
making the findings difficult to interpret. The Co-Located Collaborative Care Model has been 
deployed into routine care by VHA, Kaiser Permanente, the United States Air Force, and 
Federally Qualified Health Centers. Also, Dr. Pomerantz has conducted several pre-post studies 
evaluating the Co-Located Collaborative Care model at the White River Junction VAMC.23,24,27,28  
Patients received care based on the traditional Referral model in the pre-period and based on 
Co-Located Collaborative Care in the post period. In one study, patients screening positive for 
depression (n=670) were followed retrospectively. The Co-Located Collaborative Care model 
was associated with a greater proportion of patients who had screened positive being treated for 
depression (52.3% vs. 37.8%; p<0.001), and received guideline concordant depression 
treatment (11% vs. 1%; p<0.001). In addition, the reallocation of MH staff from the specialty MH 
clinic to the Co-Located Collaborative Care clinic resulted in more patients with a positive 
screen seeing a MHP (36.0% vs. 9.0%, p<0.001). Another pre-post study evaluated the Co-
Located Collaborative Care model implemented at both the White River Junction VAMC and its 
affiliated CBOCs.24 Results at the White River Junction VAMC were similarly positive, but 
outcomes did not improve significantly at the CBOCs. This last finding suggests that the Co-
Located Collaborative Care model needs to be adapted for smaller CBOCs.  
Blended Model of Integrated Care: In their roles as the National MH Director, Integrated 
Services, and National Primary Care Director for PC-MHI, Drs. Pomerantz and Post recently 
developed the Blended Model of integrated care, which includes clinical activities associated 
with both Care Management and Co-Located Collaborative Care. The Blended model was 
designed to be staffed by on-site PhD level psychologists, psychiatrists (or APN) and nurse care 
managers. Like the Co-Located Collaborative Care model, the psychologists and psychiatrists 
staff an open access clinic facilitated by warm handoffs from PCPs. All patients (regardless of 
diagnoses) referred to the co-located MHPs receive at least one telephone call from the care 
manager to follow-up with progress towards the treatment plan and the co-located MHPs can 
choose to involve the care manager for longer depending on the patient's need. PCPs can also 
choose to refer the patient (regardless of diagnosis) to the care manager directly, bypassing the 
co-located MHPs (e.g., to monitor antidepressant adherence). Thus, the Blended model 
addresses a wide array of disorders and behavioral problems commonly seen in PC settings. 
Drs. Pomerantz and Post have facilitated the implementation of the Blended Model at VAMCs 
nationwide. However, because there is no scientific evidence documenting that the Blended 
model is clinically effective in CBOCs lacking on-site psychiatrists and PhD psychologists, PC-
MHI does not currently mandate that the Blended model be delivered in smaller CBOCs that 
serve rural Veterans.   
 
Integrated Care for Veterans with Infectious Diseases: Hepatitis C Virus (HCV) afflicts 
approximately 5% or 1,100,000 Veterans 30; one quarter of HCV+ are also living with HIV. 31,32 
Comorbid mental health problems also disproportionately affect HCV+ and HIV+ 
individuals.33,34. Further, HCV, HIV, and their associated conditions (e.g., cirrhosis, depression, 
early mortality) are more prevalent among racial minority Veterans than white Veterans.35 There 
are evidenced-based treatments for HIV, HCV, and comorbid depression, including Care 
Management; however, because of limited treatment rates, there appear to be several barriers 
to care that need to be explored and addressed in future interventions.   

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
6 
HCV Treatment. New antiviral treatments available in VA can essentially cure HCV. VA 
continues to fund HCV treatment, including $500 million last year.36  Less than 30% of HCV+ 
Veterans have received treatment. 37 In a non-systematic literature review, researchers 
identified barriers to HCV treatment among Veterans and non-Veterans within the health care 
system (e.g., insufficient funding), patients (e.g., adherence), providers (e.g., referral behavior), 
and clinical encounters (e.g., communication issues).37 However, no data exist exclusively from 
Veterans on treatment experiences and barriers or facilitators, especially since the new HCV 
treatment emerged.  
HIV & Depression Treatment. One treatment program exists to improve a condition 
associated with HIV—depression—and it can improve viral load and reduce depressive 
symptoms among Veterans 38  and is also cost-effective in VA.39 Briefly, the HIV/depression 
treatment is Care Management for depression in HIV clinics. This model of depression care is 
preferred by HIV+ Veterans. 40  However, barriers exist at multiple levels for HIV/depression 
treatment in VA, 40 include geographic access to care for rural-dwelling Veterans. 41 There are 
implementation data of HIV/depression treatment from eight Veterans42 and more are needed to 
reach saturation of themes, especially among vulnerable Veterans. 
Exploratory research is needed on racial minority and rural-dwelling populations to identify 
treatment preferences, barriers, and facilitators to using Care Management and other evidence-
based treatments in VA for Hepatitis C, HIV, and comorbid mental health problems. 
Known and Potential Risks and Benefits, if any, to Human Participants: 
This minimal risk study will consist of participants completing questionnaires about their 
health care, health status, mental health symptoms, comorbidities and other outcomes, and also 
of providers, clinical managers and other staff discussing care processes and intervention 
components.  The study will examine the effect of care models that have been endorsed and 
implemented throughout the VA as compared with usual care in CBOCs.  Possible risks for 
participants are tiredness and boredom; however, the research staff are skilled in conducting 
assessments and will take all precautions to ensure that any distress is minimized.  Loss of 
confidentiality is also a possible risk; however all information will be treated as confidential and 
safeguarded in accordance with the Privacy Act of 1974. 
Veterans may benefit from knowing that their participation may be useful to VHA leaders 
who are seeking to improve mental healthcare and infectious disease care for patients receiving 
care in CBOCs; however, study participation in itself will not likely result in any direct benefit to 
participants. 
Population to be Studied: 
Human Subjects Involvement and Characteristics 
Veteran Participants:  For the Specific Aim 2 Formative Evaluation of the initial 
implementation at the 6 sites, which will assess adaptation of the Blended model to the CBOC 
setting during the implementation phase at each site, approximately 24 Veteran patients will be 
enrolled to participate in qualitative interviews.  For the Specific Aim 2 Effectiveness test, we 
propose to enroll approximately 750 Veterans who screen positive on routinely administered VA 
MH screens (e.g., depression, alcohol, PTSD) at the 6 study CBOCs, for 18 months beginning 
at the first implementation period and continuing until the last two sites have completed the 
implementation phase (see methods section). Only those patients receiving specialty MH 
treatment in the 6 months prior to recruitment; those with a diagnosis of substance dependence; 
and those with a psychotic disorder diagnosis (schizophrenia, bipolar disorder, other psychotic 
disorders) will be excluded. Patients who consent to participate will participate in an initial 
assessment of health status by telephone and a follow-up telephone assessment six months 
later.   

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
7 
For the exploratory third aim at the CBOCs, up to 40 total Veterans will be enrolled to 
participate in qualitative interviews about their HCV or HIV/mental health treatment experiences 
and preferences. Two subsets of Veterans will comprise the total sample for this aim: 
1. 20 Veterans diagnosed with HCV.  
2. 20 Veterans diagnosed with HIV and screen positive on a depression screen.  
Provider Participants: For post-implementation interviews (Specific Aim 2 Formative 
Evaluation), the CBOC director, site champion, and any other relevant staff they recommend to 
us at each of the 6 study CBOCs will be recruited to participate in qualitative interviews. The 
telehealth MHPs involved in the study will also be interviewed.  
Inclusion of Non-Veteran Participants:  All patient participants will be Veterans.  Because 
some of the provider and manager participants will not be Veterans, the study will request 
approval to recruit non-Veterans. 
Sources of Materials 
Study data will be collected through qualitative interviews; interviewer administered self-
report questionnaires and rating scales; and extraction of Corporate Data Warehouse (CDW) 
data for the study sites (6 CBOCs and the 3 parent VAMCs) through the VA Informatics and 
Computing Infrastructure (VINCI).  Information obtained from the CDW will be used to identify 
potential participants for the Effectiveness Trial. This information will be will be collected 
specifically for research purposes.  We will also seek access to local CPRS systems at the 
study sites, or access to CPRS information at the sites that is available in the Compensation 
and Pensions Records Interface (CAPRI).   
Statement of Compliance: 
This study will be conducted in compliance with the protocol, good clinical practice, VA and 
CAVHS HRPP requirements, as well as other applicable regulatory requirements. 
Purpose and Specific Aims: 
Providing mental health (MH) care to rural Veterans in Community Based Outpatient Clinics 
(CBOCs) is a major priority of the Office of Rural Health (ORH). Likewise, integrating MH into 
Primary Care (PC) is one of the highest priorities of Mental Health Services (MHS) and the 
Office of Mental Health Operations (OMHO). However, at most smaller CBOCs serving rural 
Veterans, on-site mid-level MH providers (MHPs) and/or off-site tele-psychiatrists and tele-
psychologists deliver traditional referral-based specialty treatment (Referral model) rather than 
integrated care. This often results in long wait times and little collaboration between MHPs and 
PC providers (PCPs). For VAMCs and larger CBOCs, the Primary Care Mental Health 
Integration (PC-MHI) initiative mandates the implementation of the Blended model of integrated 
care, which requires implementing both of the predominant models of integrated care: 1) Care 
Management model and 2) Co-Located Collaborative Care model. The cornerstone of the Care 
Management model is a care manager (e.g., nurse supervised by a psychiatrist) who proactively 
conducts structured outreach encounters with patients between visits to their PCP. There is 
rigorous scientific evidence from randomized trials that the Care Management model is effective 
in facilities with a full range of on-site MH staff. Dr. Fortney conducted a randomized trial 
demonstrating that the Care Management model can be successfully adapted using 
telemedicine technologies (Telemedicine Based Care Management model) for patients treated 
for depression at CBOCs lacking on-site psychiatrists and PhD psychologists.15The cornerstone 
of the Co-Located Collaborative Care model is an open access clinic staffed by behavioral 
specialists (e.g., PhD psychologist, psychiatrist) who work side-by-side with PCPs. There is 
evidence from quasi-experimental studies conducted by Dr. Andrew Pomerantz of Mental 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
8 
Health Services that the Co-located Collaborative Care model is effective for facilities with a full 
spectrum of on-site MH staff.23,24 However, there is no evidence that the Blended model of 
integrated care is effective in smaller CBOCs that lack on-site psychiatrists and PhD 
psychologists. As a result, PC-MHI does not mandate that the Blended model be implemented 
in small and medium size CBOCs, where many rural Veterans receive care. This project 
contributes to Specific Aim 3 (Test clinical interventions to improve quality and outcomes of MH 
care at CBOCs) of the Little Rock CREATE (see Appendices 1 and 2) by filling critical gaps in 
the scientific evidence base about integrating MH into PC in smaller CBOCs.  
The goal of this proposed Hybrid Type 243 pragmatic effectiveness-implementation trial is to 
generate the scientific evidence needed to justify the national dissemination of the Blended 
model adapted to accommodate the clinical context of smaller CBOCs and to test the feasibility 
of using facilitation and the  Evidence Based Quality Improvement (EBQI) implementation 
strategy to deploy the Blended model in smaller CBOCs. Building on Dr. Fortney's (co-I) 
experience developing and implementing the Telemedicine Based Care Management model15 
and Dr. Pomerantz's experience developing and implementing the Co-Located Collaborative 
Care model, we propose to blend these models and adapt them for CBOCs lacking on-site 
psychiatrists and psychologists by using telemedicine technology.  The resulting Telemedicine 
Blended model will be compared to the Referral model in a pragmatic trial, where the 
intervention will be delivered by clinical staff available in routine care settings and fidelity will be 
monitored but not controlled.  Similar to our previous implementation science projects, some of 
this project’s activities involve implementation of standard, evidence-based care models.  We 
consider the related implementation activities to involve standard clinical and administrative 
efforts, integral to, but separate from the research activities described in this application.  
Specific Aim 1 (Quality Improvement/Implementation Aim): Use an expert panel comprised of 
clinical providers and managers who are applying telemedicine to provide a Blended model for 
CBOCs lacking on-site PhD psychologists and psychiatrists to document the core components 
of a Telemedicine Blended model and using a facilitated PDSA process, implement this model 
in six CBOCs.  
Specific Aim 2 (Research Aim): Conduct a Hybrid Type 2 pragmatic effectiveness-
implementation trial of the adapted Telemedicine Blended model by assessing RE-AIM 
outcomes including: provider Reach into the patient population, Effectiveness at improving 
clinical outcomes, Adoption by providers and Implementation fidelity.  
Hypothesis 1 (Reach): Compared to Veterans who receive the Referral model, a higher 
proportion of Veterans who receive the Telemedicine Blended model will have a MH 
encounter over a 6 month period. 
Hypothesis 2 (Effectiveness): Compared to Veterans who receive  the Referral model, 
Veterans who receive the Telemedicine Blended model will have better clinical outcomes at 
the 6 month follow-up. 
Specific Aim 3 (Exploratory Aim): In this exploratory study, we will conduct qualitative interviews 
to identify up to 40 Veterans’ perception of factors associated with implementation of VA-
approved treatments for HCV, HIV, and mental health treatment among Veterans with 
documented health disparities of HCV and HIV (i.e., racial minority Veterans). Specifically, we 
aim to:  
a. Identify Veterans’ experiences and preferences for HCV and/or HIV and mental 
health treatment. 
b. Identify barriers and facilitators to accessing HCV and/or HIV and mental health 
treatment. 
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
9 
Because there is a lack of scientific evidence documenting that the Blended model is effective in 
smaller CBOCs, PC-MHI does not currently mandate that it be delivered in these facilities. If the 
proposed study is successful, our clinical partners, Dr. Pomerantz (MHS), Dr. Post (PC-MHI) will 
use the results to justify implementation of the Telemedicine Blended model. Further, we can 
harness information gained in our exploratory aim to begin developing implementation 
strategies to better treat racial minority Veterans with HCV and/or HIV and depression, possibly 
through Care Management.  
 
Study Design: 
The overall objective of this proposed Hybrid Type 2 pragmatic effectiveness-implementation 
study is to generate the scientific evidence (clinical effectiveness and other implementation 
outcomes) needed to justify the national dissemination of the Blended model in smaller CBOCs 
lacking on-site psychiatrists and PhD psychologists and to test facilitation and EBQI as an 
implementation strategy in CBOCs. A hybrid study design has dual aims of assessing clinical 
effectiveness and implementation success. A Type 2 hybrid design simultaneously tests a 
clinical intervention and an implementation strategy.43 Hybrid effectiveness-implementation 
studies should increase the efficiency of translating research into practice compared to the more 
traditional approach of first conducting an effectiveness trial and then conducting an 
implementation trial. The proposed research will contribute to the development and refinement 
of this newly emerging translational science methodology. The proposed study design also 
employs many core elements of pragmatic comparative effectiveness trials including: 1) 
comparing the intervention to a commonly used active treatment, 2) applying relative few 
exclusion criteria, 3) enrolling a diverse set of patients, 4) delivering the intervention using 
clinical staff available in routine care settings, 5) monitoring, but not controlling fidelity, 6) 
defining clinical outcomes as changes in patient-reported symptoms, and 7) using intent-to-treat 
analyses to examine group differences.44 
Building on Dr. Kirchner’s (co-PI) extensive experience implementing and studying the 
implementation of PC-MHI models, Dr. Owen’s (PI) previous experience developing and 
implementing side effect monitoring in mental health settings, Dr. Fortney’s experience 
implementing the Telemedicine Based Care Management model and Dr. Pomerantz's 
experience developing and implementing the Co-Located Collaborative Care model, we 
propose in Specific Aim #1 to adapt the Co-Located Collaborative Care model for CBOCs 
lacking on-site psychiatrists/psychologists using telemedicine technologies and blend it 
with the Telemedicine Based Care Management model. This aim involves typical quality 
improvement activities rather than research—we will get input from experts who have already 
implemented this model, and then work with key clinical staff at sites to adapt the model so that 
the psychologists and psychiatrists deliver evidence-based “co-located” collaborative care using 
tele-video. 
Then, in Specific Aim #2, we will conduct an effectiveness trial comparing the Telemedicine 
Blended model to usual care (i.e., Co-Located Referral model). This research trial will use a 
stepped-wedge design, which will allow us to: (1) extend implementation support to the maximal 
number of clinics, and (2) enhance the formative evaluation of our implementation process.   
In Specific Aim #3, we will conduct telephone interviews with up to 40 racial minority 
Veterans (up to 20 HCV+ and up to 20 HIV+). These participants may or may not be also 
offered the opportunity to participate in Aim #2, depending on whether they meet inclusion 
criteria for Aim #2. The proposed study will provide preliminary data to our operational funding 
partner, the Office of Health Equity, to inform HCV/HIV treatment policy in VA and future grant 
submissions.  
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
10 
Procedures: 
Adaptation and Refinement (Specific Aim 1)  
The PARiHS implementation framework proposes that successful adoption of an evidence 
based practice (EBP) depends on: 1) evidence, 2) context, and 3) facilitation.45 Evidence 
includes results from randomized trials, as well as anecdotal evidence from clinical 
experience.46,47 Context includes both factors internal to the organization such as culture, 
climate and capacity,48 as well as external forces such as mandates and performance 
measures. Facilitation typically involves an integrated set of implementation strategies to 
promote adoption. In this study, we will facilitate implementation of the telemedicine blended 
model by facilitating the development of implementation plans by the clinical stakeholders, 
guiding them about the application of the evidence base, and measuring fidelity to the core 
components of the EBP.  In addition to providing expertise, researchers also facilitate problem 
solving and provide ongoing technical support for developing data collection/analysis tools, 
informatics, and training materials. The facilitation also emphasizes continuously revising the 
adapted EBP based on feedback during Plan-Do-Study-Act (PDSA) cycles, and thus should 
lead to adapted EBPs that are aligned with MHS/OMHO policy, as well as adapted EBPs that 
are robust, user-friendly, and feasible to deploy in real-world practice settings.  
Sites: We propose to work with the three parent VAMCs, which will each identify two CBOCs 
that lack on-site full-time psychiatrists and psychologists, or CBOCs with full-time psychiatrists 
and/or psychologists, but for which the VAMC Mental Health Chief identifies a major need for 
PC-MHI services, for a total of 6 CBOCs.  Potential Parent VAMCs include the Central Arkansas 
Veterans Healthcare System (Little Rock, AR); Veterans Health Care System of the Ozarks 
(Fayetteville, AR); Overton Brooks VA Medical Center (Shreveport, LA); and Southeast 
Louisiana Veterans Health Care System (New Orleans, LA). Final selection will be based on 
recommendations from Network Mental Health leadership and VAMC willingness to participate. 
All sites will have on-site Licensed Clinical Social Workers (LCSWs) and interactive video 
equipment available for telepsychiatry. Note: as of June 29, 2017, site selection has been 
finalized. We will study implementation at four CBOCs affiliated with CAVHS, and two CBOCs 
affiliated with the Overton Brooks VAMC. 
 
Expert Panel: Prior to adapting the Blended model, we will conduct an expert panel comprised 
of clinical providers and managers who are applying telemedicine to provide a Blended model 
for CBOCs to document the core components of a Telemedicine Blended model. Based on 
these core components we will create a steps and decision guide that can be used to develop 
site specific implementation plans at each of our CBOCs. This expert panel will be led by Dr. 
Karen Oliver (our External Facilitator), an epidemiologist who has supported the implementation 
of the blended model in CBOCs in VISN 12. 
Implementation Planning: Once participating CBOCs have been identified, the CBOC 
executive director will be asked to identify a site level champion for the program implementation. 
Local champions will likely be MH staff who would be involved in the delivery of the 
Telemedicine Blended model or supervisors such as the Medical Director or Director of Nursing. 
As part of a formative evaluation, Dr. Oliver and another member of the facilitation team will 
conduct pre-site visit phone calls with the CBOC executive director, the on-site MHP and the 
site champion to discuss potential barriers to and facilitators of implementation, and how the 
program will be implemented at the site, given the site’s organizational context.  In addition, we 
will request that each informant anonymously complete the Organizational Readiness for 
Change (ORC) scale through a sharepoint or the VA Research Electonic Data Capture 
(REDCap) site which will be located behind the VA firewall.  This scale will be used to provide 
information to the site that will help them plan their implementation. The information recorded 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
11 
during these interviews about implementation barriers and facilitators and about the site’s 
experiences and preferences with regard to implementation of PC-MHI integration models will 
be reviewed and organized for each site in order to provide efficient feedback to the facilitation 
team and inform the implementation process.  The research team has used these techniques 
successfully in the past to implement evidence-based practices in VAMCs.42  
Adaptation: The blending of the Care Management and Co-Located Collaborative Care models 
requires offering both types of functions simultaneously. Using the steps and decisions guide 
developed in Specific Aim 1, the Co-Located Collaborative Care model will be adapted for 
CBOCs lacking on-site PhD psychologists and psychiatrists and then blended with the 
Telemedicine Based Care Management model. Dr. Oliver and another team member will 
conduct a site visit to each CBOC where they will provide academic detailing on the blended 
model and meet with key stakeholders to help them adapt the program to meet local needs.  
Stakeholders will include on-site champions, mid-level MHPs, nursing and PC leadership; and 
off-site telephone nurse care managers, tele-psychiatrists, tele-psychologists and VISN TMH 
leadership. One key adaption issue is whether the brief focused psychosocial assessments 
should be conducted face-to-face by the on-site LCSW or by a tele-psychologist staffing an 
open access clinic. Psychologists have a more diverse skill set than LCSWs, but warm handoffs 
via interactive video may not be effective. The availability of desktop interactive video units at 
some CBOCs could facilitate tele-handoffs in PC exam rooms. Older wall-mounted or cart-
based interactive video units may be more appropriate for team meetings/huddles. The 
facilitation team will customize the Telemedicine Blended model for each participating CBOC 
while retaining the core elements of the Care Management and Co-Located Collaborative Care 
models. This adaptation process will constitute the Plan phase of the PDSA cycle. Throughout 
this pre-implementation adaptation phase, Dr. Oliver and the other facilitation team member will 
track these activities and adaptation decisions in a facilitation field log. Dr. Drummond, a 
medical anthropologist and implementation scientist, will review these logs and conduct 
debriefings with Dr. Oliver and the other facilitation team member at regular intervals to 
document adaptation in detail.  
Implementation: The Do stage of the PDSA cycle involves the implementation of the 
Telemedicine Blended model tailored for each site based on the outcomes of the Planning 
phase. The Blended model will be staffed by reallocating and retraining existing MHPs. Tele-
psychology and tele-psychiatry staff involved in delivering the Blended model will be trained by 
research team members or collaborators with expertise in the Co-Located Collaborative Care 
and Care Management models. If possible, there will be a face-to-face training conducted in a 
central location in VISN 16; it is possible that travel constraints will require that training be 
conducted using tele-video or telephone conferences. Post-training conference calls will be 
offered every two weeks with staff involved in the delivery of the Telemedicine Blended model. 
Each CBOC will have the option to pilot the Telemedicine Blended model for half day a week for 
2 months. We know that delivering the Blended model for a half day is feasible because the 
Bennington CBOC (associated with the White River Junction VAMC) runs an open access clinic 
in the morning and a referral clinic in the afternoon as part of routine care.  Each site may pilot 
test a different version of the adapted model (e.g., one CBOC may pilot test a version where the 
on-site LCSW receives the warm handoff, while another CBOC pilot tests a version where the 
tele-psychologist receives the warm handoff). Throughout implementation, Dr. Oliver and the 
other facilitation team member will continue to provide facilitation of activities at each site, and 
will continue to track activities and decisions in an implementation log. As above, Dr. Drummond 
will review these logs and conduct debriefings to fully document facilitation and implementation 
in detail.   
Effectiveness Trial (Specific Aim 2) 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
12 
Overview: To accomplish Specific Aim 2, we will enroll patients before, during and after the 
implementation phase to assess outcomes, using a stepped-wedge design.  We will also enroll 
and interview approximately four patients at each participating CBOC during early 
implementation to assess Veteran perspectives on barriers to and experiences with PCMHI 
care.  Findings from these interviews will be provided to the facilitators and inform the formative 
evaluation of each site (see Specific Aim 1 above and Formative Evaluation section below). 
We will employ the RE-AIM framework to assess Reach, Effectiveness, Adoption, 
Implementation, and Maintenance of the program. Post-implementation, we will conduct 
interviews with key informants at each site to collect feedback which will help us to interpret 
outcomes at each site as well as to improve facilitation and implementation processes for future 
efforts.  
Overall Study Design of Effectiveness Trial- The outcomes of the effectiveness trial will be 
based on the RE-AIM Framework (Reach, Effectiveness, Adoption, Implementation, 
Maintenance).49-52 Reach represents the proportion of eligible or targeted patients who receive 
the EBP.49 Adoption represents the proportion of staff who use the EBP.49 Implementation 
represents the fidelity of the EBP as implemented in routine care.49  Effectiveness represents 
the clinical impact (on patient outcomes) of the EBP as implemented in routine care settings.51 
Maintenance represents the degree to which the implementation of the EBP is sustained.49 The 
proposed effectiveness trial will focus on Adoption, Reach, Implementation Fidelity and 
Effectiveness. To have an impact on the health of the target population, an EBP must be 
adopted by providers, reach a large proportion of the targeted patient population, be 
implemented with high fidelity, effectively improve outcomes and be maintained after the 
research funds are withdrawn. Importantly, by measuring both Reach into the patient population 
and the effectiveness of those reached, we will be able to estimate the "population level impact" 
of PC-MHI.53  
Randomization - Randomization to the intervention or usual care cannot be conducted 
practically at the patient-level, as is done in most traditional RCTs, because the Telemedicine 
Blended Model (especially the Co-Located Collaborative Care components) represents a clinic-
level transformation. The Co-Located Collaborative Care model completely reorganizes clinic 
flow (i.e., team meetings, open access, warm handoffs, shared treatment plans) and it would not 
be practical to randomize a subset of patients to the intervention. For example, randomization at 
the patient-level would disrupt the flow of warm handoffs from PCPs to MHPs and hinder the 
development of shared treatment plans if busy PCPs become confused about which patients 
had been randomized to the intervention and which had been randomized to usual care. This is 
why there are no RCTs of the Co-Located Collaborative Care model. When an intervention must 
be delivered to all or none of the patients in a "cluster", investigators traditionally use cluster-
level randomization. The two most common types of cluster randomization are provider-level 
and clinic-level. Provider-level randomization is not feasible in our context because the mid-level 
Co-Located MHP could not simultaneously staff an open access clinic for some PCPs and a 
referral clinic for other PCPs. We discussed the possibilities of using clinic-level (e.g., CBOC) 
randomization with our Steering Committee, including our clinical partners, who felt that a 
stepped wedge controlled trial design would better serve the clinical needs of our sites.  The 
stepped wedge controlled trial design will allow us to: (1) extend implementation support to the 
maximal number of clinics, and (2) enhance the formative evaluation of our implementation 
process.  Stepped wedge designs are a recent development in controlled trial design that entails 
providing the intervention of interest to all participants, but staggering the introduction of the 
intervention.  This design is increasingly used where all participants must receive the 
intervention for policy or ethical reasons.   

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
13 
Instead of starting all intervention and control sites together, the design staggers the 
introduction of the Telemedicine Blended Model.  Stepped wedge designs thus have the 
disadvantage of exposing the aggregate of sites to a longer period of potential secular trends; 
however, by aligning in time sites awaiting intervention with sites undergoing intervention, 
stepped wedge designs can identify and control for these trends better than parallel-groups 
randomized controlled trials38.  Figure 1 displays the progressive nature of the stepped wedge 
design. 
CBOCs are the unit of randomization, with CBOCs randomized to an intervention start date 
using the stepped wedge design.  A total of 6 CBOCs will participate in the study, with two 
CBOCs allocated to each of the 3 start dates.  Because sites may differ with regard to 
organizational/program characteristics, we will use the restricted selection method of 
randomization to balance key site characteristics over time.  For example, if somewhat larger 
CBOCs (with regard to number of uniques) were randomized to participate in the first wave of 
implementation and the smallest sites were assigned to the last wave, this would make it likely 
that urban and rural sites occur both in early and late waves of implementation. We will utilize a 
computer-based algorithm to balance site characteristics as much as possible using key site 
characteristics including: 
• 
CBOC size (number of uniques served) 
• 
Parent VAMC 
• 
Number of on-site mental health providers/staff 
• 
Percent of appointments that are conducted with specialists via telemedicine 
We may use additional characteristics suggested by clinical partners.  Obviously, a small 
number of sites and multiple categories of site characteristics rule out perfect balance. The 
algorithm provides as much balance as possible in the spirit of an ANOVA balanced incomplete 
block design. Randomization will be computer-generated by the statistician.  A subset of 
patients will be selected at each CBOC during each of 5 measurement periods shown in the 
diagram below (implementation planning, implementation and step down).    
 
Patient Recruitment: We will initiate recruitment at all sites 3 months into the first 6-month 
study period (labeled as “Identify Model” in the diagram above), during the implementation 
planning for Sites A and B.  We will complete recruitment 3 months before the data analysis 
period begins.  Thus, Veteran follow up will extend into the first 3 months of the data analysis 
period for all sites.  Since the duration of the implementation period may vary from site to site, 
we will consider a site’s first referral to the Telemedicine Blended Model (with a PC-MHI 
telemedicine stopcode) as the index date with all Veterans enrolled prior to that date designated 
as controls and all Veterans enrolled after that date designated as intervention participants.  
Figure 1: Implementation Timeline 
 
Inclusion Criteria: We will sample from the patient population who screen positive on routinely 
administered VA MH screens (e.g., depression, alcohol, PTSD) at the 6 CBOCs, beginning 6 
Grant 
months 
Identify 
Model 
 
Intervention 
Data 
Analysis 
6 months 
6 months 
6 months 
6 months 
6 months 
6 months 
6 months 
Site A 
 
 
Implementation 
Planning 
Implementation 
PDSA 
Step Down 
 
 
Site B 
Site C 
 
 
Implementation 
Planning 
Implementation 
PDSA 
Step Down 
 
 
Site D 
Site E  
Identify 
Model 
 
 
Implementation 
Planning 
Implementation 
PDSA 
Step Down 
 
 
Site F 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
14 
months prior to the first implementation period (for sites A and B) and continuing through the 6 
month period following the implementation phase for sites E and F (see above diagram). We will 
exclude only those patients receiving specialty MH treatment in the 6 months prior to 
recruitment; those with a diagnosis of substance dependence; and those with a psychotic 
disorder diagnosis (schizophrenia, bipolar disorder, other psychotic disorders). Receipt of 
specialty MH care will be defined as having an encounter with a 500 series stopcode (except 
534, which represents a PC-MHI encounter such as a telephone care manager encounter). The 
justification for this exclusion criterion is that patients enrolled in specialty MH care are already 
receiving a higher level of care than Telemedicine Blended and would not be expected to 
benefit clinically. The justification for excluding patients with substance dependence and a 
psychotic disorder diagnosis (schizophrenia, bipolar disorder, other psychotic disorders) is that 
currently there are no evidence based PC-MHI interventions for such patients. Finally, by 
focusing on all patients not already enrolled in specialty MH, it allows us to estimate the 
intervention's "population level impact" rather than its efficacy for a narrowly defined group of 
patients.53  
Participant Recruitment and Consent: Using methods developed in a prior RCT we propose 
to extract appointment data from the Corporate Data Warehouse (CDW) and identify those 
Veterans with upcoming PC CBOC appointments scheduled at least one month in advance and 
who have no MH encounters in the previous 6 months. Real SSN access will be requested so 
that we can obtain contact information for potentially eligible patients. We are requesting a 
Wavier of Informed Consent and a Waiver of HIPAA Authorization for this initial phase of the 
recruitment process. We will send opt-out letters to the identified Veterans, noting that we may 
contact them about participation in a research study.  Veterans may opt out by telephone or 
mail.  On a daily basis, we propose to extract mental health screening data from the Corporate 
Data Warehouse and identify those Veterans who have not opted out of the study and who 
screen positive for alcohol abuse,  depression, or PTSD.  An RA will attempt to contact each 
such Veteran by telephone in order to conduct the informed consent process and complete the 
baseline interview. We are requesting a Waiver of Documentation of Informed Consent, as well 
as a HIPAA Waiver of Authorization for participation in the Effectiveness trial because we need 
to conduct the initial assessment interview as soon as possible after the clinical encounter with 
a positive screen.  The study qualifies for these waivers because it is minimal risk, because it 
would not be practicable to conduct the study without the waivers (see above), because the 
research team will never be physically interacting with the participants in person, and the 
interaction with patients involves activities for which informed consent is not required when they 
occur in the course of standard clinical care (see Supplements L).  When we send the opt-out 
letter, we will include information required for the informed consent process, so that the potential 
participants will have this available during the telephone contact.  Those who are unable to 
understand or engage in the informed consent process (e.g., cognitive impairment, intoxication) 
will be excluded. The recruitment phase will last for 3 years. Opt-out letters will be sent to 
Veterans 3-4 weeks prior to the upcoming appointment. Those patients not opting out (by 1-800 
telephone or mail) within two weeks, attend their primary care visit and screen positive for 
depression, alcohol, or PTSD during this visit will be contacted for recruitment. If we contact 
eligible patients who have not opted out of the study, but they state they cannot recall receiving 
the opt-out letter and study information, we will send another letter and study information sheet.  
We will contact them a week after the 2nd letter is mailed.  This will ensure that we can reach 
them within the study recruitment window. 
Usual Care: Patients at CBOCs that have not (yet) implemented the Blended model will receive 
usual treatment under the standard Co-Located Referral Model which is described in the 
Background section.  

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
15 
Intervention Group: At CBOCs that are implementing the Blended model as called for by the 
stepped wedge design, patients will have access to the Blended model described in the 
Background section as adapted for CBOCs using telemedicine technologies during Specific Aim 
1.  
Administrative Data: VA service utilization will be collected from the CDW, where all VA 
facilities upload and store encounter data, using methods we have employed previously.54 Data 
will be collected on all patients at the study sites (hereafter referred to as the administrative 
sample) regardless of whether the patient was recruited and provided informed consent to 
participate in the primary data collection efforts associated with the trial. Encounter data will be 
collected for each patient for the 6 months following their first encounter in each 6-month 
observation period. Encounters to 500-599 stopcodes at CBOCs and VAMCs will be classified 
as MH. Encounters using the 534 stopcode (a stopcode used only for PC-MHI visits) will be 
classified as Telemedicine Blended. We will identify all patients (by scrambled social security 
number) with encounters to the study CBOCs and categorize them to the intervention or usual 
care group based on the date of their first encounter. We will also extract administrative data 
about MH diagnoses, age, gender, marital status, percent service connection and zipcode. MH 
diagnoses will be used to define diagnostic categories including: adjustment disorders, alcohol 
use disorders, anxiety disorders, attention deficit disorders, bipolar spectrum disorders, 
dementia, depressive disorders, drug use disorders, personality disorder, sleep disorders, 
stress reaction disorders, and other. The rurality of Veterans will be measured based on their 
zipcode using two different methods: 1) Rural Urban Commuting Area (RUCA) codes, and 2) 
VA definition. A zipcode to census tract crosswalk will be used. RUCAs are a census tract-
based classification scheme that utilizes the standard Bureau of Census Urbanized Area and 
Urban Cluster definitions. RUCA also takes into account commuting patterns to Urbanized 
Areas, Urban Clusters, or smaller population centers to classify census tracts into 33 distinct 
categories, which typically are combined into 4 categories: 1) Urban - areas have Metropolitan 
cores and substantial commuting flow patterns to Urbanized Areas, 2) Large Rural Towns - 
have Micropolitan cores and substantial commuting patterns to Urban Clusters, 3) Small Rural 
Towns - have primary commuting flows to or within population centers of between 2,500 and 
9,999 residents, and 4) Isolated Rural Towns - are less populated rural areas with no 
commuting flows to Urbanized Areas or Urban clusters.55 VA’s classification scheme defines 
census tracts that belong to Urbanized Areas as Urban and all others as Rural, except for those 
in counties with average population density of less than 7 residents per square mile, which are 
defined as Highly Rural.55 The Real SSN Crosswalk file will be used to link primary data to the 
administrative data for participants enrolled in the effectiveness trial. 
Dependent Variable for Adoption: Patients will be assigned to PCPs using the Patient Care 
Management Module from the Austin Automation Center which records every patient's PCP 
assignment start date and assignment end date. We have used this file successfully to measure 
provider adoption in a recent QUERI funded study (SDP 08-316). We will classify each Veteran 
in the administrative sample as having received Telemedicine Blended services if they had a 
534 stopcode during their first encounter or a 534 stopcode encounter during the 6 month 
follow-up period. To measure provider adoption, we will categorize CBOC PCPs as adopters if 
they have at least one patient with the 534 stopcode encounter. In addition, we will document 
the percent of the PCP’s patients that with a 534 stopcode encounter.  
Data Analysis for Adoption - The percentage of PCPs adopting the intervention will be 
calculated. The adoption percentage will also be reported at the CBOC level. 
Dependent Variable for Reach (Hypothesis 1): To measure Reach into the patient population, 
we will classify each Veteran in the administrative sample as having received MH services if 
they had a 500-599 series stopcode (including 534) during their first encounter or any encounter 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
16 
during the 6 month follow-up period. Exploratory analysis will also examine number of MH 
encounters. We will also report the percentage of Veterans in the administrative sample who 
had a PC-MHI (534) stopcode during their first encounter or any encounter in the 6 month 
follow-up period in order to document the proportion randomized to the intervention who 
received Telemedicine Blended care.  
Data Analysis for Reach (Hypothesis 1): A dummy variable representing intervention group 
assignment will be specified as the explanatory variable of interest. An alpha significance level 
of 0.05 will be used to reject/accept the null hypothesis. Significant intra-class correlation 
violates the independence assumption of standard regression models and may cause 
underestimation of coefficient standard errors, possibly leading to incorrect inferences 
concerning the rejection of the null hypotheses. Therefore, the first step of the statistical 
analysis will be to test for lack of independence among observations within clusters using intra-
class correlation coefficients at CBOC level. Specifically, using a likelihood ratio test, we will 
compare the -2log likelihoods for an unrestricted model to a model that restricts the intra-class 
correlation to be zero. Raudenbush recommends that unconditional models (i.e. without 
explanatory variables) should be estimated prior to considering conditional models (i.e., with 
explanatory variables).56 If it turns out that the -2log likelihoods are not significantly different, the 
hypotheses will be tested using a standard logistic regression model. The following casemix 
factors will be included in the regression equation: MH diagnostic categories, age, gender, race, 
marital status, percent service connection and rurality. Missing race data will be handled by 
specifying an unknown race category. Conversely, if the results of the likelihood ratio test 
suggest that the intra-class correlation is significant, a mixed logistic model will be used.57,58 The 
mixed-model will include a random effect for the intercept and fixed effects for the patient-level 
variables (including treatment group assignment). The variance-covariance matrix will be 
specified to be unstructured. 
Chart Review Data: For the Veterans who are enrolled in the Effectiveness trial during the 
intervention period (the period following the study index date at each site), we will conduct a 
chart review to determine whether they received high fidelity integrated care during the six 
months after enrollment, obtaining access to either the CPRS system at each site or accessing 
CPRS information through the national Compensation and Pension Record Interchange 
(CAPRI), as Dr. Owen did in a recent study (IRBNet #219-789). If we elect to access the CAPRI 
system, we will only request access to local data/records at the study CBOCs and their parent 
facilities (6 CBOCs and 3 VAMCs).  
Once the Blended model has been adapted, the patient-level fidelity tool will be developed with 
our national partners, Drs. Pomerantz and Post based on the Primary Care-Mental Health 
Integration Blended Programs Functional Tool (See Appendix 4). This tool was recently 
designed to assess program-level fidelity to the Blended model and will need to be modified to 
assess patient-level fidelity. Using methods developed for our NIMH R01 MH076908 depression 
care management trial, care management fidelity will be assessed based on whether the 
following clinical activities are documented in progress notes: 1) symptom severity measured 
using structured assessments: PHQ9 (depression), GAD7 (generalized anxiety), PCL (PTSD), 
AUDIT-C (alcohol) scores, 2) self-management activities, 3) antidepressant adherence 
assessments and side-effect assessments (for those prescribed antidepressants) and 4) 
counseling adherence assessment (for those initiating psychotherapy). Co-Located 
Collaborative Care fidelity will be assessed based on whether the following clinical activities are 
documented in progress notes: 1) warm handoff (determined from encounter dates/times with 
PCP and co-located MHPs) during initial visit, 2) brief initial assessment with therapist 
(determined from CPT codes – e.g., 90804), 3) brief initial assessment with MH prescriber 
(determined from CPT codes – e.g., 90805), 4) initial assessment is problem focused, 5) initial 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
17 
assessment includes behavioral health issues (e.g., smoking, weight management), 6) inter-
professional communication at each encounter (determined by PCP and co-locate MHPs co-
signing notes), 7) co-located MHPs use motivational interviewing, self-management, problem 
solving therapy, cognitive techniques, and brief alcohol interventions, and 8) symptom severity 
is measured using structured assessments (e.g., PHQ9).  
Dependent Variable for Implementation Fidelity: Fidelity will be measured as a percentage: 
with the numerator specified as the number of documented clinical activities and the 
denominator specified as the number of clinical activities relevant to that patient. For example, 
medication adherence and side-effect assessments will only be specified in the denominator if 
the patient has been prescribed medications.  
Data Analysis for Implementation Fidelity: - The mean, median, standard deviation and 
range of implementation fidelity will be reported for all Veterans enrolled during the intervention 
phase, as well as the sub-sample with a PC-MHI (534) stopcode. The fidelity measures will also 
be aggregated at the CBOC level.  
Interview Assessment Data: We will administer research assessments to those Veterans 
providing informed consent to participate in the primary data collection efforts associated with 
the trial. The baseline assessment will be conducted within two weeks following the regularly 
scheduled PC appointment and six months afterwards. Assessments will be administered over 
the phone by trained RAs using a CATI system. Table 3 lists the instruments and Appendix 5 
contains the actual survey items to be used in the baseline interview. The demographics 
section, barriers assessment, perceived access inventory, and readiness ruler will only be 
administered at baseline and used for casemix adjustment. All other instruments will be 
administered at both baseline and follow-up. The rurality of Veterans will be measured based on 
their self-reported zipcode using the methods described above.  
Dependent Variable for Effectiveness (Hypothesis 2): Change scores between baseline and 
six months will be calculated for non-disease specific outcomes including: MH status (SF12V 
MCS), physical health status (SF12V PCS), pain (Pain scale), sleep (Jenkins sleep scale), 
disease specific symptoms such as depression (PHQ9), general anxiety disorder (GAD7), panic 
disorder (PDSS-SR), PTSD (PCL-5) and alcohol use (AUDIT-C), and self-management of 
chronic disorders such as diet and exercise (Prime-Screen), smoking (BRFS tobacco use items) 
and medication adherence (Miklowitz adherence scale). The primary outcome for this 
hypothesis is the MCS from the SF12V. The other outcome domains will be examined in 
exploratory analyses. Exploratory analysis will also examine the variation in SF12V and disease 
specific outcomes across diagnostic categories (identified by baseline scores of the PHQ9, 
GAD7, PDSS-SR, PCL-5, and AUDIT-C).  
 
Table 3 - Research Assessments 
Instrument 
Construct 
Socio-Demographics 
18 items that measure socio-economic and military characteristics 
Hoge Barriers 
Assessment 
14 item measure of perceived access, need and treatment 
effectiveness59 
Perceived Access 
Inventory 
43 item perceived access instrument developed by CREATE Project 
1 
Readiness Ruler 
3 items that assess perceived readiness to seek treatment60 
Miklowitz-Adherence 
Scale 
2 item medication adherence scale61 
SF12V 
12 items addressing overall physical and mental health functioning62 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
18 
Pain Scale 
Single item participant rating of the average overall level of pain for 
the past week rated on a continuous scale from 0 to 10. 
Jenkins Sleep Scale 
4 item measure that assesses trouble falling and staying asleep, 
and feeling tired during the daytime63 
Prime-Screen 
6 item assessment of dietary and exercise habits/behaviors64 
PHQ9 
9 item inventory that yields a continuous and dichotomous 
assessment of depression65 
GAD7 
7 item inventory that yields a continuous and dichotomous 
assessment of generalized anxiety disorder66 
APA DSM 5 Severity 
Measure for Panic- Adult 
10 item inventory that yields a continuous and dichotomous 
assessment of panic disorder67 
PCL-5 
20 item inventory that yields a continuous and dichotomous 
assessment of PTSD68 
AUDIT-C 
3 items that yield a continuous and dichotomous assessment of 
alcohol use69 
BRFSS Tobacco Use 
5 items to assess current tobacco use 
CSQ-8 
8 items to assess client satisfaction with MH services 70 
Note:  The Perceived Access Inventory is currently being developed as the main product of a 
project led by Dr. Jeffrey Pyne, and is expected to be finalized in early 2016.  In the Patient 
Survey document that contains copies of all the other questionnaires, there is a placeholder 
page for this survey.  We will submit this questionnaire to the IRB once it is completed and 
before using it in our assessment interviews.   
Measures Taken to Minimize/Avoid Bias, such as Randomization and Blinding 
We will use CDW data and an opt-out approach to recruit a representative sample of 
Veterans meeting the eligibility criteria for the effectiveness trial.  Sites will be randomized with 
regard to timing of implementation.  Research assistants will be blind to the status of 
implementation at the sites. 
Formative Evaluation:  Qualitative Interviews with Veterans during Early Implementation 
and with Site Personnel Post-Implementation 
In order to minimize burden on sites, and because gold standard facilitation involves activities 
normally conducted for both pre-implementation and progress-focused formative evaluation 
(and thus will be completed as part of the quality improvement activities of Specific Aim 1), we 
will only conduct parts of the formative evaluation under Specific Aim 2. Specifically, study team 
members will conduct progress-focused interviews with a sample of eligible Veterans who 
receive PCMHI care early in the implementation effort at each site, and post-implementation 
interviews with site personnel, including the CBOC Executive Director, the study site champion, 
and any other relevant staff they recommend to us.  
Veteran participant interviews: 
Patients will be identified by the PC-MHI provider (co-located or tele-co-located), who will 
provide names and contact information via encrypted e-mail or telephone, or the list will be 
uploaded to a secure research sharepoint site at CAVHS.  The potential patient participants will 
be sent an opt-out letter and recruited by phone if they do not opt-out (by 1-800 telephone or 
mail) within two weeks. Because we need to interview patient participants soon after they 
receive their mental healthcare, we are requesting a Waiver of Documentation of Informed 
Consent and a Waiver of HIPAA Authorization for participation in this study component. We will 
obtain verbal consent over the phone, and document that we have obtained verbal consent in 
the research record. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
19 
In order to understand Veteran experiences and preferences regarding receiving PC-MHI 
services via telehealth, we will conduct interviews with approximately four patients at each site 
(approximate n=24: 4 interviews X 6 CBOC sites). We will use semi-structured interview guides 
(see Appendix 3) covering the topics of Veteran experiences with the telehealth intervention, 
barriers encountered in receiving services via telehealth, and preferences for receiving care via 
telehealth or otherwise. These results will be fed back to the implementation/facilitation team for 
use in further refining how PCMHI via telehealth is being implemented and improved at each 
site. As part of our overarching aims of the various CREATE projects, we will also ask questions 
designed to refine the SOTA model of barriers rural populations experience in accessing care 
(see Appendix 3).    
Post-Implementation Interviews:  
We will conduct key informant interviews with personnel at each CBOC to gather feedback on 
the intervention. Potential participants will include the CBOC Executive Director, the study site 
champion, and any other relevant staff they recommend to us. Dr. Oliver will also recommend 
participants based upon her experiences in working with the sites. Dr. Drummond will conduct 
these interviews and take extensive notes. She will then abstract these notes into a summary 
template for each interview, and later compile templates across interviews within a given site.    
CBOC staff recruitment and consent:  Following procedures we have used in other studies for 
these kinds of interviews, a study team member will email potential participants to explain the 
purpose of the interview and attach a study information sheet. We are requesting a Waiver of 
Informed Consent for recruitment and a Waiver of Documentation of Informed Consent for 
participation, which enhances employee participant anonymity and confidentiality.  We will 
obtain verbal consent over the phone and document that we have obtained verbal consent in 
the research record.   
 
Exploratory Study of HCV/HIV & Depression Treatment Experiences (Specific Aim 3): 
 
Overview: Given prevalence of these diseases, barriers to care, and racial disparities, research 
is needed to improve implementation of HCV and HIV depression treatment for racial minority 
Veterans. Data are needed on HCV and HIV depression barriers and facilitators to treatment for 
Veterans who have documented disparities in these conditions, especially given VA spending 
on the new HCV treatment option. Specific data are also needed to develop implementation 
strategies to increase treatment for Veterans in rural areas who have unique barriers because of 
geographic location (14,15). The proposed study will address this by conducting qualitative 
interviews with Veterans with HCV and/or HIV and depression to examine treatment 
experiences, preferences, barriers, and facilitators. 
 
Sites: We will recruit participants from the same sites utilized in Aims #1 and #2: potential 
parent VAMCs including the Central Arkansas Veterans Healthcare System (Little Rock, AR); 
Veterans Health Care System of the Ozarks (Fayetteville, AR); and Southeast Louisiana 
Veterans Health Care System (New Orleans, LA). Final selection will be based on 
recommendations from Network Mental Health leadership and VAMC willingness to participate 
– final sites are CBOCs affiliated with CAVHS and the Overton Brooks VAMC.  
 
Study Design: Because this is an exploratory aim and the goal is to generate data and 
hypotheses about barriers and facilitators to adequate HCV, HIV, and comorbid mental health 
care, we will conduct qualitative interviews with up to 40 Veterans.  
 
Patient recruitment and consent: We will utilize a one-time recruitment strategy to 
enroll participants using a recruitment process separate, but similar, to those in Aim #2. See 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
20 
Figure 2. We will harness recruitment efforts currently existing in the parent study and if we are 
unable to generate adequate sample size using this method, Veterans will be identified by their 
Patient Aligned Care Team provider. Through either method, Veterans will first be sent an opt-
out letter. If Veterans do not call our research team within two weeks to opt out of the study, 
they will be called by our research team to assess interest and confirm eligibility in the study 
using a screening questionnaire. We are requesting a Waiver of Informed Consent for 
recruitment and a Waiver of Documentation of Informed Consent for participation.  We will 
obtain verbal consent over the phone and document that we have obtained verbal consent in 
the research record.  If Veterans are eligible and consent to participate, they will be scheduled 
with Dr. Woodward for an interview via phone at their convenience. 
 
Figure 2. Recruitment Flowchart for Exploratory Study (Aim #3) 
 
Inclusion Criteria: All Veterans in the exploratory aim must identify with a race other 
than white--.e.g, black, African American, or Latino. There are two subsets of inclusion criteria, 
one for each subsample within this exploratory study aim. (1) HCV: Veterans must have a chart 
diagnosis of HCV. (2) HIV. Veterans must have a chart diagnosis of HIV and screen positive for 
depression. 
 
Data sources: There are two potential sources of data for this exploratory Aim #3:  
(1) 
Administrative Data also utilized for Specific Aim #2 (Effectiveness). Data will 
be collected on all patients at the study sites regardless of whether the patient 
was recruited and provided informed consent to participate in the primary 
data collection efforts associated with the trial. From this larger administrative 
dataset, Veterans diagnosed with either HCV or HIV will be recruited and sent 
opt-out letters, allowing them two weeks to opt out of the exploratory study.  
(2) 
Interview Data: This includes questions about HCV or HIV/depression 
treatment experiences, preferences, barriers, and facilitators; a demographics 
questionnaire. 
 
Timeline: Recruitment will occur between July – September 30, 2016. 
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
21 
Follow-up Interviews:  We will contact participants with HCV who are enrolled for Aim 3 
by telephone to assess whether they have initiated HCV treatment since the interview and if so, 
assess their reasons for doing so. These Veterans agreed at the end of the initial interview that 
they would allow our research team to contact them again, if needed. The phone call will last 
approximately 5-10 minutes and will follow a semi-structured script, allowing the interviewer 
(either Dr. Woodward or the original research assistant who called the participant) to 
systematically ask questions while also exploring other themes Veterans highlight. The purpose 
of these follow-up interviews is to assess whether the initial interview itself (especially the 
education provided to Veteran participants about the new HCV treatment) served as a proxy or 
intervention for helping Veterans initiate HCV treatment. We may also ask Veterans to clarify or 
comment on any results from our analysis of their initial interviews—this “member checking” 
procedure is common in qualitative analysis to ensure valid data interpretation.   
 
Analysis: Dr. Woodward and other research staff will code the qualitative data using 
qualitative analytic techniques.  
 
Measures Taken to Minimize/Avoid Bias, such as Randomization and Blinding:  
 
We will use CDW data and an opt-out approach to recruit a representative sample of 
Veterans meeting the eligibility criteria for the exploratory aim.  Because this is an exploratory 
aim, randomization and blinding is neither possible nor required for proper scientific rigor.  
Expected Duration: 
Participation for the formative evaluation interviews will be limited to one interaction for 
qualitative data collection.  Participants in the effectiveness trial will be interviewed at the time of 
enrollment and 6 months thereafter, as described above. Participants in the exploratory aim will 
be interviewed at time of enrollment between July and September 30, 2016.    
Stopping or Discontinuation Criteria for Individual Participants: 
Participants are free to discontinue participation in interviews at any time.  An individual’s 
participation will be discontinued if the participant appears to be experiencing severe distress 
related to the intervention or assessments. 
Selection and Withdrawal of Participants: 
For the effectiveness trial, Veterans with an upcoming CBOC appointment who screen 
positive on routinely administered VA MH screens (e.g., depression, alcohol, PTSD) at the 6 
study CBOCs will be identified for recruitment.  Only those patients receiving specialty MH 
treatment in the 6 months prior to recruitment, substance dependence, and those with a 
psychotic disorder diagnosis (schizophrenia, bipolar disorder, other psychotic disorders) will be 
excluded.  
During the formative evaluation, we will enroll key informants (employees and Veteran 
patients).   Veteran patients will be identified by the local MHP based on their exposure to the 
telemedicine blended care model, and recruited by study staff.  Potential employee participants 
at each site will include the CBOC Director and the site champion, who will recommend other 
relevant individuals. The study team facilitator will also recommend CBOC staff who participated 
in implementation activities.   
For the exploratory aim, Veterans identified for recruitment will be diagnosed with either 
HCV or HIV and screen positive for depression during phone screening. Only those patients 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
22 
without an HCV or HIV diagnosis, are diagnosed with HIV but do not screen positive for 
depression, or whose race is white will be excluded. 
Participants may withdraw at any time from any aim of this study.  If a participant withdraws 
before completion of the baseline assessment, they will be replaced with another participant.  
Data from the withdrawn participant will not be used for the study. 
Assessment of Safety: 
The study consists of qualitative telephone interviews with employees and Veterans 
(formative evaluation), and telephone assessments of clinical status and outcome with Veteran 
patients.  There are also activities by the research team to facilitate implementation of the 
telemedicine blended care model.  These activities do not constitute physical safety risks 
beyond regular healthcare activities.  In addition, the questionnaires and interview questions 
being used in this study are unlikely to cause distress, but the research staff will discontinue any 
study procedure, whether by interview or telephone contact, if a participant reports or the study 
staff member detects substantial distress.  If this occurs, the interviewer will aid the participant in 
returning to emotional equilibrium before the end of the session.  Participants will be asked at 
the end of the sessions whether they have any questions or concerns.  If distress is reported, or 
if an individual continues to appear distressed, the interviewer will probe further to determine the 
nature and severity of symptoms and provide referral as needed.  In emergent situations (e.g., 
acute psychological distress or voiced thoughts of harming themselves or others) the research 
staff member will follow the appropriate suicide risk procedure (see Suicide Risk Assessment 
document). For participants who voice thoughts of harming themselves or others, the 
interviewer will stay on the line with the patient while connecting to the National Suicide 
Prevention Lifeline, using the phone number for within-VA transfers to the crisis line: (585) 393-
7938. The interviewer will introduce the patient and relay all pertinent information and reason for 
the call.  After the warm hand off to the crisis line staff, the interviewer will alert the PI (Richard 
Owen) via encrypted email.  For patients at Overton Brooks (Shreveport) CBOCs, the 
interviewer will also alert the Shreveport Suicide Prevention Coordinator, providing all 
information about the interaction. For patients affiliated with CAVHS or Overton Brooks whose 
responses to safety risk questions indicate moderate suicide risk, Dr. Owen or the designated 
“on-call” clinician-co-investigator who is covering for Dr. Owen in his absence will be alerted 
within 24 hours via encrypted email.  
Data Analysis for Effectiveness (Hypothesis 2):   
Randomization 
CBOCs are the unit of randomization, with CBOCs randomized to an intervention start date 
using the stepped wedge design.  A total of 6 CBOCs will participate in the study, with two 
CBOCs allocated to each of the 3 start dates.  Randomization will be computer-generated by 
the statistician.  A subset of potential patients (based on inclusion criteria) will be selected at 
each CBOC on a daily basis during the 5 measurement periods shown in Figure 1: Grant 
Activity Timeline (the formative evaluation control period(s), implementation, and step down 
period(s)).  
 
Blinding 
CBOCs and the research team will not be blinded to the intervention start dates.  However, 
outcome assessment by independent evaluators (research assistants) will be blinded. 
 
Statistical Analysis 
The analyses will be conducted using the intention-to-treat principle, with CBOCs analyzed 
according to their randomized, crossover time irrespective of whether crossover took place at 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
23 
the specified time.  The results across the unexposed (control phase) observation periods will 
be compared with those across the observation periods in which the CBOCs are exposed to the 
Telemedicine Blended Model intervention (intervention phase).  Characteristics of the patients 
and the CBOCs will be summarized by exposure status to examine potential selection bias or 
lack of balance.81  Because we have a small number (3) of randomization steps, we will 
compare numbers of patients analyzed, cluster (CBOC) size, CBOC characteristics and patient 
characteristics by randomization group.  Patient-level data will be collected, with the primary 
analysis being patient-level. 
 
The CBOC-level analysis will examine the primary and secondary outcomes, summarized as 
the mean and standard deviation across CBOCs before and after implementation of 
Telemedicine Based care.  At each facility, we will determine the index start date at which the 
first utilization of the telemedicine based service began.  This will allow us to take into account 
the differing lengths of time it may take to implement the intervention at various CBOCs.  Before 
this index date, patients participating in the study at that CBOC will be considered as being in 
the control period and those participating after the index date will be considered in the 
intervention phase.  Analyses of before to after implementation will be conducted for continuous 
outcomes using linear mixed models.  The interaction of intervention by CBOC will test if results 
differ by CBOC.  We will adjust for baseline differences among CBOCs in the model.82  
 
The patient-level analysis will be a generalized linear mixed model (GLMM) which includes fixed 
terms for intervention (usual care referral model versus Telemedicine Based care) and time (6-
month time periods for each step).  The model will use random effects to model the correlation 
of patients within CBOCs.  
The random effects model is: 
Yijl = µ + βj + Xijθ + ui + εijl ;  
ui ~ N (0, σ2b) , εijl ~ N (0, σ2w) 
where βj = fixed effect of time; Xij  = 1 if intervention, 0 otherwise; θ = intervention effect; ui = 
random effect for cluster i; εijl = residual; i = 1,2,…,k clusters; j = 1,2,…,T time periods; and l = 
1,2,…,m individuals.83, 84  In addition, we may consider adjusting the analyses to deal with lags 
in the intervention effect.  This delayed intervention effect occurs if the intervention does not 
become fully effective during the step in which it is introduced.85  If, for example, we expect the 
lag in the intervention effect to be 50% in the implementation step and 100% effective within 6 
months, we could follow the suggestion of Hussey and Hughes of using fractional values for the 
treatment indicator.83  
 
Because this is a cross-sectional design in which different patients are sampled at each step in 
the study, we do not need to include a random effects term for repeated measures on the same 
individuals. The effect size adjusted for calendar time and its 95% confidence interval will be 
calculated to assess the estimated change in outcomes after introduction of the intervention. 
 
Although the primary analysis will be intention-to-treat, if necessary, we will include an analysis 
of what actually happened if the CBOCs do not crossover to the intervention at the original 
specified time.    
 
Per Hemming et al.,81 we will report the estimated intra-cluster correlation for use in design of 
future trials.  In addition, we will report the time effect from the fitted model to allow assessment 
of possible confounding effects of calendar time.  We may include an effect modifier term in the 
model representing the length of the period up to the current observation during which the 
CBOC has been exposed to the intervention.  This will allow examination of the way the impact 
of the Telemedicine Based care develops over time once introduced into the CBOC. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
24 
 
Power Calculation for Hypothesis 2. We calculated the sample size for the stepped wedge 
design following the approach of Woertman et al.,86  which has been corrected by Hemming, 
Girling, and Taljaard.84, 87, 88  
We first determined the sample size needed if individuals were randomized (Nu).  This is then 
multiplied by a design effect, DEsw, to correct for both clustering and the stepped wedge design.  
DEsw = 1 + ρ(tm + m – 1)       *   3(1 – ρ)  
            1 + ρ(0.5tm + m – 1)     2 (t – 1/t) 
where m is the number of subjects within a cluster in each measurement period, ρ is the intra-
cluster correlation (ICC), t is the number of randomization steps, and k clusters are measured at 
each of T measurement periods.  The required sample size for analysis using the stepped 
wedge design is then Nsw = Nu*DEsw*T.  Finally, we adjusted the required sample size formula to 
account for 20% attrition over the 6-month period to obtain the number of patients that need to 
be selected for assessment, N. 
In our study, the 6 CBOCs will be randomly allocated two at a time to one of the t = 3 
randomization steps.  Based on earlier work, the intracluster correlation coefficient, ρ will be 
estimated as .01.  There will be a total of 5 measurements for each cluster, including the control 
period(s), implementation, and maintenance period(s) (see schematic diagram).  We assumed 
for the power calculations that approximately m=18 participants would be available within each 
CBOC in each 6-month time interval. 
The calculations are based on having 80% power at a significance level of 0.05 to detect a 
medium effect size (Cohen’s d=0.42) between the usual care referral model (EU) and the 
Telemedicine Blended model intervention group (TB) in our primary outcome measure for the 
effectiveness hypothesis, MCS from SF-12V.  This represents a difference of 4 points in the 
MCS scores between the referral model group and the Telemedicine Blended Model group, 
assuming a pooled standard deviation of 10.  This gives the unadjusted total sample size, Nu, of 
128 for the two-sided, two-sample t-test of equal means. 
The total number of patients to be selected for assessment across the 6 CBOCs will be 750.  
This was determined after adjusting the total sample size required under individual 
randomization Nu, for the design effect, multiplying by the 5 measurement periods, and 
accounting for a 20% attrition rate. This will result in approximately 600 patients providing 
baseline and 6-month follow-up data for analysis, accounting for attrition.     
 
The assumption of approximately 18-30 participants at each CBOC in each 6-month period 
does not represent the maximum that can be recruited at a site in any six-month period; it is a 
possible average based on our experience in the first six-months of recruitment.  We will recruit 
all eligible participants at the study sites as described elsewhere in the protocol.   
 
Direct Access to Source Data/Documents: 
Only authorized persons will have access to the information gathered in this study.  
Authorized persons may include regulatory agencies such as, the Government Accounting 
Office (GAO) or the Office for Human Research Protection (OHRP), Office of Research 
Oversight (ORO), as well as members of the Research Administration staff of CAVHS. 
Data Safety and Monitoring Plan: 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
25 
Although unlikely and unanticipated, a possible risk for this study is loss of confidentiality.  
All data will be managed to minimize the risk of loss of confidentiality.  Consent forms and raw 
data forms will be stored in locked file cabinets in locked offices, or will be stored on the 
password protected VA computer network as appropriate.  Study staff will only have access to 
the data they need for their work.  Participants' names and contact information will not be 
included in the main study database nor in any datasets used for analysis.  Instead, participants 
will be identified by study ID code only.  A table linking participants' names with their study IDs 
will be stored in a separate database on the server.  Only investigators and the research staff 
(including the study programmer) will have access to the key linking participants' names and 
study ID codes.  The identifiers will be destroyed as early as possible as allowed by VA policies.  
All study databases will be stored on the secure VA network, not on local hard drives.  
Specifically, data will be stored on the HSR&D Service Drive at 
\\VHAV16FPC42.V16.MED.VA.GOV\PI Project Folders\Owen\Adapting and Implementing the 
Blended Collaborative Care Model in CBOCs. Data will also be stored on VA VINCI. The study 
protocol will be submitted to the CAVHS IRB and R&D Committee for approval before research 
begins. 
Ethics: 
This study will comply with principles of the Belmont Report.  Ethical considerations specific 
to this study include the potential for loss of confidentiality.  To avoid loss of confidentiality in 
any publication or other communication based on this data, no information specific enough to 
identify a participant will be included. 
Data Handling and Recording: 
Data will consist of questionnaires filled out by participants, qualitative data from interviews, 
and data extracted from the CDW.  All data, other than CDW data extracts and required 
electronic enrollment logs, will be de-identified according to VHA policy for truly de-identifying 
data.  All paper materials, including consent forms, will be stored in CAVHS building 58, rooms 
105A or 254 (the private offices of the study coordinator and the PI) in a locked filing cabinet.  
Databases consisting of the data collected will be maintained in files on the password protected, 
secure VA HSR&D server \\VHAV16FPC42.V16.MED.VA.GOV (see above folder location) and 
VA VINCI.  Removal of study access will be completed for research personnel when they are no 
longer part of the research team.  Records will be retained indefinitely until the VA RCS has 
been revised regarding disposition of research records.  VA approved methods will be utilized to 
destroy data when permitted. 
If, in spite of the implemented precautions, participant data is lost, stolen, or compromised, it 
will be reported immediately to the PI.  The PI or designee will report the event to the IRB, the 
ISO, the PO, and the ACOS/R&D as soon as possible, but no longer than 1 hour after learning 
of the breach of confidentiality. 
 
 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
26 
References: 
 
 1  Williams JW, Jr., Gerrity M, Holsinger T, Dobscha S, Gaynes B, Dietrich A. Systematic 
review of multifaceted interventions to improve depression care. General Hospital 
Psychiatry. 2007; 29(2):91-116. 
 2  Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, Bystritsky A, Welch 
SS, Chavira DA, Golinelli D, Campbell-Sills L, Sherbourne CD, Stein MB. Delivery of 
Evidence-Based Treatment for Multiple Anxiety Disorders in Primary Care: A Randomized 
Controlled Trial. J Am Med Assoc. 2010; 303(19):1921-1928. 
 3  Zivin K, Pfeiffer PN, Szymanski BR, Valenstein M, Post EP, Miller EM, McCarthy JF. 
Initiation of Primary Care-Mental Health Integration Programs in the VA Health System: 
Associations with Psychiatric Diagnoses in Primary Care. Med Care. 2010; 48(9):843-851. 
 4   About the Office of Rural Health. http://www.ruralhealth.va.gov/about_orh.asp.  
 5  Maciejewski ML, Perkins M, Li YF, Chapko M, Fortney JC, Liu CF. Utilization and 
expenditures of Veterans obtaining primary care in community clinics and VA medical 
centers: An observational cohort study. BMC Health Services Research. 2007; 7:56. 
 6  Liu CF, Chapko MK, Perkins MW, Fortney J, Maciejewski ML. The impact of contract 
primary care on health care expenditures and quality of care. Medical Care Research and 
Review. 2008; 65(3):300-314. 
 7  Wallace AE, Weeks WB, Wang S, Lee AF, Kazis LE. Rural and Urban Disparities in 
Health-Related Quality of Life Among Veterans with Psychiatric Disorders. Psychiatr Serv. 
2006; 57(6):851-856. 
 8  West AN, Weeks WB. Health care expenditures for urban and rural veterans in Veterans 
Health Administration care. Health Services Research. 2009; 44 (5 Part 1):1718-1734. 
 9  West A, Weeks WB. Physical and mental health and access to care among 
nonmetropolitan Veterans Health Administration patients younger than 65 years. Journal 
of Rural Health. 2006; 22:9-16. 
 10  McCarthy JF, Blow FC, Ignacio RV, Ilgen MA, Austin KL, Valenstein M. Suicide among 
patients in the Veterans Affairs health system: Rural-urban differences in rates, risks, and 
methods. American Journal of Public Health. 2012; 102 (Suppl 1):S111-S117. 
 11  Zeiss AM, Karlin BE. Integrating mental health and primary care services in the 
Department of Veterans Affairs health care system. J Clin Psychol Med Settings. 2008; 
15(1):73-78. 
 12  Department of Veterans Affairs: Veterans Health Administration. VHA Handbook 1160.01:  
Uniform Mental Health Services in VA Medical Centers and Clinics. Washington, D.C.: 
2008. 
 13  Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the Chronic Care Model in the 
new millennium. Health Affairs. 2009; 28:75-85. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
27 
 14  Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. 
Milbank Quarterly. 1996; 74(4):511-544. 
 15  Fortney J, Pyne JM, Edlund MJ, Williams DK, Robinson DE, Mittal D, Henderson KL. A 
Randomized Trial of Telemedicine-Based Collaborative Care for Depression. J Gen Intern 
Med. 2007; 22(8):1086-1093. 
 16  Fortney JC, Pyne JM, Edlund MJ, Robinson DE, Mittal D, Henderson KL. Design and 
implementation of the Telemedicine-Enhanced Antidepressant Management Study. Gen 
Hosp Psychiatry. 2006; 28(1):18-26. 
 17  Fortney JC, Pyne JM, Smith JL, Curran GM, Otero JM, Enderle MA, McDougall S. Steps 
for implementing collaborative care programs for depression. Population Health 
Management. 2009; 12(2):69-79. 
 18  Fortney JC, Pyne JM, Steven CA, Williams JS, Hedrick RG, Lunsford AK, Raney WN, 
Ackerman BA, Bonner LM, Smith JL. A web-based clinical decision support system for 
depression care management. American Journal of Managed Care. 2010; 16(11):849-854. 
 19  Fortney J, Enderle M, McDougall S, Clothier J, Otero J, Altman L, Curran G. 
Implementation outcomes of evidence-based quality improvement for depression in VA 
community based outpatient clinics. Implementation Science. 2012; 7:30. 
 20  Collins, C., Hewson, D. L., Munger, R., and Wade, T. Evolving Models of Behavioral 
Health Integration in Primary Care. Milbank Report available at 
www.milbank.org/uploads/documents/10430EvolvingCare/EvolvingCare.pdf.  Last 
accessed 4/25/12.  
 21  Robinson PJ, Reiter JT. An overview of primary care behavioral health consultation. In: 
Robinson PJ, Reiter JT, editors. Behavioral Consultation and Primary Care: A Guide to 
Integrating Services  New York: Springer, 2007: 3-16. 
 22  Engel AG, Malta LS, Davies CA, Baker MM. Clinical Effectiveness of Using an Integrated 
model to Treat Depressive Symptoms in Veterans Affairs Primary Care Clinics and Its 
Impact on Health Care Utilization. The Primary Care Companion for CNS Disorders. 2011; 
13(4):PCC.10m01096. 
 23  Pomerantz AS, Shiner B, Watts BV, Detzer MJ, Kutter C, Street B, Scott D. The White 
River model of colocated collaborative care: A platform for mental and behavioral health 
care in the medical home. Families, Systems, and Health. 2010; 28(2):114-129. 
 24  Shiner B, Watts BV, Pomerantz A, Groft A, Scott D, Street B, Young-Xu Y. Access to 
what?  An evaluation of the key ingredients to effective advanced mental health access at 
a VA medical center and its affiliated community-based outreach clinics. Military Medicine. 
2009; 174(10):1024-1032. 
 25  Watts BV, Shiner B, Pomerantz A, Stender P, Weeks WB. Outcomes of a quality 
improvement project integrating mental health into primary care. Quality and Safety in 
Health Care. 2007; 16(5):378-381. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
28 
 26  Pomerantz A, Cole BH, Watts BV, Weeks WB. Improving efficiency and access to mental 
health care: Combining integrated care and advanced access. General Hospital 
Psychiatry. 2008; 30(6):546-551. 
 27  Bryan CJ, Morrow C, Appolonio KK. Impact of behavioral health consultant interventions 
on patient symptoms and functioning in an integrated family medicine clinic. Journal of 
Clinical Psychology. 2009; 65(3):281-293. 
 28  Cigrang JA, Dobmeyer AC, Becknell ME, Roa-Navarerete RA, Yerian SR. Evaluation of a 
collaborative mental health program in primary care: Effects on patient distress and health 
care utilization. Primary Care and Community Psychiatry. 2006; 11(3):121-127. 
 29  Begley CE, Hickey JS, Ostermeyer B, Teske LA, Vu T, Wolf J, Kunik ME, Rowan PJ. 
Integrating behavioral health and primary care: The Harris County Community Behavioral 
Health Program. Psychiatric Services. 2008; 59(4):356-358. 
 30  Dominitz JA, Boyko EJ, Koepsell TD, Heagerty PJ, Maynard C, Sporleder JL, Stenhouse 
A, Kling MA, Hrushesky W, Zeilman C, Sontag S, Shah N, Ona F, Anand B, Subik M, 
Imperiale TF, Nakhle S, Ho SB, Bini EJ, Lockhart B, Ahmad J, Sasaki A, van der Linden B, 
Toro D, Martinez-Souss J, Huilgol V, Eisen S, Young KA. Elevated prevalence of hepatitis 
C infection in users of United States veterans medical centers. Hepatology. 2005; 
41(1):88-96. 
 31  Centers for Disease Control and Prevention. HIV/AIDS and Viral Hepatitis. Population and 
Settings. Available at : http://www.cdc.gov/hepatitis/Populations/HIV.htm Last accessed: 
6/28/2016.  
 32  Brau N, Bini EJ, Shahidi A, Aytaman A, Xiao P, Stancic S, Eng R, Brown ST, Paronetto F. 
Prevalence of hepatitis C and coinfection with HIV among United States veterans in the 
New York City metropolitan area. American Journal of Gastroenterology. 2002; 97:2071-
2078. 
 33  Fireman M, Indest DW, Blackwell A, Whitehead AJ, Hauser P. Addressing Tri-Morbidity 
(Hepatitis C, Psychiatric Disorders, and Substance Use): The Importance of Routine 
Mental Health Screening as a Component of a Comanagement Model of Care. Clinical 
Infectious Diseases. 2005; 40:S286-S291. 
 34  Rabkin JG. HIV and depression: 2008 review and update. Current HIV/AIDS Reports. 
2008; 5:163-171. 
 35  Peterson K, McCleery E, Waldrip K. Evidence Brief: Update on prevalence of and 
interventions to reduce racial and ethnic disparities within the VA. Report No.: #09-199. 
2014.  
 36  Department of Veterans Affairs National Heptitis C Resource Center Program, National 
Viral Hepatitis Program, Office of Patient Care Services. Chronic Hepatitis C Virus (HCV) 
Infection: Treatment Considerations. Available at: http:www.hepatitis.va.gov/pdf/treatment-
considerations-2016-03-28.pdf 
Last accessed 6-28-16.  2016.  

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
29 
 37  McGowan CE, Fried MW. Barriers to hepatitis C treatment. Liver International. 2012; 
32:151-156. 
 38  Pyne JM, Fortney JC, Curran GM, Tripathi S, Atkinson JH, Kilbourne AM, Hagedorn HJ, 
Rimland D, Rodriguez-Barradas MC, Monson T, Bottonari KA, Asch SM, Gifford AL. 
Effectiveness of collaborative care for depression in human immunodeficiency virus 
clinics. Archives of Internal Medicine. 2011; 171(1):23-31. 
 39  Painter JT, Fortney JC, Gifford AL, Rimland D, Monson T, Rodriguez-Barradas MC, Pyne 
JM. Cost-effectiveness of Collaborative Care for Depression in Human Immunodeficiency 
Virus Clinics. Journal of Acquired Immune Deficiency Syndrome. 2015. 
 40  Minick SG, Stafford CL, Kertz BL, Cully JA, Stanley MA, Davila JA, Dang BN, Rodriguez-
Barradas MC, Giordano TP. Veterans' perspectives on interventions to improve retention 
in HIV care. PLoS One. 2016; 11:e0148163. 
 41  Ohl ME, Richardson K, Kaboli PJ, Perencevich EN, Vaughan-Sarrazin M. Geographic 
access and use of infectious diseases specialty and general primary care services by 
veterans with HIV infection: implications for telehealth and shared care programs. Journal 
of Rural Health. 2014; 30:412-421. 
 42  Curran GM, Pyne J, Fortney JC, Gifford A, Asch SM, Rimland D, Rodriquez-Barradas M, 
Monson TP, Kilbourne AM, Hagedorn H, Atkinson JH. Development and Implementation 
of Collaborative Care for Depression in HIV Clinics. AIDS Care. 2011; 23(12):1626-1636. 
 43  Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid 
designs: Combining elements of clinical effectiveness and implementation research to 
enhance public health impact. Medical Care. 2012; 50(3):217-226. 
 44  Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, Tunis S, 
Bergel E, Harvey I, Magid DJ, Chalkidou K. A pragmatic-explanatory continuum indicator 
summary (PRECIS): A tool to help trial designers. Journal of Clinical Epidemiology. 2009; 
62(5):464-475. 
 45  Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based 
practice: A conceptual framework. Quality in Health Care. 1998; 7(3):149-158. 
 46  Greenhalgh T. Intuition and evidence--uneasy bedfellows? British Journal of General 
Practice. 2002; 52:395-400. 
 47  Haynes RB. What kind of evidence is it that Evidence-Based Medicine advocates want 
health care providers and consumers to pay attention to? BMC Health Services Research. 
2002; 2:3. 
 48  Glisson C, Green P. The effects of organizational culture and climate on the access to 
mental health care in child welfare and juvenile justice systems. Administration and Policy 
in Mental Health and Mental Health Services Research. 2006; 33(4):433-448. 
 49  Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion 
interventions: The RE-AIM framework. American Journal of Public Health. 1999; 
89(9):1322-1327. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
30 
 50  Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating 
interventions: What can it tell us about approaches to chronic illness management? 
Patient Education & Counseling. 2001; 44(2):119-127. 
 51  Glasgow RE. Translating research to practice: lessons learned, areas for improvement, 
and future directions. Diabetes Care. 2003; 26(8):2451-2456. 
 52  Glasgow RE, Lictenstein E, Marcus AC. Why don't we see more translation of health 
promotion research to practice? Rethinking the efficacy-to-effectiveness transition. 
American Journal of Public Health. 2003; 93(8):1261-1267. 
 53  Zatzick DF, Koepsell T, Rivara FP. Using target population specification, effect size, and 
reach to estimate and compare the population impact of two PTSD preventive 
interventions. Psychiatry. 2009; 72(4):346-359. 
 54  Fortney JC, Maciejewski M, Tripathi S, Deen TL, Pyne JM. A budget impact analysis of 
telemedicine-based collaborative care for depression. Medical Care. 2011; 49(9):872-880. 
 55  West AN, Lee RE, Shambaugh-Miller MD, Bair BD, Mueller KJ, Lilly RS, Kaboli PJ, 
Hawthorne K. Defining "rural" for veterans' health care planning. J Rural Health. 2010; 
26:301-309. 
 56  Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis 
Methods. Second Edition ed. London: Sage Publications; 2002. 
 57  Gibbons RD, Hedeker D. Application of random-effects probit regression models. Journal 
of Consulting and Clinical Psychology. 1994; 62(2):285-296. 
 58  Hedeker D, Gibbons RD, Davis JM. Random regression models for multicenter clinical 
trials data. Psychopharmacology Bulletin. 1991; 27(1):73-77. 
 59  Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, Koffman RL. Combat Duty in 
Iraq and Afghanistan, Mental Health Problems, and Barriers to Care. N Engl J Med. 2004; 
351(1):13-22. 
 60  Rollnick S, Mason P, Butler C. Health Behavior Change: A Guide for Practitioners. New 
York: Churchill Livingstone; 1999. 
 61  Morisky DE, Ang A, Krousel-Wood M, Ward H. Predictive validity of a medication 
adherence measure for hypertension control. Journal of Clinical Hypertension. 2008; 
10(5):348-354. 
 62  Kazis LE, Miller DR, Skinner KM, Lee A, Ren XS, Clark JA, Rogers WH, Spiro A, III, Selim 
A, Linzer M, Payne SM, Mansell D, Fincke RG. Patient-reported measures of health: The 
Veterans Health Study. Journal of Ambulatory Care Management. 2005; 27(1):70-83. 
 63  Jenkins CD, Stanton BA, Niemcryk SJ, Rose RM. A scale for the estimation of sleep 
problems in clinical research. Journal of Clinical Epidemiology. 1988; 41(4):313-321. 

Richard R. Owen, MD, PI 
Adapting and Implementing the Blended  
Collaborative Care Model in CBOCs 
TMH-PCMHI Protocol, v9 
April 17, 2018 
31 
 64  Rifas-Shinman SL, Willett WC, Lobb R, Kotch J, Dart C, Gillman MW. PrimeScreen, A 
brief dietary screening tool: Reproducibility and comparability with both a longer food 
frequency questionnaire and biomarkers. Public Health Nutrition. 2001; 4(2):249-254. 
 65  Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity 
measure. Journal of General Internal Medicine. 2001; 16(9):606-613. 
 66  Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized 
anxiety disorder: the GAD-7. Arch Intern Med. 2006; 166:1092-1097. 
 67  Shear MK, Brown TA, Barlow DH, Money R, Sholomskas DE, Woods SW, Gorman JM, 
Papp LA. Multicenter collaborative panic disorder severity scale. American Journal of 
Psychiatry. 1997; 154:1571-1575. 
 68  Blanchard EB, Jones-Alexander J, Buckley TC, Forneris CA. Psychometric properties of 
the PTSD Checklist (PCL). Behaviour Research and Therapy. 1996; 34(8):669-673. 
 69  Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol 
consumption questions (AUDIT-C): An effective brief screening test for problem drinking. 
Archives of Internal Medicine. 1998; 158:1789-1795. 
 70  Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient 
satisfaction: Development of a general scale. Eval Prog Plann. 1979; 2(3):197-207.