An eHealth Intervention to Increase Depression Treatment Initiation and Adherence Among Veterans Referred for Mental Health Services (CDA 18-189)
Phase: NA
Conditions: Depression
Sponsor: VA Office of Research and Development

$QH+HDOWKLQWHUYHQWLRQWRLQFUHDVHGHSUHVVLRQ
WUHDWPHQWLQLWLDWLRQDQGDGKHUHQFHDPRQJYHWHUDQV
UHIHUUHGIRUPHQWDOKHDOWKVHUYLFHV&'$
1&7
$SULO

A1. BACKGROUND AND SIGNIFICANCE 
Depression and Veterans’ health, functioning, and quality of life. Depression prevalence among OEF/OIF 
Veterans is high (e.g., 59% screened positive; Seal et al, 2008), with increasing rates among this cohort of 
Veterans over the past few decades (Seal, 2009), making depression the most prevalent mental health concern 
in the VA. Although some recover quickly, the majority experience recurrent and often chronic depression 
(Whiteford, 2013). By the third episode, a person with a history of depression is 90% likely to experience another 
episode, highlighting the critical role of early intervention and care. Depression is a vastly heterogeneous 
syndrome (Monroe & Harkness, 2012) primarily characterized by low mood and anhedonia. Among Veterans, 
depression is a comorbid condition that complicates the course of PTSD and TBI recovery. Depressed people 
are 30 times more likely to commit suicide than are healthy individuals (Joiner, 2010) and 5 times more likely to 
abuse substances. The prevalence of depression and its associated morbidity and mortality make care for 
depression a high priority in the VA (DVA/DoD, 2016).  
Depression care in the VA. Cognitive behavioral psychotherapies and psychotropic medications are 
recommended by VA practice guidelines and are available as first line treatments for depression, yet Veterans 
often do not engage (i.e., initiate, adhere) in offered care. Despite great efforts to promote evidence-based 
treatments (EBTs) as the gold standard of care for depression and to increase access to mental health care in 
the VA (Mott, 2014), any mental health care utilization among veterans continues to be low (i.e., 18.5-27%; 
Burnam, 2009; Mott, 2014). Low utilization of care highlights a major gap in the guidelines for depression care 
in the VHA. VA clinical practice guidelines for the management of major depressive disorder (MDD; VHA/DoD, 
V3, 2016) omits any discussion of the management of patients who require clinical care but disengage. Given 
the sequelae of untreated depression, it is paramount to both identify those at risk for dropping out of care and 
develop strategies for management of patients that prematurely disengage from traditional care.  
Barriers to care. To better understand patterns in access to care, Levesque’s framework proposes that patient 
characteristics interfere with successful treatment utilization alongside system barriers (see Levesque et al, 
2013). Relevant patient characteristics that ultimately lead to care access are ability to perceive, seek, reach, 
pay, and, finally, engage in care (see Figure 1). These abilities map onto a process that incorporates identifying 
health care needs, perception of needs and desire for care, healthcare seeking and reaching, health care 
utilization, which increasingly lead to health care consequences.  
Empirical research has shown consistently that lack of perceived need for treatment is a major barrier to 
care. Level of illness severity may impact perceived need; in one study, low perceived need was more often a 
reason for not seeking treatment among individuals with mild (57.0%) or moderate (39.3%) than severe (25.9%) 
disorders (Mojtabai et al, 2011). Unfortunately, depression is related to negative health outcomes and poor 
functioning across severity levels, even when subthreshold symptoms are present (Boulenger, 2004).  When 
need for treatment is present, emotional and evaluative barriers become more powerful in interfering with 
treatment engagement (Mojtabai et al, 2011). In depression especially, withdrawal behaviors likely stem from 
low emotional awareness which is often linked to maladaptive coping and low self-efficacy (Good, Dell, & Mintz, 
1989; Schwarz & Clore, 1983, 1996). Military training may also impact future treatment seeking and adherence 
by instilling emotional control (Nash, Silva, & Litz, 2009) and valuing the ability to control emotions under stress 
to promote survival and mission completion. Beliefs that promote emotional avoidance and self-reliance may 
inadvertently delay treatment seeking (Mackenzie, Gekoski, & Knox, 2006; MacKenzie, Knox, Gekoski, & 
Macaulay, 2004). Despite poor awareness and avoidance contributing to depression treatment 
engagement, few interventions have targeted these mechanisms to improve care. Given the low care 
utilization, pretreatment interventions may help patients to understand their health concerns and 
facilitate engagement by providing tools for treatment planning. Pre-treatment interventions could also 
include practical and emotional support to further drive engagement. In a recent meta-analysis, both practical 
and emotional support played roles in treatment engagement (DiMatteo, 2004).   
Preliminary step to overcome treatment barriers: Patient phenotypes associated with treatment 
utilization. Despite evidence that various patient factors represent treatment barriers, we continue to lack a 
reliable and valid method for identifying patients at risk for poor engagement in mental health care. An emerging 
method for more precisely aligning treatments with patient needs involves identifying subgroups, or “phenotypes” 
that describe patients with unique characteristics within a diagnostic category that are predictive of successful 
treatment engagement. This process is aligned with current initiatives to provide patient-centered health care. 
Precision Medicine refers to the tailoring of services based on classification of patients based on their risk, 
prognosis, and response. Understanding treatment initiation and adherence in this framework is at the core of 
patient centered care in the VHA. To start addressing these gaps, our preliminary work has capitalized on the 
availability of patient information in medical records where we have started to investigate symptom trajectories 

and profiles (Panaite…Luther, resubmitted for review) (Appendix 4). These analyses are starting to build 
phenotypes of patients that may be at highest need and most likely to prematurely disengage from care and in 
need of a support system at the start of mental health care.  
The use of analytics in understanding patient phenotypes. Health care has lagged behind other industries 
in the use of advanced analytics to drive performance. The VHA’s EHR provides a unique opportunity for efficient, 
large-scale investigation of potential clinical markers or patient phenotypes. Ideally, such studies will employ  
advanced analytics like machine learning to define relevant constructs. The application of analytics tools has 
enabled accurate and efficient determination of longitudinal outcomes, enabling existing EHR data to be applied 
to clinical research. Machine learning, for example, was used to predict suicidal ideation among adults recently 
discharged from psychiatric inpatient or emergency room settings (Cook et al, 2016). Preliminary work has been 
conducted to improve methods of estimating use of EBP. Shiner and colleagues (2012) evaluated administrative 
data and text notes for patients newly enrolling in six VHA outpatient PTSD clinics in New England during the 
2010 fiscal year. They developed machine learning algorithms that mimic human raters in classifying note text 
and found that 6.3% of their study population received at least one session of evidence-based psychotherapy 
during the initial 6 months of treatment. (Shiner et al, 2012). The current CDA will build on prior work by 
using machine learning to define Veteran patient phenotypes associated with risk for not engaging in 
evidence-based care when the EHR indicates evidence of probable depression.   
Overcoming treatment barriers through self-monitoring. Prior work to overcome treatment barriers focused 
on improving patient motivation, however, attempts at increasing treatment initiation via single session 
motivational interviewing strategies have been largely unsuccessful. Although depression is linked to 
motivational deficits, the mechanism by which care is interrupted may be diverse based on patients’ needs and 
characteristics. The literature suggests variability in perceived need and cognitive and emotional barriers once 
need is identified. Furthermore, the pre-treatment process is dynamic, likely involving various barriers depending 
on time lag between referral and treatment onset, ultimately limiting self-efficacy and ability to engage in care 
(Levesque et al, 2013).  
Results from large national trials like the Sequenced Treatment Alternatives to Relieve Depression 
(STAR*D) Study found that management of depression was facilitated by active monitoring of symptoms (Rush 
et al, 2004). This principle has been a key component to integrated care treatment trials including Re-Engineering 
Systems of Primary Care for PTSD and Depression in the Military (RESPECT-Mil), Behavioral Health Laboratory 
(BHL), and Translating Initiatives in Depression into Effective Solution (TIDES), the main VA and DoD programs 
(Rush et al, 2004; Tew et al, 2010; Engel et al, 2008; Rubenstein et al, 2010). Systematic symptom 
measurement through self-monitoring may give patients a new role in their care (Valenstein et al, 2009). 
For example, Guo et al. (2015) conducted an RCT in which patients were either monitored using a DSM checklist 
or not monitored. Not only did the monitored group have greater improvement in symptoms, there was evidence 
for better treatment management (Guo et al, 2015). Patients who regularly complete standardized assessments 
may become knowledgeable about their symptoms and depression and become comfortable assessing their 
own symptom trajectory, all key aspects of disease self-management. Symptom reports also have been an 
integral part of CBT for depression. A growing body of evidence supports the benefits of regular monitoring with 
feedback to patients to improve outcomes. For example, both theory and research suggest that mood labeling 
(the ability to identify and categorize one’s moods) rather than ambiguous monitoring (a tendency to scrutinize 
and focus on one’s moods) has value in the self-regulation of behavior (Swinkels & Giuliano, 1995). It follows 
that self-management strategies may help both non-engaged and newly referred patients: non-engaging patients 
understand their need for care and may later elect to initiate treatment, while newly referred patients experience 
increased ability to engage. Active engagement in care increases a sense of self-efficacy. Furthermore, data 
that are obtained from the period prior to entering treatment can help patients readily guide treatment planning.  
eHealth platforms have the potential to provide dynamic pre-treatment support. eHealth/mHealth 
interventions are tools or treatments, typically behaviorally based, that are operationalized and transformed for 
delivery via the internet or mobile platforms (Ritterband et al, 2009). In the VA smart phone-based tools and 
interventions have been explored as a potential adjunct to therapy, given their ease of use in any context in 
helping with treatment initiation, adherence, and monitoring for chronic illness. In fact, patients in all age groups 
have indicated greater than 50% interest in using a mobile application daily to monitor their mental health 
conditions (Torous et al, 2014). Even among homeless Veterans, 89% had a mobile phone (one-third were 
smartphones), and 76% used the Internet. Among those with a mobile phone, 71% used text messaging. Nearly 
all respondents (93%) were interested in receiving mobile phone reminders about upcoming medical 
appointments, and a similar proportion (88%) wanted mobile phone outreach asking if they would like to schedule 
an appointment if they had not been seen by a health provider in over a year (McInnes, et al., 2014). Mobile 

capabilities may significantly improve patient compliance with treatment (Reger et al, 2013). Symptom monitoring 
during the treatment process can also be significantly enhanced via mobile applications and adherence to real-
time self-monitoring may be enhanced when conducted via mobile phone or a similar handheld device. Many 
patients who complete paper self-monitoring, do so retrospectively (Matthews, Doherty, Coyle, & Sharry, 2008), 
and this process is more prone to error (Shiffman, Stone, & Hufford, 2008). Conversely, mobile phone monitoring 
enables interventionists to see precisely when the ratings were taken. Daily ambulatory monitoring of symptoms 
using smartphone software applications was also feasible and valid way of assessing psychotic phenomena 
(Palmier-Claus et al, 2012), suggesting this as a viable method for even the most severe patients. We propose 
to leverage an eHealth platform to provide a dynamic pre-treatment mechanism to help patients stay 
engaged with prescribed mental health care for depression through self-monitoring and emotional and 
practical support.  
Gaps Addressed. This CDA will address multiple gaps in our understanding of how to optimize depression care. 
Although studies have examined barriers to mental health treatment at a population level, this is the first study 
to apply machine learning algorithms to a large body of data in order to identify discrete individual patient 
phenotypes associated with poor treatment engagement (Aim 1). Second, despite past efforts to identify barriers 
to treatment, little work has focused on trying to tailor interventions to address barriers to treatment proactively 
among individuals most at risk of failing to engage. This is the first attempt at designing an intervention that uses 
individualized feedback (i.e., measurement based care)—shown to help both with treatment adherence and 
efficacy—to create a patient level support system from the moment of referral to mental health care to the 
initiation of treatment (Aims 2 and 3)—a period that is known to be associated with disengagement.    
A2. CONCEPTUAL MODEL: PATIENT-CENTERED ACCESS TO MENTAL HEALTH CARE 
Using 
both 
theory 
and 
empirical 
evidence, the focus of this CDA will be 
to: (1) identify risk profiles associated 
with depression treatment engagement, 
to test prediction models and select 
patients for the proposed intervention; 
(2) design an eHealth intervention 
program using technology driven self-
monitoring; and (3) conduct a formative 
evaluation 
and 
pilot 
this 
eHealth 
intervention.  
First, we will identify possible patient 
profiles or phenotypes associated with 
treatment engagement. Profiles will be 
constructed using structured data and 
will be used to calculate risk scores 
associated 
with 
patient 
lack 
of 
engagement 
in 
treatment 
for 
depression 
(Aim 
1). 
Guiding 
the 
development 
of 
a 
pre-treatment 
intervention (Aim 2) will be Levesque’s 
Framework 
which 
proposes 
that 
patient’s ability to engage is at the core of successful health care consequences (i.e., outcomes). Therefore, the 
focus will be on the development of an intervention that will provide patients the mechanism to initiate treatment, 
and then ideally remain actively engaged with their own care, which prior work has shown to increase adherence 
and improve treatment outcomes (e.g., Guo et. al, 2015). This will be done via active self-monitoring and specific 
labeling of depression symptoms, as well as via emotional and practical support. Based on this framework, the 
process by which behavior change about treatment is activated is by helping patients more clearly identify health 
care needs (i.e., via symptom monitoring) help rectify perception of needs (by providing feedback on symptoms, 
whether they are elevated and a provider should be contacted), helping with care seeking and reaching (i.e., 
sending provider contact information and sending reminders).  
Formative methods will be used to evaluate and pilot this process (Aim 3) in preparation for a future 
efficacy/effectiveness RCT (IIR developed in year 3). Our study focuses on solidifying engagement in care for 
those patients whose needs have been identified by provider and have been referred for care. Focusing on this 
segment of the patient population with mental health care needs ensures that our sample already meets basic 
Figure 1. Conceptual framework.  

steps in accessing care. Based on our conceptual framework we hope to systematically and progressively reach 
patients with more complex barriers to care in future work. The proposed CDA sets the ground work for such 
future studies (see Future Research Projects section).  
A3. PRELIMINARY STUDIES  
A3a. Candidate Research  
Depression care and outcomes among Veterans (2016-present): During my HSR&D postdoctoral fellowship 
I completed a series of studies to better understand depression care in the VA. (1) Dr. Belanger and I published 
a systematic review of the literature on post deployment mental health screening among OEF/OIF Veterans to 
understand the role and outcomes of screening, a service developed to enhance Veterans’ mental health 
(Panaite, …Belanger, 2018). Rates of positive depression screens were highly variable ranging between 1.9% 
and 25.4% across thirteen studies, with one outlier showing a prevalence rate of 41.8% among Veterans (Seal, 
2008). Two notable findings were: (1) Few studies reported outcomes associated with depression screening. 
Veterans who were not screened were less likely to have follow-up assessments, such as suicide assessment 
(Dobscha, 2013); and (2) Only three studies with representative samples reported referral rates associated with 
screening, which were highly variable. Despite efforts to increase access to care, we need further empirical 
support for the role of screening in access and as a facilitator in seeking mental health care – two major goals of 
the screening efforts in the VHA. Relevance to the proposed CDA: Findings highlight the need to increase 
Veterans’ engagement with mental health services in a manner that may be accessible and private, such as 
using technology. The project provided an opportunity to collaborate and co-author with mentors and develop a 
foundational understanding for the complexity involved in ascertaining accurate rates and predictors of who does 
and does not engage in treatment for depression. (2) In collaboration with my mentor, Dr. Pfeiffer, I investigated 
individual and neighborhood characteristics that predicted depression symptom response in Veterans receiving 
VA care, paper now published in Health Services Research (Appendix 4). We used EHR data from 2016 merged 
ZLWKFHQVXVWUDFWGDWDWRSUHGLFWLPSUovement in 4-8-month PHQ-9 scores in VA patients (N=4,269) with 
a unipolar depressive disorder diagnosis and a PHQ-VFRUH at baseline. Patient factors associated with 
decreased likelihood of improvement included Black race, comorbid PTSD, and disability.  Among neighborhood 
characteristics, living below the poverty line was found to be most predictive of low symptom improvement when 
adjusting for patient characteristics. An unintended finding was that only a small minority of VA patients with 
depression received a follow-up PHQ-9 assessment contrary to clinical quality guidelines (Panaite, Pfeiffer, 
2019). Relevance to the proposed CDA: These findings highlight the importance of patient information available 
in the EHR in understanding patient behaviors and outcomes, and that measurement-based care remains an 
emerging clinical practice that could be shaped to include patients who fail to engage in care. (3) With Dr. Luther, 
my primary postdoctoral project focused on analyzing data extracted from the Corporate Data Warehouse (CDW) 
accessed through the VINCI infrastructure. This work is now resubmitted for review at Health Services Research 
(Appendix 4). These data are composed of mental health (ICD9/10, PHQ9) and mental health service data over 
the course of 11 years extracted from the EHRs of over 1 million Veterans. An initial project from these data 
focused on studying the impact of depression symptom severity and trajectory on initiation of treatment for 
depression in Veterans ZLWKD3+4. In our full sample, men were 46% less likely to initiate treatment than 
women, while both an elevated initial PHQ9 score and increased severity across assessments were associated 
with a greater likelihood of initiating treatment. Relevance to the proposed CDA: Patient symptom trajectories 
appear to be associated with mental health utilization in a national representative sample of Veterans. These 
preliminary findings support the value of investigating patient phenotypes associated with use of care. Data 
extracted in preparation for this paper will be used to perform Aim 1, speeding up the initiation of the CDA.  
Daily assessments of functioning among depressed individuals (2014 - present): Starting in graduate 
school, I have become interested in daily life functioning of individuals with depression. Papers stemming from 
my dissertation and two additional studies with collaborators used data from ecological momentary assessments 
or daily diary methodology that prompted participants through their own cellphones to report on daily events, 
symptoms, or emotions throughout the day. In the three studies I have conducted or reported on, participants 
were prompted anywhere from 1 to 10 times a day for 3 to 7 days. Although there was some variability in 
assessment completion rate among depressed versus healthy individuals, generally differences were not 
statistically significant suggesting that this manner of assessment is acceptable to samples of individuals with 
psychopathology. Relevance to the proposed CDA: Completion of daily assessments is highly feasible even 
among individuals with verified depression diagnoses. Furthermore, these studies provided ample experience 
with programing daily diary protocols, recruiting patients for daily diary studies, collecting data using this 
methodology and analyzing multilevel data. My dissertation alone provided the requisite skills to conduct complex 
studies as it was a two-step study involving an experimental laboratory protocol and daily diary study.     

Phenotypes associated with depression and depression course (2009-present):  Over the course of my 
graduate training I was involved in two NIH funded studies led by my long standing mentor and consultant on 
the current CDA, Dr. Jonathan Rottenberg: 1. Vagal Fluctuation as a Predictor of Current and Future Depression; 
2. Biobehavioral Inflexibility and Risk for Juvenile Onset Depression. Subprojects that I contributed to or led, 
showed differences between current and remitted depression in how it impacts emotional reactivity (Salomon et 
al, 2013). In a secondary analysis I led on currently depressed individuals I found that patients' engagement with 
sad stimuli predicted symptomatic improvement over 30 weeks (Panaite et al, 2016). In fact, in a recent 
manuscript we discuss findings from the national MIDUS study suggesting that depressed individuals that retain 
some level of wellbeing during a depressed state are more likely to experience high functioning 10 years later 
(Rottenberg,…, Panaite, et al., 2018). My most recent manuscript shows that daily affect predicts both 
symptomatic recovery and wellbeing in adults with depression (Panaite,…Rottenberg, under review). Relevance 
to the proposed CDA: There is observable variability among depressed individuals even during a depressed 
state that predicts their functioning well into the future. My role as project manager on these NIH funded studies 
have also provided the experience of managing changing requirements, varying regulatory needs, and managing 
and training the staff necessary to conduct daily tasks. My work on MIDUS will more directly support the 
development of the CDA as we will be using these data to compute national average levels of daily depression 
and mood levels. Finally, I developed the skills to perform longitudinal analyses using multilevel data.       
Barriers to behavioral change and decreasing barriers through primary care intervention (2006-2009): 
Before starting graduate school, I contributed to a variety of studies investigating barriers to behavioral change 
among primary care patients and subsequent development of targeted interventions to facilitate health 
behaviors. This work suggested that primary care physicians were more amenable to integrating brief targeted 
interventions that did not burden care and helped facilitate behavioral change in their patients. Relevance to the 
proposed CDA: Primary care physicians may be supportive of adopting a system like that proposed in the current 
CDA. This has implications for implementation efforts in the future. This series of studies also provided 
knowledge and skills necessary to develop healthy collaborations with primary care physicians and other care 
providers in busy primary care offices, as well as intervention development skills. I was the primary statistician 
on 6 publications coming out of this work.  
A3b. Mentor/Consultant Research 
Stephen L. Luther, PhD (Primary mentor; Measurement and health services analytics, informatics 
research). Dr. Luther has been PI on multiple HSR&D funded studies using big data.  He was a Site PI of the 
VA HSR&D Consortium for Healthcare Informatics Research (CHIR). The CHIR was comprised of a multi-
disciplinary group of collaborating investigators affiliated with VA sites distributed across the US. The mission of 
the CHIR was to improve the health of Veterans through foundational and applied informatics research. The 
primary purpose of the research was to advance the effective use of unstructured text and other types of clinical 
data in the EHR. Relevance to the proposed CDA: Dr. Luther, primary mentor, will provide mentorship in the 
development and conduct of research using big data and machine learning protocols (Research/Training Aim 1) 
in collaboration with Dr. Kip (Stage 1 Aim 1) and Dr. Finch (Stage 2 Aim 1). 
Jolie Haun, PhD (Co-primary mentor; eHealth intervention development, electronic data collection, 
formative evaluation and implementation science). Dr. Haun has had a series of projects focused on 
Veterans’ and VA employees’ experiences using VA’s electronic resources (i.e. My HealtheVet, Kiosks, Mobile, 
Telehealth) to support national redesign efforts, uptake, and sustained use to support integrated proactive use 
of these tools to increase access and support care delivery, and self-care management. Dr. Haun has also had 
several projects funded and supported by HSR&D and QUERI to support electronic collection of patient-reported 
outcomes using multiple platforms (e.g. texting, secure messaging) and data management systems (e.g. 
Qualtrics). Finally, Dr. Haun is currently conducting a large multi-site RCT evaluating the use of a dyadic 
mobile/web-based intervention to help Veterans manage chronic pain and PTSD related symptoms. Relevance 
to the proposed CDA: Dr. Haun, co-primary mentor, will provide guidance in the development and problem-
solving stages of the proposed CDA with a focus on the development of Aims 2 and 3, in planning next steps for 
a merit award, and in planning for implementation activities for my program of research.  
Paul Pfeiffer, MD (Secondary mentor; Mental health innovation, services, and outcomes) 
Dr. Pfeiffer has conducted several intervention trials using mobile health technologies to improve and inform care 
for depression. For example, he developed a text-messaging intervention which sent daily text messages to 
individuals with depression for 3 months, conducted a VA pilot study of an automated depression monitoring 
program in conjunction with support from a peer specialists or family member for patients who had been 
psychiatrically hospitalized with a diagnosis of depression, conducted extensive inquiry into the quality of VA 
depression care using health system data and utilized mixed methods approaches to extend our understanding 

of the barriers and facilitators to implementation of mental health services, particularly for depression. Relevance 
to the proposed CDA: Dr. Pfeiffer, secondary mentor, will support the development of the technology driven 
intervention in collaboration with Dr. Haun, and in understanding VA systems of care related to depression 
management. He will also mentor on the development of a follow-up IIR proposing an RCT in year 3.   
Heather Belanger, PhD (Consultant; symptom reporting). Dr. Belanger’s most recent HSR&D funded grants 
was a 4-year RCT to investigate the utility of a smart phone application to reduce postconcussion symptoms and 
improve family and community participation among Veterans with mild traumatic brain injury, through symptom 
self-monitoring. Relevance to the proposed CDA: Dr. Belanger, consultant, has extensive expertise in developing 
innovative interventions including by using technology to assess symptomatology among Veterans from Iraq and 
Afghanistan at the Tampa VA. Dr. Belanger will advise in the development of the proposed technology driven 
intervention, patient recruitment and retention for the proposed CDA.  
A4. RESEARCH DESIGN AND METHODS 
Aim 1 consists of an analysis of data extracted from the EHR to identify risk profiles (scores) and determine 
which patients are unlikely to engage in care for depression. These risk scores will be used to select patients for 
a proposed intervention that will be developed in Aim 2 and subjected to a formative evaluation in Aim 3. The 
intervention is technology-based and includes automated symptom assessment with emotional and practical 
support texts to provide interim support to Veterans referred to mental health care who are awaiting their initial 
mental health treatment visit.  
A4a. Aim1.  
Study/Design overview: I will conduct a secondary data analysis of EHR data. This will be a retrospective cohort 
study to identify patient phenotypes associated with lack of treatment engagement for depression. This aim was 
developed based on empirical evidence and Levesque’s Framework (2013) (Figure 1), which proposes that 
patient level factors impact health care utilization and outcomes. Our proposed analyses will focus on data 
available from EHRs (observed variables). Based on this framework, patient level factors may impact the process 
of initiating care along a continuum of identifying health care needs, perception of needs and desire for care, 
health care seeking, and reaching. These factors will be captured through proxy measures available in the EHRs. 
For example, based on the literature reviewed in the background and my own preliminary work, identification of 
care needs, perception of needs, and desire for care are all a function of depression severity, gender, and 
ethnicity, such that higher severity, female gender, and white race tend to all be associated with higher levels of 
perception of needs and openness to mental health care. Specifically, for this study, we plan to use the 88,456 
OEF/OIF Veterans with a Patient Health Questionnaire 9 (PHQ9) >10 (indicative of probable depression) 
randomly split into sub-samples to accommodate our goals. I plan to develop Aim 1 in two related stages. (Stage 
1) First, I will develop patient profiles using conventional logistic regression analysis. In this analysis, the 
estimated 88,456 OEF/OIF Veterans with a PHQ9 >10 will be randomly split into development and validation 
sub-cohorts. Estimates extracted from the development sub-cohort will be used to compute risk scores, with 
ROC analysis performed to determine a presumed optimal cut-off score. These profiles and cutoff score will be 
evaluated for prediction accuracy on the validation sub-cohort. (Stage 2) To evaluate the results from Stage 1, 
patient profiles will then be developed and validated with machine learning paradigm on the 88,456 OEF/OIF 
Veterans with a PHQ9 >10. Use of this approach is based on the fact that while various characteristics have 
been investigated in the past in an attempt to identify patients’ likelihood to use services, we have yet to apply 
advanced machine learning techniques to the rich data available in the EHRs to define a profile of patients at 
risk to not engage in mental health care for depression. Ultimately, our goal is to identify the best fitting model in 
terms of prediction performance and interpretation, whether by logistic regression analysis or machine learning 
methods. The risk scores identified in this Aim will be utilized to recruit patients for Aims 2 and 3. While the 
models developed for Aim 1 will be based on “observed” variables only, there may be important “unobserved” 
variables we are not able to include. Methods related to this potential concern are described below in the Analysis 
Plan section. 
Table 1. List of variables extracted from electronic medical record data.  
Observed Independent Variables 
Demographics  
Age, gender, race/ethnicity, marital status, service connected disability.  
Non-psychiatric illness 
Elixhauser measure 
Depression diagnoses 
ICD-9 Depression Codes e.g., 296.2-296.36; 293.83; 309.0. 
Anxiety 
Disorders/ 
PTSD diagnoses 
ICD-9 CM codes for Anxiety and Stress-related conditions e.g.,293.84; 309.24; 309.28; 
309.81; 300.00-300.09. 
Serious Mental Illness 
diagnoses 
ICD9-CM codes for Bipolar Affective Disorder Type 1, Schizophrenia e.g., 293.81-293.83; 
296.40-296.7; 295.40-295.90; 296.56-293.83. 

Alcohol/Substance Use 
Disorder Diagnoses 
ICD9-CM codes for Alcohol, Amphetamine, Cocaine, Inhalant Use Disorders:  291.89-
304.40. 
Depression severity 
PHQ-9 items and scores 
Depression chronicity 
Time from first to last ICD9 code or time between elevated PHQ9s  
Anxiety severity  
GAD-7 items and scores 
PTSD severity 
PCL-5 items and scores 
Affective 
characteristics 
PCL-5 clusters C (emotional avoidance), D (negative mood and cognitions), E (arousal and 
reactivity) 
Somatic difficulties 
Somatic factor of the PHQ9 (e.g., fatigue, sleeping difficulties) 
Instrumental Variables 
Distance to the VA  
e.g., GIS tracking, zip code mapping 
System approachability 
Is the mandatory annual depression screen completed in the past year?; Does pt have 
PCMHI routine check in over the past two years? 
Ability to reach VA 
Has available transportation/eligible for VA transportation; full time or multiple jobs 
Family support 
Asian and Black race (as proxy) have been associated with increased social stigma of mental 
illness, including lack of family/friends support to seek mental health services 
Beliefs about care 
Gender and race have been associated beliefs about MH care 
Unobserved Variables 
Transportation 
availability 
These variables have been repeatedly identified in the literature as barriers to mental health 
treatment. We will not have this information available in the structured data.  
Employment 
Child or family care 
Outcome Measure (Dependent Variable) 
Treatment initiation 
A primary or secondary stop code showing a visit in mental health (with a psychologist or 
psychiatrist) or primary care AND an ICD9 for depression diagnosis.  
Sampling plan and data extraction and processing: 
Data source and cohort selection. We will use data from the VA Corporate Data Warehouse (CDW), a centralized 
data repository which contains electronic medical records of all Veteran patients who receive care through the 
VHA. The patient cohort (N = 88,456) used for completion of Aim 1 includes 1) OEF/OIF Veteran patients with a 
3+4, 2) who were not already in treatment for depression prior to this PHQ9 assessment, 3) were never 
diagnosed with bipolar, personality, psychotic, and pervasive developmental disorders to be consistent with the 
National Committee on Quality Assurance measure of quality of care, HEDIS (HEDIS, 2018), as well as 
substance dependence or patients who are in treatment for substance related disorders at the time of 
recruitment. The exclusion criteria are based on the fact that the presence of these disorders may result in 
divergent treatment practices compared to unipolar depression in the absence of these conditions. Our primary 
outcome, treatment initiation, is conceptualized as a combination of service encounters with specialty mental 
health (psychologists, psychiatrists) or primary care providers charted into the patients’ VHA medical records 
(excluding assessments) that were associated with any depressive disorder diagnosis. Our data are already IRB 
exempt, R&D and DART approved, and have already been extracted.  
Data processing. Data will be initially checked for out of range values and missing data where appropriate 
through plotting and descriptive analyses. Data will be transformed as needed. Logistic regression is relatively 
free of restrictions and can accommodate a mix of predictors (continuous, discrete, dichotomous) (Tabachnick 
& Fidell, 2007). With continuous predictors, the assumption is that there is a monotonic relationship (increase or 
decrease) between the predictor variable and odds of the binary outcome (i.e. engagement or lack of 
engagement in care). The functional of this linear assumption can be evaluated by use categorical indicator 
variables as well as transformations of the continuous variable, such as a quadratic form.  Multicollinearity will 
be tested using two approaches. First, we will explore correlations among IVs and exclude variables based on 
high correlations (> .65). Second, we will examine the size of the standard error (SEs) relative to the beta 
coefficient for each predictor in the model. Large SEs will be indicative of potential multicollinearity. Residuals 
will be tested to identify possible outliers.  
Analysis plan:  
Modeling risk for lack of treatment engagement. For all analyses, lack of treatment engagement is the main 
dependent variable and will be investigated using three related definitions based on the date in which a PHQ9 
VFRUHLVLGHQWLILHGLQWKH(HR. Specifically, we will examine whether treatment is initiated within 3-months, 
6-months, or 12-months from the date of the PHQ9 assessment. Independent variables of interest will be 
extracted based on proposed framework, empirical evidence, and availability in the EHR (see Table 1), including 
insight from a select number of semi structured surveys collected prior to data analysis (described below). 

Fortunately, the EHR is rich in terms of wealth of potential predictors to carry out the analyses. This work is a 
first step in the development of an automated system that can be implemented in the VHA EHR to more easily 
identify patients who may be in need but at risk for lack of engagement in mental health treatment. A direct 
logistic regression will be performed to evaluate the battery of proposed and potential factors as possible 
predictors of lack of treatment engagement. The final model will be estimated using the validation sample (half 
of our overall available sample). Regression (beta) coefficients and odds ratios will be extracted for interpretation 
and evaluation of risk to not engage in treatment in our next step.  
Computation of health service utilization risk score. Using coefficients extracted from final logistic regression 
testing the best fitting patient phenotype associated with lack of treatment engagement, we will compute a risk 
score for lack of treatment engagement. This score will be used to select high risk patients enrolled in our 
intervention development and evaluation studies (Aims 2 and 3). The logistic regression equation will be used to 
calculate the risk scores using the intercept and co-variated specific coefficients that relate to the probability of 
lack of treatment initiation.  Therefore, the equation will be based on the logistic regression model which will be 
developed in the log odds scale.   
Machine learning process. This step of the data analysis will be conducted to assess the extent to which machine 
learning methods can potentially improve upon the prediction accuracy derived from the conventional logistic 
regression analysis. As with the conventional logistic regression analysis, the cohort of 88,456 OEF/OIF 
Veterans will be used. A range of machine learning methods will be used to compare results on variable 
importance and interpretability, with the goal of providing insight for clinicians on potential risk mitigation 
strategies for veterans at risk of not engaging in treatment for depression. The methods to be compared will 
include Lasso penalized logistic regression (Tibshirani 1996), random decision forests (Liaw 2002), decision 
trees (Hastie 2009), and support vector machines (Hastie, 2001). Each approach offers potential pros and cons 
in terms of modeling complex data and in interpretation. For example, decision trees accommodate high 
dimensions, model non-linear relationships, and perform well with missing data and are highly interpretable by 
humans. Support vector machines provide automated scaling of predictor variables and support linear and 
polynomial kernels for model training. Regardless of the method used, to prevent overfitting, 10-fold cross 
validation will be employed. This process splits the data into ten separate sets and builds a model on each set 
using the other nine sets. Variance across the ten models will be examined to evaluate model stability. Model 
performance will be evaluated using the standard F-Measure which is the harmonic mean of precision and recall. 
This will be augmented with use on metrics of predictive performance including area under the curve (AUC), 
sensitivity, and specificity, with a focus on clinical interpretability. Error analysis will be employed to improve 
model performance on 10 separate samples until model performance is maximized on the test set and then 
validated on the final set. As models are being developed, pruning algorithms will be used to make the model 
more generalizable. Finally, although there are no specific guidelines or power analysis for the determination of 
the exact sample size needed for specific models, generally samples such as n > 1000 have been found to be 
adequate for machine learning. Our sample for this step of the analysis will be N = 88,456 and we will be using 
the sample to its fullest. Therefore, we propose to use 10 (n = ~ 7000) samples as training sets to test and fine-
tune our phenotype model. We will use 2 additional (n = ~9000) samples to do a final model testing and finally 
validate the model. 
Steps to control for unobserved variables. Importantly, the logistic regression and machine learning methods 
described above are based on analyses of observed and instrumental variables that will be present in the VA 
CDW for OEF/OIF veterans. While these methods are established and generally well supported, there exists the 
possibility that some important “unobserved” variables will not be in the dataset for analysis. In this circumstance, 
the prediction equations may be suboptimal not only in terms of precision (lack thereof) due to missing variables, 
but also potential incorrect attribution of observed variables as being highly predictive of the decision to engage 
or not engage in depression care when, in fact, the observed variable is directly attributed to (caused by) a 
proximal unobserved variable. 
Instrumental variable (IV) analysis is one of the methods used to: (1) counteract issues with measurement 
error in explanatory (observed) variables which result from a lack of accurate information available for analysis 
and (2) overcome the issue of omitted variables (Angrist& Krueger, 2001; Greenland, 2000). Identification of an 
appropriate “instrument” (e.g. actual travel time for the veteran from home to treatment facility) that addresses 
the potential bias from inaccurately measured and unobserved variables is often challenging. This circumstance 
in difficulty in being able to a priori identify appropriate IVs is anticipated for the present analysis. Therefore, to 
address unobserved selection bias we propose an evaluation of multiple methods prior to the initiation of data 
extraction and modeling: (1) conduct a semi-structured survey combining questions based on literature on 
environmental and system barriers and an open ended question inviting Veterans to provide feedback on any 

other barriers that were not gleaned from the literature (see Appendix 3). Such an instrument will help us identify 
possible instrumental variables and unmeasured unobserved variables. (2) If warranted, IV analysis will also be 
used to test alternative models. Semi-structured surveys will be completed by 10 patients the PI has access to 
in the primary care clinic that meet our inclusion criteria (see Table 2) and that were referred to mental health 
services but missed their appointment based on EHR. (3) Empirical results from the development and validation 
analyses (described above) will be reviewed. First, if the ROC analysis indicates suboptimal model fit (e.g. AUC 
<0.8), this alone will suggest that important variables (whether observed or unobserved) were not included in the 
models. However, assuming overall good model fit (e.g. AUC >0.8), the next step will be to assess both 
“concordant” and “discordant” results. This analysis will again be based on semi structured surveys with 10 
veterans who did or did not engage in treatment for depression.
To illustrate, each veteran in the analysis will have both an observed and predicted outcome score (i.e. 
engagement in treatment). For the “discordant” analysis, a random sample of 5 veterans who had very high 
predicted scores for engagement in treatment, yet did not engage in treatment, will be asked to complete our 
semi-structured survey. Emphasis will be on identifying the principal reason(s) for which the veteran did not 
engage in care despite having a very high predicted engagement score. Similarly, a random sample of 5 veterans 
who had very low predicted scores for engagement in treatment, yet engaged in treatment, will be asked to 
identify the reasons that influenced their decision to engage in treatment. For the “concordant” analysis, a random 
sample of 10 veterans who engaged or did not engage in treatment consistent with their predictive score will 
complete a semi-structured survey. In this circumstance, emphasis will be on comparing the extent to which rank 
ordering of the strength of the observed variables (predicted) in the prediction models actually reflected the 
treatment engagement decision making process employed by the veteran. This qualitative information will 
provide insight into to validity of the variables used in the prediction models, and again, whether important 
unobserved variables influenced the veteran’s decision for treatment engagement.
Next, I will conduct a 2-phase intervention development and formative evaluation process. Phase 1 
entails formative development using a human-centered design completed in Aim 2. Phase 2 entails formative 
evaluation using mixed-methods completed in Aim 3. The two aims will be achieved using two separate samples 
(Aim 2, n = 10; Aim 3, n = 15). Findings from Aim 1 will be used to select and recruit patients for Aims 2-3. See 
the Human Subjects section for safety concerning using an asynchronous tool to collect mental health status.
A4b. Aim 2.
Study/Design overview: This aim is phase 1 focusing on the formative development of the eHealth intervention 
using a human-centered design (see Appendix 2). Guiding the development of this Aim (and Aim 3) will be 
Levesque’s Framework which proposes that patient’s ability to engage is at the core of successful health care 
consequences (i.e., outcomes). Therefore, the focus will be on providing patients the mechanism to initiate 
treatment. This will be done via active self-monitoring and specific labeling of depression symptoms, as well as 
via emotional and practical support. Based on this framework, the process by which behavior change about 
treatment is activated is by helping patients more clearly identify health care needs (i.e., via symptom monitoring) 
help rectify perception of needs (by providing feedback on symptoms, whether they are elevated and a provider 
should be contacted), helping with care seeking and reaching (i.e., sending provider contact information and 
sending reminders).
Sample: This aim will be achieved by involving 10 stakeholders (patient n = 6 and provider n = 4). Patients will 
be selected based on risk scores generated in Aim 1 and the following inclusion/exclusion criteria. Patient and 
provider input will be used to revise intervention content. 
Initial programming 
of intervention on 
Qualtrics 
y
ng 
on
Patients & 
provider 
interviews 
(Aim 2)
g
g
y
Revise 
intervention 
content    
(Aim 3)
y
g
Deploy 
intervention 
(texts)     
(Aim 3)
Post 
intervention 
interviews 
(Aim 3)
Intervention 
revised for 
RCT IIR in 
year 3 
Table 2. Patient inclusion/exclusion criteria for Aims 2 and 3. 
Inclusion criteria: 
x
OEF/OIF Veteran: high prevalence of depression in this population and generalizability across adult ages
x
PHQ9>10; DSM-5 criteria for major depressive disorder will be used to extract an MDD diagnosis.   
x
No gender or minorities will be excluded from this study; all ages will be included. 
Exclusion criteria: 
x
Current/Past Bipolar disorder and Current Psychotic disorder: to avoid potential exacerbation from treatment for 
depression
x
Alcohol Use disorder or Substance Use disorder: Depression treatments cannot reverse the adverse effects of 
substances on mood, and therefore impact on adherence and treatment characteristics may be confounded 

Sampling plan: The target population is a clinical sample of Veterans with depression who meet the criteria in 
Table 2. Exclusion criteria are consistent with conventional depression studies. Veterans will be excluded if they 
have a medical disorder that would significantly increase their risk of depression secondary to a medical 
diagnosis or due to treatments for those medical diagnoses (e.g., cardiovascular disease; SUD) or those 
experiencing side effects with standard treatment or would require a significant adaptation of treatment (e.g., 
bipolar disorder). Veterans will also be excluded consistent with the standard of care for cognitive and/or 
behavioral treatment of depression at Tampa VA (e.g., actively suicidal patients that need intensive and 
immediate treatment). Patients will need to have a risk score associated with lack of treatment initiation as 
determined in Aim 1.  
Recruitment: The Tampa VA is an ideal site because it has a large and diverse mental health population. In 
FY17 over 2000 patients had a first visit to primary and specialty mental health care, with over half showing a 
depression diagnosis in their charts.  Veterans will be recruited using the following methods: 1) Depression 
treatment consults. Veterans referred for mental health treatment (psychotherapy or medication) through the 
mental health consult service will be informed about the proposed CDA clinical trial either by their provider or the 
study RA who will regularly monitor MH referrals on CPRS. 2) Advertisement. IRB-approved brochures will be 
placed throughout Tampa VA (e.g., primary care, pharmacy). Interested Veterans can contact study staff to learn 
more about participating. 3) Referrals from providers. Tampa VA providers in primary care - mental health service 
lines will be informed of the study and given research brochures. Providers can then give information to Veterans 
referred to specialty care for treatment.  
Ability to recruit, retain, and justification for sample: A sample size of 10 stakeholders (both patients and 
clinicians) is in line with samples utilized in human centered designs, given the richness of the data being 
collected. Patient sample size is clearly attainable given the Primary Care clinic volume at the Tampa VA. I have 
access to patients given my direct patient contact through my clinical appointment with the Primary Care clinic. 
The Tampa VA PCMHI/PACT clinics have 12 psychologists and 1 psychiatrist, most of whom I have had direct 
contact with and therefore interviewing 4 clinicians would be highly feasible. Both Dr. Milsom (Supervisory 
Psychologist for the Primary Care clinics) and Dr. Gironda (Psychology Section Associate Chief) have approved 
and are supportive of my data collection in the primary care and mental health clinics at the Tampa VA (see 
letters of support). I have ample experience with recruitment and retention of participants with mental illness. My 
involvement in the Women’s Center PCMHI clinic will ensure women are represented.  
Procedures and Methods:  
Intervention content. I am currently in the process of building an initial draft of the intervention in Qualtrics to 
prepare for formative development. The intervention has three features (see Appendix 2): depression symptom 
and mood monitoring designed to increase engagement (i.e., treatment initiation), practical support (i.e., 
appointment reminders), and emotional support (i.e., encouraging messages). Symptom monitoring will be 
accomplished through repeated assessment of depression symptoms with the PHQ9 (with the exception of the 
suicide item which will be excluded in concordance with IRB approved protocols in the past). Mood will be 
monitored using a mood adjective list composed of 7 positive mood adjectives (talkative, enthusiastic, confident, 
cheerful, energetic, satisfied, and happy) and 7 negative mood adjectives (tense, anxious, distracted, restless, 
irritated, depressed, guilty). Specifically, participants will be asked to report on the PHQ9 (except the suicide 
item): 1. The 8 depressive symptoms by rating each one in turn from 0 to 3 (0 “none of the time”, 1 “less than 
half the time”, 2 “more than half the time”, 3 ”most or all of the time”, in the past 24 hours). They will also be 
asked to rate the 14 different adjectives on an analog scale of 0-100. Finally, they will be asked whether anything 
has happened since the last report to trigger a symptom change (both positive and negative events). Patients 
will be asked to report how important, expected, pleasant, unpleasant, and stressful this event was on an analog 
scale of 0-100. Assessments will be completed through online surveys. Practical support will consist of reminders 
of upcoming mental health appointments. Emotional support will be encouraging messages thanking patients for 
reaching out and prioritizing their mental health needs. A list of possible messages will be created as a result of 
our interviews with our patient and provider stakeholders. Practical and emotional support messages will be 
delivered through push texts (i.e., texts that don’t require a response).   
Intervention programming. The intervention will use a daily diary assessment procedure and is programmed 
using an electronic platform, i.e. Qualtrics. Qualtrics is approved by VA central office for research use. Qualtrics 
x 
Other current severe or unstable, psychiatric and medical disorders that necessitates clinical management that can 
confound results (e.g., cancer [in chemotherapy], suicidality, recent hospitalization [medical/surgical] for which 
recovery overlaps with study onset and duration, open skull/brain injury, moderate to severe TBI)  
x 
Moderate to severe cRJQLWLYHLPSDLUPHQW6/806and/or diagnosis in medical record) 
x 
Potentially temporary states/situations that may significantly impair mood/capability to engage in treatment: unstable 
environment that is not in one’s control (e.g., homeless, temporary group home, extensive care taking duties).  

has an ATO agreement for use for government data collection as it is the only Experience Management Platform 
with FedRAMP authorization, ISO 27001 certification, and FISMA compliance. The intervention will be on a set 
schedule to deploy once a day in the evening when patients would receive a text prompt to respond to questions 
described in the prior section which is estimated to take 5-10 minutes, or patients may receive a push text 
message with information only (practical or emotional support). Qualtrics has the capacity to automatically deploy 
contact information for the patient’s provider when respondents’ depression score is elevated beyond a clinical 
cut-off point of 15 (indicative of moderately severe depression which likely warrants treatment for depression 
using antidepressant, psychotherapy, or a combination of treatments).     
Phone screening. Veterans interested in participating will initially contact the study research assistant (RA) by 
phone and participate in a phone screen for eligibility. Alternatively, an RA will be checking for new mental health 
referrals to the mental health clinic and contact patients, initially via mail and later by phone per VA approved 
procedures. The screening interview was developed and used at USF for screening participants for NIH funded 
grants and the PI’s dissertation. The interview will assess for inclusion and exclusion criteria in Table 2. Veterans 
who are eligible following the phone screen will be invited to a face-to-face session. Veterans will be informed 
that study participation does not affect their usual mental health care. Patients that present with mental health 
emergencies will be screened with the CSSRS per VA policies and VA police will be contacted if patient is 
deemed to need emergency care.  
Data collection. Informed consent. The initial face-to-face session will include answering questions, addressing 
concerns, and administering written informed consent. After written informed consent, participants will be 
enrolled in the study and qualitative interviews will be performed when a draft of the intervention is tested.   
Stakeholder qualitative interviews. Semi-structured interviews will be conducted as part of my training in human 
centered design methodology with guidance from Dr. Haun, co-primary mentor on this application. The interviews 
will be completed in-person. The interviews will be conducted using a semi-structured interview guide (see 
Appendix 3) and rapid iterative process which would allow for more in-depth questioning where appropriate. All 
interviews will be audio recorded with participant permission and written notes will be used to support rapid 
iterative analysis. Interviews will be transcribed verbatim for final analysis. The initial interview guide will address 
needs, benefits, expected challenges, and suggested modifications associated with the proposed intervention 
as stakeholders are given an opportunity to use it and experience it in vivo while they trial a preliminary version 
of the intervention. Follow-up prompts will be used as needed to obtain rich descriptions of interviewee 
experiences and needs. Feedback will be sought about the functionality of the intervention, its perceived use, 
and ways to enhance it, including suggestions for emotionally supportive messages patients may want to see as 
they prepare to attend mental health sessions. The intervention will be subsequently revised with the feedback 
received during this process in preparation for active data collection for Aim 3.  
Qualitative Data Collection and Analysis. Data collection and analysis will occur concurrently allowing for insights 
from data analysis to iteratively guide subsequent data collection (e.g., modification of interview questions).  This 
data will be analyzed using an NCT analysis model (Seidel, 1998). The NCT model was chosen because it was 
specifically developed for computer-assisted qualitative data analysis software (CAQDAS). This approach 
consists of three basic components: noticing, collecting, and thinking about interesting things in the data.  The 
NCT model uses coding structures, memoing, process mapping, and diagramming to describe, categorize, and 
connect the data. This process helps to determine common themes, patterns, and inconsistencies relating to the 
interviewees’ experiences, perceptions, and opinions. Interview transcripts will be uploaded into ATLAS.ti v8.0. 
This CAQDAS will assist me in the systematic development of a code book which consists of operationalized 
codes and thematic categories. Memoing and analytic writing will be done at each step of the analysis to 
document the process and to develop conceptual ideas relevant to the data.  I will meet routinely with Dr. Haun 
to review ongoing coding results, resolve coding issues that arise, and discuss the development of thematic 
coding categories.  Interrater reliability with a second coder will be used in the development of the code book 
and to ensure internal validity of the analytic process.  
Data Security. Raw interview data including field notes and audio files will be stored in a locked filing cabinet in 
the PI’s office. Interviews will be transcribed verbatim and saved on a secure server behind the VA firewall.   
A4c. Aim 3.  
Study/Design Overview: Aim 3 will focus on completing phase 2 of the eHealth intervention development, which 
will be a formative evaluation. For this portion of the study we will collect both qualitative and quantitative data 
from a new sample of patients using same inclusion/exclusion criteria in Table 2. This portion of the study will 
help to determine usability, acceptability, engagement with the intervention, and potential problems with 
completing the intervention and collecting data in a complete and rigorous manner.  

Participant recruitment and sampling: Participant recruitment and sampling will follow steps described in Aim 2. 
The sample size selected for this aim (n=15) is in line with samples recommended for formative evaluations and 
pilots. The sample is attainable based on rationale presented in Aim 2. The analytical rationale for the sample 
size is based on the proposed analyses presented below. 
Procedures: Veterans recruited for this aim will be asked to complete 1) baseline questionnaires to understand 
the clinical make-up of the sample and unobserved patient characteristics that may impact our formative 
evaluation findings, 2) the intervention, and 3) a semi structured interview to complete the formative evaluation.  
Self-report measures: These quantitative measures will be used to better understand the makeup of our sample 
in terms of clinical characteristics and treatment readiness that may not otherwise be observable from the 
structured data in their EHRs: Patient Health Questionnaire 9 (PHQ-9), 9-item instrument that assesses the 
symptoms of depression corresponding to the Diagnostic and Statistical Manual Version 5 (DSM-5) diagnostic 
criteria for a major depressive episode. Item responses are on a four-point scale (from occurring “not at all” to 
“nearly every day” over the past two weeks) resulting in a score range from 0 to 27. The PHQ-9 has similar 
psychometric properties to other measures of depression. PHQ-9 is favored for its brevity and ease of use in 
clinical settings (Cameron et al, 2011).  Posttraumatic Stress Disorder Check List for DSM-5 (PCL-5), 20-
item self-report of the severity of PTSD symptoms in the past month using a 5-point scale (0-4), ranging from 
“Not at all” to “Extremely.” A cut-point of 50 (range 0-80) represents a probable diagnosis of PTSD. Negative 
affect and distress: The PANAS (Watson, Clark, & Tellegen, 1988), 20-item scale consisting of adjectives that 
describe mood states. Respondents will rate each mood adjective on a 0–4 scale (0 very slightly or not at all to 
4 extremely). The questionnaire is divided into two subscales of positive affect and negative affect. The PANAS-
N subscale will be used as a measure of subjective distress. The PANAS is widely used and has good reliability 
and validity (Mackinnon et al., 1999; Watson et al., 1988). Difficulties in Emotion Regulation Scale (DERS; 
Gratz & Roemer, 2004), 41-item, self-report measure developed to assess clinically relevant difficulties in 
emotion regulation: (a) awareness and understanding of emotions; (b) acceptance of emotions; (c) the ability to 
engage in goal-directed behavior, and refrain from impulsive behavior, when experiencing negative emotions; 
and (d) access to emotion regulation strategies perceived as effective. These characteristics are hypothesized 
to predict treatment initiation and may impact intervention uptake. Affective Control Scale (ACS; Williams et al., 
1997), 42-item self-report instrument designed to assess anxiety about loss of control over a person's emotions 
and a person's reactions to those emotions (Williams et al., 1997). The scale is composed of four subscales 
measuring apprehension about the experience of different emotions: anger, positive emotion, depressed mood, 
and anxiety. These characteristics may impact intervention engagement. Pretreatment Self-Efficacy (Yeager 
et al, 2018), 8-items that begin with the sentence stem “I am confident that I can start using an eHealth application 
and continue until I see my therapist.” The sentence stem will be followed by items representing technological 
and coping related barriers. Participants will respond on a 5-point scale ranging from not at all confident to very 
confident. Cronbach’s alpha (.95) showed high internal consistency in a young adult sample. This scale will 
provide quantitative data on patient pre-self-efficacy. eHealth Intervention Self-Efficacy (Yeager et al, 2018), 
8-items that begin with the stem “I am confident I could continue to use an eHealth intervention over the next 
two weeks” followed by items measuring treatment self-efficacy related technology and depression coping self-
efficacy. Participants will respond on a 5-point scale ranging from not at all confident to very confident. This scale 
has high internal consistency with Cronbach’s alpha of .94-.96. This scale will provide data on patient post-
intervention self-efficacy. Acceptability of intervention: Rate of agreement to participate and study enrollment 
factors. Engagement in intervention: Percentage of prompts completed/participation days. Other outcomes 
will be extracted as themes from our analyses of semi-structured interviews.  
Intervention: Patients will receive text prompts once a day on their own phones with a Qualtrics survey link (see 
Appendix 2 and detailed description of intervention content in Aim 2). Participants will receive feedback on how 
their depression score compares to a nationally representative average extracted from MIDUS (Midlife in the 
United States: A national epidemiological study of health and well-being), data which we already have and used 
for publication (Rottenberg, Panaite, et al, 2018; Panaite, Rottenberg, under review). 
Quantitative Analyses:  
Data processing. Data will be checked for out of range values and missing data where appropriate through 
plotting and descriptive analyses. Finally, data will be transformed as needed to create the variables of interest 
(e.g. age of depression onset as age of first episode visible in the EHR).  
Missing Data Strategies. We will minimize the level of missing data (due to loss to follow-up, withdrawal, or 
missing assessments) to the extent possible. If missing data occur, I will work with Dr. Kip to assess the missing 
data mechanism for data missing due to loss to follow-up or withdrawal. I will compare baseline measures (e.g., 
demographics, clinical history, etc.) between subjects that have complete outcome data and those that do not 

using logistic regression. Multiple imputation will be used for missing data in conjunction with a pattern of 
missingness. Missing data will be evaluated on an ongoing basis and recruitment may be continued beyond the 
original sample to achieve the desired power for the proposed analyses.  
Data Analysis. First, descriptive analyses will be performed on all baseline patient characteristics extracted for 
all patients approached for study enrollment, including those who agreed and did not agree to participate. To 
compare these two groups, we will use student t-tests (for continuous variables) and chi-square analysis (for 
categorical variables). This information will be used to derive acceptability of the proposed intervention. Next, 
descriptive analyses will be performed on the baseline questionnaires collected from the group of participants 
enrolled. This will give us additional data about patient characteristics that may further refine our phenotypes or 
improve study feasibility in the future. Engagement will be derived from percentage of prompts completed per 
days of participation in the intervention. Finally, we will calculate rates of treatment initiation (and corresponding 
95% confidence interval) and compare to the initiation rate identified in our recent study on the Aim 1 cohort. To 
compare study treatment initiation to rates observed in Aim 3 to those in Aim 1 sample we will calculate the 95% 
confidence interval (CI) for the difference score between proportions (see Franklin, 2007). A significant difference 
will be denoted if the 95% CI does not include zero (Cumming & Finch, 2005). I will use negative binomial 
regression which accounts for overdispersion if treatment initiation is low (Coxe, West, & Aiken, 2009). 
Qualitative interviews and analysis: The interviews will be conducted using a semi-structured interview guide 
(see Appendix 3) and the same methods as Aim 2 interviews and analysis. These interviews will focus on the 
user’s experiences using the self-monitoring intervention. Feedback will be sought about the functionality of the 
intervention, its perceived use, and ways to enhance it. We will specifically inquire about patients’ perceptions 
about the three parts of the intervention: the symptom self-monitoring, practical support (i.e., appointment 
reminders), and emotional support (i.e., encouragement). Patients will also be asked to provide suggestions for 
emotionally supportive messages patients may want to see as they prepare to attend mental health sessions. 
Qualitative data will be analyzed in the same manner as presented in Aim 2.   
A5. DISSEMINATION AND IMPLEMENTATION PLAN.  
Our team has collaborations with operational partners (Deyne Bentt, MD, CPHIMS, Clinical Director, Mobile 
Health Deployment and Evaluation, Office of Connected Care), clinical stakeholders (Dr. Ronald Gironda, Chief 
of Psychology at the Tampa VA; Dr. Vanessa Milsom, Supervisory Psychologist for the Primary Care Mental 
Health Clinics), consultants (Miles, Kozel, Rottenberg, Belanger) and the Tampa Rehabilitation Outcomes 
Section, Research Service, Veteran Engagement Council to ensure successful dissemination of findings. 
Evidence derived from the proposed CDA will inform the applicant’s ongoing program of research in mental 
health service utilization. We will publish findings and present at professional meetings, including the HSR&D 
Annual Meeting and psychology conferences with a mental health focus. Dissemination to stakeholder groups 
will be developed in collaboration with my mentors and consultants on this CDA. Findings will be ultimately used 
in the development of next stages in research, such as the development of a follow-up IIR (see A7 below).  
A6. PROJECT MANAGEMENT PLAN.  
I will direct overall implementation of the project with guidance from my four mentors and four consultants. I will: 
(a) supervise the data extraction and collection; (b) consult with experts to guide the intervention development; 
(c) conduct planned meetings to ensure achievement of benchmarks. The PI will contribute to data collection, 
data analysis, and interpretation. A research assistant will be hired to help with data collection and management 
along Mr. Lapcevic and myself. I will oversee administrative duties (including IRB). In collaboration with my 
mentors, consultants, and support team I will interpret, report, and publish data during the proposed CDA.  
A7. SUMMARY AND FUTURE RESEARCH PROJECTS. 
This CDA will help to establish a foundation for future efficacy/effectiveness research for patient engagement 
interventions to increase treatment initiation and adherence among Veterans with depression. Before 
widespread dissemination and implementation of the proposed intervention, it is important to identify factors that 
may help further individualize this intervention and barriers to broad dissemination and implementation, as 
proposed in this CDA. From prior work in this area we know that complicated interventions take more time, 
require more expertise, and will be less appealing to end users. We hypothesize that the proposed intervention 
will be easier to implement and be more appealing for adoption in primary care. Finally, my relationships with 
local, front-line providers and national leaders in dissemination and clinical operations will be extremely helpful 
in conceptualizing such a rollout. The proposed intervention may help increase access for Veterans by increasing 
adherence to treatment with minimal effort from providers to remind and reschedule sessions. This type of 
intervention could also improve access by providing ample information about patients’ symptoms and mood 
therefore possibly decreasing initial assessment and allowing clinicians to intervene in a timely manner.   

Future Project 1 (proposed HSR&D Merit in CDA year 3). An initial HSR&D Merit will be an efficacy/ 
effectiveness RCT testing whether the eHealth intervention developed and piloted in the current CDA increases 
treatment initiation and adherence with a focus on treatment outcomes. The proposed RCT will be enhanced 
through a study design that includes both an active (i.e., engaging with technology) and passive control group 
(treatment as usual which entails waiting between referral and treatment initiation), more robust sample sizes, 
and if deemed appropriate, a pre-implementation qualitative aim designed to more thoroughly investigate 
implementation barriers and facilitators. Outcomes of the proposed Merit will include treatment initiation and 
adherence, as well as depression outcomes, such as increased awareness of depression, higher depression 
knowledge, and clear identification of treatment goals. Future Project 2. Although we are no longer using NLP 
in the proposed CDA, we have submitted an HSR&D Pilot and received promising feedback that indicates that 
a Merit is warranted. I will be revising this application with reviewer feedback and develop it into another HSR&D 
Merit. The proposed study will employ NLP to develop methods that reliably extract routinely reported patient 
characteristics by mental health care providers in clinical notes, as a promising route to identifying a clinical 
marker for treatment engagement and depression outcomes. We will develop the NLP system to determine 
consistency and variability of documentation across providers and facilities. These data will enable an 
examination of the value of this clinical marker for depression outcomes.  
My proposed CDA and future projects represent the next stage in developing my program of research to 
implement behavioral treatments for depression within primary care settings in the VHA. The proposed research 
will also help to move the field of depression research forward by contributing to our understanding of depression 
management; it could result in new mental health care models for depression, decrease disease burden, and fill 
the gap in the current depression treatment guidelines. 
Table 3. Timeline of CDA Research Activities (Years 1-5). 
Research Activity* 
 
Year 1 
Year 2 
Year 3 
Year 4 
Year 5 
Quarters 
 
1 
2 
3 
4 
1 
2 
3 
4 
1 
2 
3 
4 
1 
2 
3 
4 
1 
2 
3 
4 
Aim 1 
IRB, R&D, train RA 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Semi structured surveys of treatment 
barriers (PI) 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Stage 1 analyses: regression models   
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Stage 2 analyses: machine learning 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Secondary analyses for publications 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Aim 2 
Program Qualtrics intervention 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Obtain Veteran patient input 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Obtain Clinician input 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Aim 3 
Refine intervention 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Recruit Veterans to pilot intervention 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Intervention formative evaluation & 
pilot 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Qualitative data analysis 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Further evaluation and preparation of 
qualitative data for manuscripts 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
IIR 1: RCT (1st submission) 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
IIR 2: NLP Merit (1st submission) 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
IIR 3 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Manuscripts**  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
* Per HSR&D RFA: research funds are provided for years 1-3; years 4-5 are dedicated to dissemination and grant 
writing activities, setting up a sustainable research lab 
**based on manuscript productivity during postdoctoral fellowship: I published an average of 3 manuscripts per year.