Improving Sleep as a Strategy to Reduce Suicide Risk Among At-Risk Veterans: A Real World Clinical Trial
Phase: NA
Conditions: Insomnia, Suicidal Ideation
Sponsor: VA Office of Research and Development

 
Document: 
Study Protocol & Statistical Analysis Plan 
 
Date:  
Last Revised 5/18/2022 
 
Study Title:   Improving Sleep as a Strategy to Reduce Suicide Risk Among at-Risk Veterans: A Real World  
Clinical Trial 
 
Brief Title:   
RCT of Brief CBT-I in Primary Care Veterans With Suicidal Thoughts 
 
NCT#:  
NCT03603717 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Study Protocol 
 
 
 
 
Background 
Given the annual loss of life due to suicide among Veterans (~ 6,000 per year) and service members (~ 350 
per year), the Department of Veterans Affairs (VA) and Department of Defense (DoD) have each identified 
suicide prevention as a key priority area1, 2, working both independently and jointly on suicide prevention 
initiatives3, 4. Despite these and other laudable efforts, suicide rates among all Veterans have not diminished. 
As more service members with combat exposure continue to transition to Veteran status, augmenting existing 
suicide prevention efforts will continue to be a key VA priority. Every additional military or Veteran suicide is a 
tragic call to continually improve prevention efforts. This application focuses on further enhancing VHA care in 
a manner that addresses overall Veteran care and supports suicide prevention.  
Sleep disturbance is an independent risk factor for suicidal thoughts and behaviors5 and is highly comorbid in 
depression and PTSD, which are each themselves associated with suicide. Our rationale for the proposed 
study rests on testable hypotheses: 1) that improving sleep can reduce the symptom severity of insomnia, and 
co-occurring conditions (depression and PTSD) that contribute to suicide risk and reduce the severity of 
suicidal ideation (SI) and 2) that a sleep intervention is feasible to deliver with fidelity to Veterans in primary 
care experiencing insomnia and SI in the context of a co-occurring condition(s) for which they may or may not 
be receiving care.  
Insomnia is seldom addressed by behavioral interventions for depression or PTSD. Therefore, for Veterans 
with these conditions who are also having suicidal thoughts, we posit that treating insomnia represents an 
opportunity to modify risk. We propose to use cognitive behavioral therapy for insomnia (CBT-I) to reduce SI 
severity. An HSRD-funded pilot study has provided proof-of-concept for this approach (see Preliminary Data). 
We now propose a definitive clinical trial to further test the approach of delivering a brief CBT-I in primary care. 
In our pilot work, we trained study therapists to deliver the intervention. Now we propose to utilize integrated 
behavioral health providers consistent with Primary Care-Mental Health Integration (PC-MHI) practices in this 
real-world clinical trial. We also propose to take advantage of a Hybrid I design to gather data for subsequent 
implementation work. Since insomnia is a common denominator across several Veteran populations at 
increased risk for suicide, delivering CBT-I could become a critical feature of existing suicide prevention efforts. 
Current recommendations are that truly comprehensive suicide prevention efforts must target the prevention 
and treatment of SI as well as the prevention of suicidal behavior6, 7. As a common manifestation of suicidality, 
SI is arguably the single most appropriate outcome to study in prospective studies with general clinical 
samples as opposed to suicidal behaviors (suicide, suicide attempts) which occur at much lower incidence 
rates8. We propose to measure the severity of SI as our main suicide outcome, but, importantly, we plan to 
assess whether and how the intervention effects on SI severity are mediated.  
Suicide risk among Veterans with Co-Occurring Depression and/or PTSD 
The need to enhance clinical services to treat behavioral disorders and to prevent suicide is underscored by 
the strong etiological link between behavioral disorders and suicide9, 10. In psychological autopsy studies 
approximately 90% of suicide victims had one or more behavioral health disorders during their last weeks of 
life11. Approximately half of Veteran suicide decedents in a national cohort study had a history of one or more 
behavioral health disorders12. Major depression, for instance, is a potent risk factor for eventual suicide among 
Veterans using VHA services12, 13. Similarly, PTSD has been found to predict SI in recent combat  Veterans14. 
The VHA has adopted consensus recommendations to, and is a leading healthcare system in, promptly 
recognizing symptoms of depression and PTSD15, 16. These efforts stem from VHA recognizing the prevalence 
of these symptoms in Veterans and the impact these symptoms have on morbidity, mortality, and quality of life 
decrements17. So, considerable thought, effort and resources have been directed at identifying Veterans with 
one or more of these conditions; conditions that are each associated with suicide.    
Few proven interventions for these disorders exist that impact suicidal thoughts and behaviors7, representing a 
critical gap in care both in and outside the VHA. A recent meta-analysis of psychotherapy for depression, 
for example, found only a small effect on suicidal thoughts and behaviors for these interventions18. We propose 

that insomnia, a problem that co-occurs in the majority of patients with depression and PTSD, represents an 
ideal target to enhance care in the service of reducing SI and potentially suicide risk. 
A Conceptual Model Guiding the Hypothesis that Improving Sleep can Alter Suicidal Ideation  
Our simplified conceptual model that is guiding the current project is based upon the supposition that 
improving sleep with CBT-I will have direct and indirect effects on SI (Figure 1). This can include indirect 
benefits from decreased symptom severity in the comorbid conditions, elimination of cognitive-behavioral 
factors giving rise to SI and increase suicide risk (e.g., disinhibition, negative rumination, hopelessness), 
and a reversal of neurobiologic abnormalities.  As the stressor of insomnia is removed and replaced with 
the rest and restoration that allow a Veteran to marshal their ability to cope, a direct reduction in SI may 
ensue. The analytic strategy of the current study is designed to test both direct effects and some, though not 
all, of these indirect effects, representing an important step in a broader research program to elucidate whether 
and how targeted sleep interventions may decrease morbidity associated with psycho-behavioral conditions 
and become integrated into suicide prevention strategies. 
 
 
 
 
 
 
 
 
 
Why Focus on Primary Care? 
Primary care, not specialty mental health care, is the setting where patients with multiple comorbidities (e.g., 
PTSD, depression) often seek help for insomnia19. For instance, we found that 50% of Veterans being seen in 
primary care reported having insomnia in the past 12 months, with 35% reporting it as a concern in the past 
week19. In addition, a substantial number of these Veterans with co-occurring insomnia and mental health 
symptoms (e.g., depression, PTSD) are managed in primary care and do not engage in specialty mental health 
services17, 20. In fact, a majority of patients who die by suicide have been seen in primary care, not specialty 
mental health care, within a month prior to suicide21, suggesting that any prevention/intervention efforts to 
help those at increased suicide risk needs to occur in primary care. In 2012, 86.5% of VHA primary care 
clinics have at least one Primary Care-Mental Health integrated (PC-MHI) provider22 and that percentage is 
now approaching 100%.These PC-MHI providers fill a critical gap by providing the mental health knowledge 
and expertise to the Patient Aligned Care teams, complementing existing specialty mental health services by 
providing brief behavioral treatments to Veterans with a focus on improving functioning as well as serving as a 
bridge for those Veterans not currently seeking specialty mental health services by increasing comfort with this 
type of service resulting in increased engagement23 However, these providers desperately need evidence-
based behavioral interventions that can be used with Veterans at high suicidal risk within a format consistent to 
their clinical practice (i.e., 4 or fewer 15-30 minute sessions)24 The proposed project is poised to meet such 
needs.  
Gathering Implementation Data 
Consistent with recommendations in the literature to improve speedy translation of effective interventions to 
community settings that need them25, we propose a Type 1 Hybrid design; it is the most appropriate approach 
in an RCT study based on promising data testing an intervention (CBT-I) in a community setting (VHA primary 
Care)26. This hybrid design rigorously tests intervention outcomes and, secondarily, gathers data (feasibility, 
acceptability), through qualitative, process-oriented or mixed methods to inform subsequent implementation 
research26. Therefore, we will collect data on implementation in the context of a clinical trial where 
randomization is at the level of the treatment conditions and not at the level of implementation strategies. Our 
approach is fully consistent with these goals and guided by Proctor et al.’s framework27.  For example, the 
current study will be conducted in multiple VHA settings with those communities’ therapists. Our pilot study 
therapists, in contrast, were dedicated study therapists who had a clinical Master’s degree or were post-
doctoral fellows. It is critical to learn about factors related to training and monitoring ‘community’ therapists, 
their experiences and feedback regarding delivering CBT-I in usual practice settings, as well as site level 
 
Insomnia 
+ 
Comorbidity 
+
Co-Morbid Condition(s)  
CBT for 
Insomnia 
Improved
Sleep 
Neurobiologic Factors 
SI 
Severity 
Cognitive-Behavioral Factors 
 Figure 1. Conceptual Model of How Treating Insomnia Can Impact Suicide  

factors that are likely to impact implementation of the treatment (e.g., size, patient acuity). Therefore, we will 
examine a number of facilitators and barriers (for patients, therapists, sites) important to informing future 
studies that will explicitly assess and test implementation strategies (i.e., Hybrid II) and aid successful 
dissemination and implementation of brief, primary care based CBT-I across VHA. 
Preliminary Data Gathered to Support this Application 
In addition to completing the first meta-analysis assessing the association of sleep disturbance to suicidal 
thoughts and behaviors5 and publishing the first Veteran specific manuscript on the association of sleep 
disturbance to suicide28, we have used both internal funding and an HSRD Pilot Award to gather preliminary 
data to support this next stage of research. First, in an uncontrolled study, we delivered full length CBT-I (8 
individual sessions) to combat Veterans with co-occurring PTSD or depression (n=11). There were large 
effects on insomnia severity, with statistically significant reductions in both depression and PTSD severity29. 
Three participants endorsed SI at baseline; one following the CBT-I intervention, none at 3-month follow-up.  
Second, based on a signal that CBT-I diminished SI and our interest in developing brief insomnia interventions 
for Veterans with common co-occurring conditions within primary care, we designed a CBT-I intervention 
comprised of two individual face-to-face sessions, two telephone sessions, and a 50-page patient workbook. 
Following focused interviews with six Veterans who reviewed our protocol and materials, we reduced the 
content and size of patient workbook and prepared our second pilot trial.  
Third, we conducted a small (N=23) comparative effectiveness study of 2 vs. 4-session CBT-I for Veterans with 
depression seen in primary care and determined that the interventions could be delivered in primary care (with 
50% of sessions delivered by telephone)30. We also observed a large within-group effect of CBT-I on both 
insomnia (d=1.03) and depression severity (d=.75). Eight of twelve participants (67%) endorsing SI at baseline 
no longer endorsed SI following CBT-I. Among these twelve participants, there was a large effect (d = 1.31) on 
SI intensity based on the Columbia Suicide Severity Rating Scale (C-SSRS)31. These uncontrolled study 
findings supported the need for a controlled trial of brief CBT-I to reduce suicidality.  
Fourth, with HSRD pilot award support (HX001473) we conducted a small randomized trial of our 4-session 
primary care version of CBT-I compared to treatment-as-usual in primary care. A total of 50 Veterans with 
insomnia, recent SI and either MDD and/or PTSD completed the study. The effect sizes (conservatively 
adjusted for sample size) were very large for insomnia (g = 1.79) and depression (g = 1.19; g = 1.13 with sleep 
item removed) and modest for SI Intensity (g = 0.44). While it is modest, the effect size for SI is in line with 
effects observed in a study of dialectical behavior therapy for women Veterans with borderline personality 
disorder32 and similar or larger than effects achieved by depression psychotherapies18, 33.  
Fifth, for the HSRD pilot we developed reliable therapist fidelity measures for Adherence to the treatment 
manual for all 4 CBT-I sessions and Competence in delivery of the CBT-I using gold standard observational 
methods34-36.  
Finally, the HSRD pilot provides recruitment & retention feasibility. We enrolled 54 participants over 8 months 
(~7 per month) and retained 50 (7.4% attrition rate). This was achieved using two primary care clinics at the 
Canandaigua VA Medical Center (one at the VAMC and one at its Rochester Outpatient Clinic). This was also 
accomplished at the same time that our study team was successfully recruiting primary care patients with 
depression for another study.  This demonstrates our ability to recruit participants at these sites. In the current 
application, we propose to add primary care clinics at two additional VAMC’s (Syracuse and Buffalo), which will 
significantly enhance recruitment capacity. Although these data are for a study that did not have longer term 
follow-up assessments, the high retention rate is a promising indicator for the proposed study. 
Summary 
Our overarching hypothesis is that a brief 4-session primary care version of CBT-I, an intervention targeting 
insomnia, can diminish suicide risk among Veterans who endorse insomnia and co-occurring conditions 
(depression, PTSD) that are associated with suicide. An accumulating literature suggests the possibility that 
suicide prevention efforts can be augmented by considering an adjunctive insomnia intervention. The rationale 
for our approach is strong. First, insomnia has a robust and intricate relationship with each of these conditions 
and with suicidal thoughts and behaviors. Second, CBT-I improves sleep in a number of co-morbid conditions 
with a preliminary suggestion that it can also improve SI. Third, CBT-I can potentially diminish the symptom 
severity of depression and PTSD. Just as the presence of two or more conditions increases the risk of SI, 
improvements in two or more conditions may diminish that risk. Fourth, a positive experience with CBT-I can 
improve Veteran attitudes towards other treatments that could further reduce symptom severity. Finally, 
primary care is an ideal setting for this treatment due to the increased likelihood of Veterans at high suicidal 

risk presenting to primary care and represents a setting for which Veterans report a preference. 
 
Specific Aims 
 
Aim 1: To test the direct effects of a brief CBT-I intervention, compared to SH, on the trajectories of 
severity of insomnia, depression, PTSD and SI through 6 month follow-up.  
Hypothesis 1a: CBT-I, compared to SH, will be associated with a greater rate of reduction in severity of 
insomnia over time. 
Hypothesis 1b: CBT-I, compared to SH, will be associated with a greater rate of reduction in depression 
severity over time.  
Hypothesis 1c: CBT-I, compared to SH, will be associated with a greater rate of reduction in PTSD 
severity over time.  
Hypothesis 1d: CBT-I, compared to SH, will be associated with a greater rate of reduction in SI severity 
over time.  
Aim 2: Utilize structured equation modeling to determine if (and how) effects of CBT-I on SI are 
mediated by sleep, by depression and by PTSD.  
Hypothesis 2a: CBT-I effects on SI at 6 months will be both direct and indirect, mediated by direct 
intervention effects on sleep, depression, and PTSD.  
Hypothesis 2b:  Depression at 3 months will be positively related to SI at 6 months.  
Hypothesis 2c:  PTSD at 3 months will be positively related to SI at 6 months.  
Hypothesis 2d: Post-intervention sleep will be positively associated with 3 month depression, 3 month 
PTSD, and 6 month SI. 
Exploratory Aims:  
(1) Gather information on the feasibility and acceptability of delivering CBT-I as a PC-MHI intervention to 
Veterans with SI and co-occurring conditions including factors that facilitate and impede intervention delivery 
using a mixed methods approach to aid future implementation efforts. 
(2) Examine whether CBT-I enhances positive attitudes towards non-pharmacologic treatment for depression 
or PTSD among those Veterans not currently engaged in psychotherapy. 
 
Methods 
 
Design Overview 
The proposed study is a multi-site, randomized, clinical trial conducted in a real world setting while collecting 
implementation data in a Hybrid I design. This study is designed to test the effectiveness of a brief, primary-
care based form of CBT-I in a sample of Veterans (N=240) endorsing SI, insomnia, and co-occurring 
depression or PTSD. Following an initial telephone screening, all patients who agree to participate and who 
meet all initial inclusion criteria will be asked to participate in the study and to schedule a full baseline 
assessment. Eligible Veterans will then be randomized to one of two conditions: (1) CBT-I or (2) Sleep Hygiene 
(SH). All participants will be encouraged to maintain appropriate treatment as usual for their co-occurring 
condition(s) as recommended by their VA treatment providers. Comprehensive assessments will occur at pre-
treatment (baseline), post-treatment (6 weeks), and every 6 weeks until a 6-month follow-up assessment (at 30 
weeks). 
Randomization will occur following informed consent and the subsequent pre-treatment assessment on a 1:1 
ratio, stratifying on site, gender and current psychotherapy for depression/PTSD treatment. We will use a 
stratified block design with block sizes of six. As discussed in the analytic strategy, we will assess for equal 
distribution of other covariates of interest.  
Study Population 
This study will seek to enroll 320 Veterans in order to randomize 240 eligible Veterans to accrue 192 
completers. Based on our prior studies and the composition of the Veteran population served by our 
recruitment sites, we expect to have approximately 20% female Veterans and approximately 25-30% minority 
Veterans. In our most recent study, 21% of enrolled Veteran participants were women and 26% were 
minorities. We will be adding two recruitment sites (Buffalo and Syracuse) which have a higher minority 
Veteran population than our pilot site (Canandaigua). We project a 30-month recruitment period (8 subjects per 
month across three sites).  

Inclusion Criteria: (i) English speaking Veteran aged 18-80 seeking or receiving services at the Canandaigua, 
Buffalo or Syracuse VAMCs, or associated CBOCs; (ii) demonstrate understanding of informed consent; (iii) 
endorse current death/suicidal ideation on item 9 of the Patient Health Questionnaire-9 (PHQ-9)37 anchored to 
the last three months,; (iv) an Insomnia Severity Index (ISI)38 score > 10 with trouble sleeping ≥ 3 months and 
at least 1 insomnia-related daytime consequence; (v) either [a] a current diagnosis of Major Depressive 
Disorder, Depression not otherwise specified or PTSD or [b] a score of  > 16 on the PHQ-937 or a score  ≥ 31 
on the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5)39; and 
(vi) if using psychotropic medications the dosage must be stable.  Exclusion Criteria: (i) history of serious 
mental illness such as schizophrenia, Bipolar I or II disorder, or current psychiatric conditions such as 
psychosis, mania, dementia or cognitive impairment(ii) a suicide attempt within the last 6 months or report of SI 
with active plan and intent in the past month (iii) currently engaged in inpatient or partial hospitalization 
programs; (iv) recent substance dependence disorder with < 3 months abstinence; and (v) narcolepsy, 
circadian rhythm disorders, restless legs syndrome, or untreated sleep apnea based upon chart review, a 
sleep disorders screening questionnaire and the STOP-BANG sleep apnea questionnaire40. Sleep medications 
are not exclusionary, but participants using them must still meet insomnia criteria. 
Settings 
Procedures will be performed at the Canandaigua, Syracuse, and Buffalo VA medical centers and utilize 
Center of Excellence for Suicide Prevention and Center for Integrated Healthcare onsite offices. 
Procedures.  
Recruitment Methods. Recruitment will occur using two methods utilized in prior studies conducted by the 
investigators and which ensure the involvement of participant’s VA primary care and/or integrated PC-MHI 
provider: letters from providers to patients and direct referrals. A list will be developed via data pull or chart 
review of Veterans who have had an encounter in primary care or specialty mental health care in the past 60 
days, who have (i) diagnoses listed on their encounter form of Major Depressive Disorder, Depressive Disorder 
NOS, or PTSD (ii) elements of the suicide prevention processes (such as PHQ-9 item #9, comprehensive 
suicide risk evaluation, Columbia-Suicide Severity Scale, safety plan progress note) in the electronic medical 
record. Letters describing the study and signed by the lead primary care physician or by the patient’s primary 
care provider will be sent to these Veterans to alert them that a research staff will call them within 1-2 weeks to 
ask about their interest in participating in a research study utilizing a telephone screening script. Individuals 
meeting initial eligibility during this verbal screen will be scheduled for the informed consent process and 
baseline assessment, which will take place at one of the recruitment sites. We will do an initial data pull or 
chart review  and repeat as necessary until enrollment goals are met. For Veterans in the initial data pull who 
do not meet initial eligibility criteria following the phone screen but who re-appear in subsequent data pulls or 
chart reviews, we will send a letter (see re-contact letter template) reminding them about the study and call 
within 1-2 weeks to re-assess their eligibility utilizing the telephone screening script. We will request a HIPAA 
waiver for the data pulls or chart reviews, which will be conducted by an information technologist within VISN 2 
or will be requested and extracted from the VA Corporate Data Warehouse (CDW) and placed within VA 
Informatics and Computing Infrastructure (VINCI). In addition, VA health care providers can directly refer 
patients to the study that they feel meet eligibility criteria by providing our recruitment flyer, walking them to our 
research offices, calling the study coordinator, or placing Dr. Funderburk as an additional signer to their 
progress note.  
Participants will receive compensation for their participation in the study. This compensation will be in the form 
of electronic funds transfer (EFT) or a Direct Express prepaid debit card, per Department of Treasury 
regulation. Participants will be compensated up to $250 for their participation according to the following 
schedule: 
Baseline Assessment  
 
 
$50  
 
6-week assessment (post-treatment)  
$40 
 
12-week assessment   
 
 
$40 
    
18-week assessment (3 month follow-up)  $40 
 
24-week assessment  
   
 
$40 
 
30-week assessment (6 month follow-up) 
$40  
 
Feasibility of Recruitment Methods. We have conducted a preliminary assessment of the target sample 
population at our recruitment sites to determine that over 4,000 unique patients could be identified as potential 

subjects based on their diagnoses in the past three months. In our previous research studies, recruitment 
method #1 (i.e., sending letters to Veterans from their providers) has been extremely successful. For instance, 
in one of our preliminary studies providers reviewed 80% of potentially eligible patients within one week and 
the study team contacted 95% of them by phone over a 6-month period.  
Using previously collected data in a similar Veteran population, we calculated a 50% decline rate at the initial 
telephone screen. Among those completing the phone screen, we expect a 75% rate of ineligibility due to sleep 
apnea, lack of insomnia, or absence of SI. Starting with a recruitment pool of 3,000 Veterans, this would leave 
500 eligible to participate. We expect approximately 25-50% to decline the invitation into the study following the 
phone screen, leaving a minimum of 250 participants eligible to complete the informed consent process and 
baseline assessment after the first data pull. Given that a similar number of potential participants will likely be 
identified in subsequent data pulls (e.g., a total of 1,000 over four data pull periods), which compares favorably 
to the target sample size of 240 (approximately 60 needed every 6 months).  
As noted in the Preliminary Data section, our HSRD-funded pilot study recruited and randomized 54 
participants over 8 months from the Canandaigua site alone. In this application we are proposing to enroll 320 
participants to randomize 240 participants over 30 months (8 per month) from three sites (2-3 per month per 
site). In other words, although the target recruitment rate is slightly higher than in the pilot study, we are adding 
two additional recruitment sites (the Buffalo and Syracuse VAMCs) for this larger trial.     
The initial eligibility screen will occur by telephone and be conducted by research assistants trained in this 
procedure following a written screening script; any Veteran may request to be seen in person. During the initial 
assessment, the research assistant conducting the screening will describe the study and all aspects of consent 
to the patient and evaluate comprehension of consent using two comprehension questions (see phone screen 
script). Following verbal consent, patients will be asked to verbally complete: the Insomnia Severity Index (ISI), 
sleep apnea screening questions, and a suicide assessment. The screening is constructed so as to avoid 
inviting Veterans to the intake assessment who are highly unlikely to meet eligibility criteria (e.g., due to the 
presence of sleep apnea or absence of insomnia). Participants meeting initial eligibility criteria who wish to 
participate will be scheduled for a baseline (pre-treatment) assessment.  
We are requesting a waiver of consent documentation for this phone screening; full written informed consent 
will take place at the pre-treatment assessment for those participants who are eligible following the phone 
screen and wish to participate in the pre-treatment assessment. 
If the participant is ineligible to participate in our study or is eligible for other studies, we would like to be able to 
ask them if they are interested in hearing about other research studies going on at their local VA if they are 
interested in participating. If interested, we would help connect that participant and the research team with 
them to obtain additional information. 
Upon completion of the initial eligibility screen and scheduling of the pre-treatment assessment, research staff 
will ask Veterans whether they currently receive VA payments via EFT (e.g., travel reimbursement) or via 
Direct Express debit card. For those Veterans not currently enrolled in EFT or the Direct Express debit card, 
research staff will inform the Veteran on the information required for enrollment in EFT or the Direct Express 
debit card and that the appropriate form will be provided during the in-person pre-treatment assessment. Once 
the Veteran completes the enrollment paperwork, it will be immediately turned over to the Syracuse VA or 
Canandaigua VA agent cashier. 
If there are any issues with enrollment or receipt of compensation, research staff will contact the Veteran over 
the phone or the Veteran may call the research staff to communicate about the issues and work toward a 
resolution. Since this often requires a lot of information to be conveyed over the phone (i.e., research staff 
contacting the fiscal office for information for the Veteran) a follow-up letter with this information may also be 
sent to the Veteran in order to assist in the resolution of the payment issues.  No identifiable data or sensitive 
payment information will be included with these payment letters.  
The pre-treatment assessment will be conducted by assessors immediately following an informed consent 
process. The assessment will consist of self-report instruments described in the Measures section below.  
Participants who remain eligible will then be randomized (stratified by gender, study site and treatment status). 
Following informed consent and enrollment in the study, research staff will enter a study initiation note into the 
participant’s electronic medical chart. This note will notify other VA providers, including the patient’s primary 
care provider, only that the Veteran is participating in the study and receiving an intervention for sleep. 
Research staff will also enter progress notes into the participant’s chart following completion of assessment 
sessions to document any suicide risk. A study completion note will be entered into the participant’s chart 

following completion of the study treatment and assessments, if the participant decides to withdraw their 
participation at any point, or if their participation is discontinued by Dr. Pigeon or Dr. Funderburk. Progress 
notes will only be entered following study intervention sessions only if the patient is in crisis – this is to maintain 
the blind for the assessors. 
Modifications during the COVID-19 outbreak: 
To address the public health concerns during the outbreak of COVID-19, the initial telephone eligibility 
screening and pre-treatment assessment will be modified to include precautions to safeguard Veterans’ health 
during the outbreak. The phone screen will be conducted as described, and will be modified to include working 
with interested and eligible Veterans to establish the VA Video Connect option for the pre-treatment 
assessment. The pre-treatment assessment will be modified as follows: prior to the pre-treatment assessment 
appointment a research assistant will send two copies each of the IRB-approved informed consent form and 
HIPAA authorization form via mail, along with a pre-addressed, pre-stamped envelope for the participant to 
return their signed informed consent form and HIPAA authorization form to the research staff at the Syracuse 
VAMC. The mailed packet will also include copies of the response options for some of the baseline 
questionnaires as well as an information sheet with relevant phone numbers for VA behavioral health services 
and crisis resources including the Veterans Crisis Line. At the appointment, the assessors will follow standard 
VA Video Connect procedures to create a confidential medical virtual room. At the beginning of the 
appointment, the assessor will verify the participant’s current location and phone number to ensure if any 
disruptions occur, we can re-contact the Veteran and provide assistance. During the informed consent 
process, the participant will be able to reference the paper copy of the informed consent and HIPAA 
authorization forms and the assessor will visually show materials within the virtual medical room. All research 
staff will be trained in standard VA Video Connect procedures and follow the same suicide risk protocol that we 
established for our other in-person and telephone appointments (which matches how video connect behavioral 
health appointments are being conducted). Research staff will have the participants hold up the signed consent 
and HIPAA forms to take a screenshot of the signatures and stamps, which will be saved in password-
protected files in the study folder on the approved VA server to which only IRB-approved research staff will 
have access. Participants will be asked to send back the signed informed consent and HIPAA authorization 
within the self-addressed envelope. If the paper copies of the consent and HIPAA are not received quickly, 
research staff will work with participants to obtain signed paper copies before the end of the study, with the 
screenshots of the signed forms as back-up documentation. Research staff will also print the images of the 
signed consent and HIPAA forms and attach them to the new paper copies in the event that there is a 
significant delay or new copies are sent to the participant to sign. All consents and authorizations will be stored 
in the same locked file cabinets and the same rooms as usual. All other appointments will occur via telephone, 
which is already an option in our standard protocol. At the pre-treatment assessment and all follow-up 
assessments, we will also administer 10 items related to the experience and impact of COVID-19 and related 
social isolation/quarantine recommendations. There will be one additional COVID-19 item given only at the 
post-treatment assessment regarding the impact of COVID-19 on their experience of the study intervention. 
Treatment-as-Usual (TAU). All participants will be encouraged to maintain appropriate TAU for their co-
occurring condition(s) as recommended by their VA treatment providers. This means that they may be 
receiving psychotherapy and/or medication management as often as the VA treatment providers deem 
necessary. This can include treatment within the primary care teams, through behavioral telehealth, and/or 
specialty outpatient mental health. We will monitor the participant’s engagement in TAU throughout the study. 
All Participants, in addition to having access to TAU for co-occurring conditions, will also be allowed to use 
pharmacotherapy for insomnia provided that they have begun such treatment prior to enrollment. Medication 
use will be tracked on the sleep diaries. Changes in sleep medication use will not be precluded after the post-
treatment assessment has been completed, but will continue to be tracked via sleep diaries and medication 
usage questions at follow-up assessments. Other sleep treatments for insomnia (e.g. CBT-I for participants in 
the SH condition; mindfulness based CBT-I for those in the CBT-I condition) will be precluded until after the 
post-treatment assessment. Utilization of non-pharmacologic insomnia treatments between the post-treatment 
and subsequent assessments will be monitored and recorded at follow-up assessments. Finally, all participants 
will be asked to complete sleep diaries for the duration of the intervention period and one week prior to the 
follow-up assessments. Capturing medication and other treatment data will allow us to examine whether they 
confound study intervention treatment effects. 
Sleep Hygiene (SH) Participants will participate in four individual sessions over the telephone that will last 15-
30 minutes each across a 6-week time period. Guided by a manual and patient workbook (see Appendices), 

interventionists will guide patient through SH material (i.e., basic psychoeducation about sleep, discuss SH 
factors that disrupt and improve sleep, set SH goals, develop action steps to achieve those goals). At each 
session, participants will complete three items from the PHQ-9 (items for depressed mood, anhedonia, and 
suicide/death ideation) to monitor mood and suicide risk over the course of treatment. This data may also be 
used for research purposes. Sessions will be audiotaped for fidelity to SH principles. 
CBT-I Intervention Participants will receive four individual sessions that will last approximately 15-30 minutes 
each across a 6-week time period. Some sessions may be conducted over the telephone. Following the 
manual and a patient workbook (see Appendices), CBT-I will consist of a standard, structured, multi-
component CBT-I intervention used in previous studies by the PI’s (sleep education, sleep hygiene, sleep 
restriction, stimulus control, and cognitive therapy).  At each session, participants will complete three items 
from the PHQ-9 (items for depressed mood, anhedonia and suicide/death ideation) to monitor mood and 
suicide risk over the course of treatment. This data may also be used for research purposes. Participants will 
also be given sleep diaries at each session to fill out for the next appointment. Sessions will be audiotaped for 
fidelity. 
Patient Treatment Adherence. Participant adherence will be monitored by homework logs, which participants 
complete each session. Daily sleep diaries, which are used to guide the refinement of the sleep restriction 
component of CBT-I, will also be used to monitor adherence. Rates of participation and adherence will 
contribute to our assessment of feasibility and acceptability.  
Therapists & Therapists Training. Dr. Pigeon will train at least one PC-MHI provider at each site to deliver 
efficacious CBT-I therapy in a manner consistent with the VACO CBT-I Training model41. Training includes 
individual and group face-to-face didactics, independent reading, role-playing, and supervised training cases 
until competency is met. A similar procedure will follow for SH training. Dr. Funderburk will provide direct 
clinical supervision and Dr. Pigeon will meet bi-weekly with therapists along with Dr. Funderburk and Dr. 
Bishop in group supervision to review cases. 
Therapist Treatment Fidelity. Implementation science research emphasizes the importance of assessing 
fidelity to evidence-based treatments shown to be efficacious in highly controlled settings. We will assess 
fidelity by audio taping intervention sessions (securely stored on a VA server) and use reliable fidelity 
measures developed in our pilot study to rate therapist fidelity. Our measures reflect two core components of 
therapist fidelity: 1) Adherence to the intervention manual content, and 2) Competence in the delivery of the 
intervention (i.e., quality)34, 35. These two components are important to measure because a therapist may 
demonstrate high levels of adherence (that is, provides the treatment content as described in the manual), but 
does so in a manner that demonstrates low levels of competence. Thus, the treatment would be delivered in a 
manner that is “rote,” poorly timed or insensitive to the individual patient. The converse may also occur: low 
adherence and high competence. Both scenarios would compromise conclusions about the outcomes of the 
treatment study. Therefore, it is critical to measure both components of therapist fidelity in an intervention trial. 
Follow-Up Assessments (5 in total) will occur at post treatment (6 weeks), and every 6 weeks until the 6 month 
follow-up (at week 30) using most questionnaires used at the pre-treatment assessment (see Table 1). The 
initial follow-up assessment will be completed in person, with the option to conduct them over the phone if 
necessary. All other follow-up assessments will occur by phone unless face-to-face is requested by 
participants. If a participant misses a follow-up, we will send a letter (see missed follow-up letter template) to 
help facilitate rescheduling the follow-up. 
Assessors for all assessments will be blinded research staff who have completed training in human subjects 
research and in specific research activities (recruitment, screening, informed consent, assessments, qualitative 
interview skills, and data management) via a didactic series of independent reading, lecture and role play. 
An Exit Interview will occur immediately following the post-treatment assessment and be conducted by a 
trained and blinded interviewer. It will consist of both self-report instruments and a semi-structured interview to 
assess treatment feasibility and acceptability of CBT-I (participants in CBT-I condition only) and attitudes 
towards psychotherapy for co-occurring conditions (all participants). 
Measures 
Screening/Eligibility Measures will include: review of the electronic medical record; the PHQ-9 to assess for the 
presence of SI; an unpublished instrument that we have developed and used in prior work the Demographic & 
Veteran Information Form to capture basic demographic and military service information; the STOP-BANG 
sleep apnea questionnaire40; an unpublished instrument the Health Behaviors Interview to screen any drug or 
substance use over the past 3 months; MINI International Neuropsychiatric Interview (MINI)42 modules for 

mania and psychosis, medical chart review to assess current enrollment in inpatient/partial hospitalizations, 
medications, and specific exclusionary diagnoses; and as noted in eligibility criteria, the ISI, PHQ-9, and PCL 
will be used.  
Outcome Measures. The Scale for Suicidal Ideation (SSI)43 is an interview that measures the “intensity of the 
patient’s specific attitudes, behaviors, and plans to commit suicide.” Participants will be asked about “current” 
SI, defined as occurring during the past week (SSI-C), at pre-treatment and each subsequent time point. The 
SSI is considered the gold standard measure for SI with good internal consistency, construct validity, and 
predictive validity, is sensitive to change over time44, and has been used with Veterans45. The SSI-C will be our 
primary SI severity outcome.  
In addition, each assessment will include: several instruments administered for screening (ISI, PHQ-9 and 
PCL); the 20-item Beck Hopelessness Scale (BHS)46; a one week sleep diary to ascertain total sleep time, 
sleep efficiency variables, the frequency and intensity of nightmares and medication use; the Suicide Attempt 
Self-Injury Interview (SASII)47 to assess suicidal behaviors. 
 
Feasibility, Acceptability, and Treatment Receptivity Measures will be collected/administered at the Patient Exit 
Interview using a mixed methods approach as follows. A Sleep Diary is completed upon awaking and used to 
guide the sleep restriction portion of CBT-I; it will also be used to measure treatment adherence (an aspect of 
feasibility) with respect to sleep scheduling. Two instruments with high reliability, the 10-item Attitudes Toward 
Seeking Professional Psychologist Help48 and the 19-item Thoughts about Psychotherapy Survey49 will be 
administered both at baseline and at the exit interview to assess attitudes toward psychological care. 
Semi-Structured Interviews will be conducted by a trained research interviewer to gather: (1) additional 
participant feedback about acceptability, facilitating factors, and obstacles to CBT-I engagement (feasibility) 
from CBT-I instruments and (2) feedback from all participants on attitudes toward PTSD and depression 
treatment. The Interviewer will use a written guide (see “Exit Interview” in Appendices) to conduct the ~15-30 
minute interviews, which includes a protocol for engaging the respondent and establishing rapport, questions 
and topics that need to be covered during the conversation, the order of inquiry, and prompts for follow up 
questions. Interviewers follow the guide but are able to pursue topical trajectories in the conversation that may 
stray from the guide when appropriate to fully understand respondents’ comments and to enrich the data. The 
interviewer records responses in writing, however, sessions will be audiotaped for consensus on themes.  
We will assess therapist perspectives on the feasibility and acceptability of delivering CBT-I to veterans with 
suicide ideation in primary care using a Provider Survey.  We will also include items enquiring about therapists’ 
Table 1. Measures 
Screen 
Pre 
Post 12 wks 18 wks 24 wks 
30 wks 
Insomnia Severity Index (ISI) 
X 
X 
X 
X 
X 
X 
X 
Scale for Suicidal Ideation (SSI) 
 
X 
X 
X 
X 
X 
X 
STOP-Bang Sleep Apnea Questionnaire 
X 
 
 
 
 
 
 
Demographic and Veteran Information Form 
 
X 
 
 
 
 
 
MINI IntNatl Neuropsychiatric Interview (MINI) 
 
X 
 
 
 
 
 
Non-VA Treatment Questionnaire 
 
X 
 
 
 
 
 
Health Behaviors Interview 
 
X 
 
 
 
 
 
PTSD Symptom Checklist-DSM 5 (PCL-5) 
 
X 
X 
X 
X 
X 
X 
Patient Health Questionnaire (PHQ-9) 
 
X 
X 
X 
X 
X 
X 
Beck Hopelessness Scale (BHS) 
 
X 
X 
X 
X 
X 
X 
Contemplation Ladder 
 
X 
X 
X 
X 
X 
X 
Columbia-Suicide Severity Rating Scale 
(CSSR) subscale 
 
X 
X 
X 
X 
X 
X 
COVID-19 Items 
 
X 
X 
X 
X 
X 
X 
Suicide Attempt Self-Injury Interview (SASII) 
 
X 
X 
 
X 
 
X 
Sleep Diary (with nightmare items) 
 
X 
X 
X 
X 
X 
X 
Treatment Engagement Assessment 
 
 
X 
X 
X 
X 
X 
Semi-Structured Exit Interview 
 
 
X 
 
 
Two measures of Treatment Attitudes 
 
X 
X 
 
 
 
 
Electronic medical Chart Review 
 
X 
X 
 
 
 
X 

experiences participating in the study (e.g., burden, benefits). A Provider Survey to gather therapist 
perspectives on feasibility/implementation factors will take place at study conclusion. 
Therapist Fidelity.  We will code therapy audio tapes using reliable fidelity measures (See Appendices) 
developed in the pilot study. Coding will be conducted by a team of coders trained to criteria, who will rate both 
adherence and competence such that sessions 1 or 2 and sessions 3 or 4 will be rated for each subject. Thus, 
2 sessions for will be rated for fidelity in each case. We will also double code 20% of cases to assess inter-
rater reliability. Adherence measures: there is one measure for each of the four CBT-I treatment sessions each 
with a unique range of scores (e.g., Session 1 range = 0-23). Scores for overall adherence for a given session 
can be captured by raw scores and by transforming the total number of points earned into a “percent of content 
delivered” score. Given that adherence items code the presence or absence of events, measuring internal 
consistency is not appropriate50, 51. A reliable Competence measure will be used to assess quality of CBT-I 
treatment delivery, comprised of 4 items (pacing/focus, language, therapeutic stance, tailoring treatment) 
scored on a 3-point scale (1=needs work, 2=satisfactory work, 3=good work); total range of 4-12. For the SH 
intervention, we will code a similar number of sessions, and use nearly identical Adherence and Competence 
measures, but in the case of Adherence, we will be coding to make sure only SH elements, and not other CBT-
I elements, ae delivered.       
Site-based factors that may impact implementation will be assessed through data collection on each of the 
three study sites including: therapist factors (e.g., training/degree; CBT-I and/or sleep treatment experience; 
years of experience in primary care; years in current position; full/part time status), and site factors (e.g., 
rural/urban; number/FTE providers; patient population; turnover; number of EBTs provided; patient acuity). We 
will consider how these factors vary across sites and how they are associated with fidelity and outcomes. 
Data Analysis and Statistical Considerations 
General Considerations. A check of randomness will be conducted for each baseline demographic, 
meditational, and outcome variable collected using standardized bias estimates. Any estimate greater than .15 
will be controlled for statistically in subsequent analyses.   
Power Analysis Considerations. Mixed effect analyses will be used to examine the effects of CBT-I on the 
developmental trajectories of the primary outcomes of interest; insomnia, depression, PTSD, and SI severity. 
Here, the power approach for longitudinal mixed effects models put forth by Hedeker, et al. was used52. This 
tests for the statistical power to detect change over time (i.e., slope) using a random effects model (both slope 
and intercept). For the current proposal, necessary estimates were supplied via pilot results examining 
unconditional change in SI, depression, PTSD, and insomnia. In all instances, we are interested in the effect of 
CBT-I on the linear slope (the five time points after the intercept time point) using a two-sided alpha of .05 and 
.80 power, a 1:1 group ratio, no correlation between intercept and slope, and no correlated residual variance. 
Higher correlations between intercept and slope and among residual variance both serve to increase power 
estimates. To help improve power to detect change, two additional data collection time points have been 
added to this revised proposal53. For the outcomes of SI, depression, and insomnia severity, pre-post (6 
weeks) standardized effect sizes from the pilot study were all positive and exceeded .40. For power 
simulations, we tested for linear change from baseline through 6 month follow-up (time points 0, 6, 12, 18, 24 
and 30 weeks), testing for the power to detect an effect size of .4 at the 30 week (6 month follow-up) time 
point. A 20% attrition rate was incorporated into the model, assuming a typical longitudinal attrition pattern with 
most participants lost early (7% at post-treatment and 5%, 3%, 3%, and 2% at subsequent assessments, 
respectively). Remaining estimates were supplied via pilot results examining unconditional change in SI, 
insomnia, depression, and PTSD.  For the main outcome of SI, pilot data using the C-SSRS suggests an 
intercept variance of 10, slope variance of 1.2, and residual variance (error variance) of 24. With these 
estimates, 240 participants are required to achieve .80 power. Fewer participants are required to achieve .80 
power to detect linear change for the remaining outcomes (N = 223 for depression, N = 222, for insomnia, and 
N= 184 for PTSD) with intercept, slope and residual variances based on pilot data. 
For the main SI outcome, power likely improves with the change to the SSI as we suspect that more variance 
in this instrument will be captured in the model tested, resulting in less residual variance and a smaller residual 
variance/slope variance ratio. Both are important components of statistical power to detect change in mixed 
effects models53. 
For the structural equation model to be tested, the logic of MacCallum, Browne, and Sugawara was followed54. 
This approach allows for the testing of a null hypothesis of not good fit, reversing the role of the null hypothesis 

in conventional tests of model fit, so that a significant result provides strong support for good fit.  This approach 
also allows for an estimate of power, where effect size is defined in terms of a null and alternative value of the 
root mean square error of approximation fit index. Here, adequate power (> .98) to reject a hypothesis of close 
fit (RMSEA > .08) with 136 degrees of freedom (a degrees of freedom of 136 is a somewhat conservative 
estimate as models including more covariates would provide more degrees of freedom), given a population 
RMSEA of .05 was obtained. That is, if the true value of RMSEA = .08, and we test the hypothesis that 
RMSEA < .05, power is > .98 for rejecting the hypothesis of close fit in the population.  Additionally, adequate 
power (.98) to reject a hypothesis of not-close fit (RMSEA > .05) was also obtained. If model fit is actually 
extremely good (RMSEA < .01), and we test the hypothesis that fit is not close, we have over .95 power to 
reject the null hypothesis that RMSEA > .05. 
With respect to Exploratory Aims, the number of participants likely to complete CBT-I represent a considerable 
sample with which to conduct fidelity assessments and to identify qualitative themes and the overall sample 
size offers the ability to explore quantitative data related to participant attitudes.  
Data Analysis 
Aim 1 Hypotheses (abbreviated): CBT-I, compared to SH, will be associated with a greater rate of 
reduction over time in the severity of: (1a) insomnia; (1b) depression; (1c) PTSD; (1d) SI.  
Mixed effect analyses will be used to test for linear change in each of the above outcomes over time, as well as 
examining the effects of CBT-I on these trajectories using SAS software (SAS Institute, Inc.). Time will be 
modeled to reflect baseline status (i.e., time = 0; intercept corresponds to the start point of the trajectory) and 
time will be a count of weeks from intercept to each follow-up (i.e., 0, 6, 12, 18, 24, and 30 weeks). The effects 
of CBT-I on the intercept (start point) and on linear change will be examined (effect on intercept expected to be 
zero due to random assignment). Both intercept and slope will be treated as random effects. Effect sizes (ES) 
for each outcome, based on work by Feingold55, will also be examined. Here, estimates of the means at each 
time point for each treatment group are derived from the model and used for effect size calculations controlling 
for the influences of gender, race, ethnicity, and medication use. In addition we recognize that our assumption 
of linearity may not be met, so we are prepared to examine different trajectories of change to achieve the most 
parsimonious fit. Specifically, we will examine quadratic change, which suggests rapid change with a 
plateauing over time, and a piecewise-linear model, which joins together two or more linear trajectories to 
model non-linearity in the data.] 
Aim 2 Hypotheses (abbreviated): (2a) CBT-I effects on SI at 6 months will be both direct and 
indirect, mediated by direct intervention effects on Sleep, Depression and PTSD; (2b and 2c) 3 month 
Depression and 3 month PTSD will be positively related to 6 month SI; and (2d) post-intervention 
Sleep will be positively associated with 3 month Depression, 3 month PTSD and 6 month SI. 
Structural equation modeling (SEM) will be used to test these mediational hypotheses. The structural model to 
be tested is presented in Figure 2, illustrating key mechanisms/processes for how CBT-I influences suicide 
risk in Veterans. Modeling the processes underlying interventions, that is, addressing questions of “how” do 
interventions work, is a key benefit of SEM techniques56. As noted in the model, SI at 6 month follow-up is the 
key outcome of interest (SI Severity as assessed by the SSI total score). We hypothesize that intervention 
effects on 6 month SI are not exclusively direct, but that CBT-I effects on Sleep and Depression and PTSD 
mediate the intervention effects on SI. In the model, SI will be measured by 6 month SSI. Depression will be 
measured using two indicators from the 3 month follow up assessment, the PHQ-9 (omitting the SI and the 
sleep items) and the Beck Hopelessness Scale. PTSD will be measured using a single indicator from the 3 
month follow-up assessment, the PTSD-Checklist score (omitting the insomnia item) Sleep will be measured 
using three manifest indicators from the immediate post assessment point, the total ISI score and two sleep 
diary indicators (Percent Sleep Efficiency and Total Sleep Time). Treatment status will be measured with a 
dichotomous CBT-I group variable. The single indicator latent variables (Treatment, PTSD, and SI) will be 
modeled with the assumption that it is completely reliable (i.e., residual variance = 0). As one form of 
sensitivity, we will adjust PTSD and SI based on internal reliability estimates. 

Figure 2. Proposed structural model to be tested 
 
 
 
 
 
 
 
 
 
 
 
 
 
Key demographic variables (gender, age, minority status), site, (Rochester, Syracuse vs. Buffalo), and 
baseline values of clinical variables (SI, insomnia, nightmares, depression, PTSD, concomitant medications) 
will be statistically controlled for in the model. The Mplus statistical package will be used to test the SEM. 
To provide a metric for each latent construct and to identify the model, the first construct loading for each latent 
variable will be set at 1.0. Following Anderson and Gerbing57, a two-step approach examining the model will be 
undertaken.  First, using confirmatory factor analytic techniques, a test of the proposed measurement model 
will be conducted and any needed modifications to this model will be undertaken prior to examining the fit of 
the full model. The second step in this methodology involves an examination of the structural portion of the 
model.  As shown, the model suggests that CBT-I effects on SI Severity are both direct and indirect, mediated 
by effects on Sleep, Depression and PTSD. In the model, baseline severity of Sleep, Depression, PTSD and SI 
will be statistically controlled for, aiding precision in detecting treatment effects. Fit of the model will be 
assessed using the associated chi-square statistic, as well as the more approximate fit indices of the root 
mean square error of approximation (RMSEA), the comparative fit index (CFI), and the standardized root mean 
square residual (SRMR). Hu and Bentler58 empirically examined various cutoffs for many of these approximate 
fit indices and their data suggest that to minimize Type I and Type II errors under various conditions, one 
should use a combination of cutoffs for fit assessment. They recommend that adequate fitting models have CFI 
> .95, RMSEA < .06, and SRMR < .08; such recommendations will be adopted here. 
CBT-I indirect effects on 6 month SI severity (i.e., via Sleep, Depression, and PTSD) will be tested using the 
bias-corrected confidence limits59. Each of the three indirect effects (9 total) will be assessed for significance 
as will the overall total effect. Significance will be assessed by whether or not the 95% confidence limits 
contain zero. The bias-corrected approach takes non-normality of the multiplicative distribution into account, 
resulting in asymmetric confidence limits and has been shown to provide the most accurate confidence limits 
and greatest statistical power when compared with other existing approaches for detecting mediation59. 
Exploratory Aims: As described under Therapist Fidelity in the Measures section, we will rate both adherence 
and competence to the CBT-I intervention. Interrater reliability will be assessed using the kappa statistic and 
intraclass correlation approach. Once established >.8 agreement), we will calculate and report on the therapist 
fidelity to treatment (from adherence measures) and competence in delivering the intervention (from 
competence measures). Exit and therapist interview data will be assessed for thematic content by using a 
conventional content analysis approach60 to avoid imposing preconceived themes/categories on the data. 
Inductive coding60 will be used by two research staff trained by Dr. Cross to listen to the interviews and develop 
preliminary sets of codes. The two research staff will independently rate one interview at a time, meet to 
discuss their coding process for that interview, and reach consensus on codes created and applied until 
NOTE:         denotes residual variance unexplained in the model.  Residual covariance among 
Depression and PTSD as well as demographic covariates and medication use have been omitted 
for presentation clarity (see text). FU = Follow-Up; HS = Beck Hopelessness Scale; ISI = Insomnia 
Severity Index; SE% = Percent Sleep Efficiency; PHQ9 = Patient Health Questionnaire; SSI = 
Scale for Suicidal Ideation; TST = Total Sleep Time.  
Baseline  
Post-Treatment 
      3 Month FU                 6 Month FU  

agreement on code definitions can be used consistently. All analytic decisions will be documented. Remaining 
interviews will be analyzed independently, but regular meetings will be held to discuss the evolving analysis 
and how similar codes cluster together. Once complete, discussion of the codes within and across categories 
of data and interpretation of themes across categories will occur. Themes, categories, and specific codes that 
emerge can be used to inform any subsequent implementation efforts. 
If treatment attitudes instruments are normally distributed, ANCOVA will be used to test for significant 
treatment effects from baseline to post-treatment among those participants not engaged in psychotherapy at 
baseline, otherwise non-parametric tests will be applied (with covariates statistically controlled).  
Missing Data. Mixed effects models and SEM incorporating full-information maximum likelihood make efficient 
use of all available data; we will also conduct analyses using multiple imputation of missing data as a form of 
sensitivity analysis. Multiple imputation uses a regression based approach to impute values for data that are 
missing, providing that data meets the missing at random criteria. To improve estimates, the procedures put 
forth by Allison61 regarding intervention studies will be followed. Here, each treatment condition is imputed 
separately and resulting datasets are merged. Following Rubin62, ten imputations will be performed and each 
resulting datasets will be analyzed as above. The results obtained will be combined following the techniques 
employed by Rubin62. Multiple imputation has consistently demonstrated less biased parameter estimates than 
most other traditional approaches to the handling of missing data63. If missingness appears biased by 
treatment condition, a pattern-mixture approach to missing data will be assessed64. Here, different patterns of 
missingness are identified and participants are placed into groups based on their missingness pattern. Dummy 
variables representing group membership are entered into the mixed effect and SEM analyses; intervention by 
group membership interaction terms are assessed to examine for differential effects on outcomes dependent 
on missing data patterns.  
The SEM to be tested will make use of the full-information maximum likelihood estimation method, which uses 
estimates throughout the model to provide more accurate estimates where data is missing. Again, this 
approach provides more realistic parameter estimates than other missing data techniques65 when data is 
missing at random or missing completely at random.   
Type 1 Error Protection.  To help control for Type 1 error, the Benjamini-Hochberg (BH) method will be used to 
adjust for the multiple comparisons proposed in the current study66. The BH method adjusts for multiple 
comparisons by controlling false discovery rate instead of family-wise error rate.  It is less conservative than 
the more traditional Bonferonni methods, yet still provides adequate protection against Type 1 error.  Since its 
inception, there has been growing evidence suggesting that the false discovery rate based BH method is the 
optimal solution to the multiple comparison problem in most practical situations67. 
Patient Safety 
All Veterans will be instructed in the informed consent and at the initiation of the SH or CBT-I intervention of 
the limits of confidentiality and the fact that if we feel they may be in imminent danger of harming themselves 
that we will involve other medical providers/emergency staff to protect their safety. All Veterans will be regularly 
assessed for SI throughout the study and received emergency contact information for Dr. Pigeon and Dr. 
Funderburk as well as the suicide hotline information at the initial session (see suicide protocol). 
Information on VINCI. VINCI is a secure, central analytic platform and includes a cluster of servers designed to 
host databases integrated from national VA data sources, such as the Corporate Data Warehouse. This data 
includes the … from the sites identified in this protocol. Once approved, the data will be placed within this 
secure environment of VINCI so only IRB-approved staff can access the data. To ensure the protection of 
Veteran data, VINCI maintains compliance with the guidelines set forth by the VA Handbook 1200.12 and all 
other applicable VA and VHA policies and regulations. All data will remain within the VINCI environment. 
Data Safety. To ensure safety of participants in the study proposed and validity and integrity of data collected, 
the PIs will oversee all data and safety monitoring functions. Both PIs will assume responsibility for these 
activities, but the research team will be advised that Dr. Funderburk will be the primary contact PI overseeing 
these activities. Drs. Funderburk and Pigeon will meet regularly to monitor study progress and discuss the 
implementation of monitoring procedures. Dr. Funderburk will meet regularly with the research coordinator and 
staff to review monitoring procedures and ensure all efforts are being taken to minimize risks to participants. 
To help monitor safety issues, we will have an independent Data Safety Monitoring Board (DSMB). As 
described below, we will track all negative outcomes and incidents as well as conduct interim data analysis 

every 12 months after the study has started. The study design will be significantly modified (and even 
screening stopped) on the advice of the DSMB that the study is creating potential harm to our participants.  
Both PIs will regularly oversee all aspects of the study, including participant recruitment, informed consent, 
data collection, management, and analysis, as well as regularly assess the risk/benefit ratio associated with 
participation in the study. As a result, all research staff will participate in an intensive training to help them 
understand the importance of reducing the risk for participants and learning how to recognize and report any 
adverse event or serious adverse event to Drs. Pigeon or Funderburk. Serious adverse events (SAE) may 
include: death, hospitalization due to worsening depressive symptoms or suicidal ideation, or all life threatening 
or disabling/incapacitating events among research participants. Adverse events may include (but is not limited 
to): physical injuries, worsened physical or mental health, or inadvertent disclosure of confidential research 
information. 
In the event of a SAE, the co-PIs will immediately communicate with the DSMB and Institutional Review 
Boards, followed by a written report in 24 hours. Jointly, we will make a decision whether there is sufficient 
evidence to necessitate suspension of data collection, further IRB review, modification of the protocol, or other 
changes to reduce potential risk to participants. Resumption of the study shall be based on the concurrence of 
the co-PIs, the chairperson of the IRBs, and the DSMB.  
In the event that an adverse event that is not an SAE is reported to the co-PIs, they will discuss the event with 
the DSMB. Immediate evaluation will occur to determine if any extra steps can be taken to minimize the 
likelihood of that type of adverse event occurring again. If changes can be made, a report/amendment will be 
written and submitted to the DSMB and Institutional Review Boards.  
As part of a standard practice within our Centers, Dr. Funderburk will supervise the implementation of one 
audit within 4 months after study recruitment and one regularly per year afterwards of the materials collected 
and produced as part of the study at each site to ensure proper confidentiality and compliance with ethical 
principles, including informed consents, electronic medical record documentation, questionnaire data, and to 
make sure that the research staff are following established protocols.  
Dr. Funderburk, with support from Dr. Pigeon, will provide an annual summary report of all adverse events to 
the IRB and the DSMB as part of the annual review. If no adverse events have occurred, the report will state, 
“No adverse events affecting human participants have occurred during this project year.”    
Data Monitoring. To ensure adequate participant recruitment and enrollment, the co-PIs will weekly discuss the 
current numbers of participants contacted, screened, and enrolled from each site and compare those numbers 
to the expected based on our preliminary data. If after the first 4 months, it appears we are not reaching our 
expected N’s, the co-PIs will discuss potential barriers/obstacles and solutions with the DSMB, including the 
option of extending to other outpatient clinics associated with the primary medical centers, if necessary. 
Discussions regarding recruitment and enrollment will continue at each meeting with the DSMB to ensure 
proper implementation of the study. 
Data Safety Monitoring Board. The purpose of the DSMB is to review protocols and consent documents for this 
study, monitor safety issues throughout the study, provide an overview of the quality of the accumulating data, 
and provide guidance on interim analyses and stopping rules. 
The DSMB will be comprised of 3 individuals with no direct involvement in the study or conflict of interest with 
the research team conducting the clinical trial. The DSMB will include individuals with expertise in: research 
and monitoring at-risk research participants; research in longitudinal clinical trials with Veterans; and research 
expertise with mental health and implementing collaborative care models. The DSMB responsibilities will 
include:  
i. 
Review and approve, disapprove, or suggest modifications to the study protocols and/or consent 
documents to assure both scientific integrity and study adherence to human subject protection policies. 
ii. Monitor, provide feedback, and report on scientific and ethical issues related to study implementation 
for the protection of human subjects and advise on ethical issues related to adverse events. The DSMB 
will monitor adverse event reports for purposes of determining whether their nature, frequency and 
severity are consistent with expectations.  
The DSMB, in coordination with the co-PIs, will report to the VA IRBs any unanticipated problems 
involving risks to subject.  

The DSMB can recommend remedies or other appropriate actions such as introducing new monitoring 
protocols, altering inclusion or exclusion criteria, or recommending changes in the informed consent 
documents.   
iii. Ensure that the study protocol maintains patients’ confidentiality in a manner that is appropriately 
balanced with issues of clinical care and safety. 
iv. Monitor data regarding effectiveness. The DSMB will review data for outcomes by treatment group. If 
differences in results between groups appear to be clinically significant, the DSMB will review whether 
the clinical trial should continue with or without further enrollment of new subjects. The DSMB has the 
authority to halt the trial as needed. 
v. Monitor data management activities. The DSMB may ask to review data relevant to quality control. The 
DSMB will review requests for interim analyses and approve, disapprove, require additional information, 
or defer decisions. 
At a minimum, the DSMB will convene on an annual basis throughout the study DSMB meetings will be 
held in-person or via video teleconference. The Board Chair and co-PIs will decide upon the format of the 
meetings. Additional meetings or telephone conferences will be held on the recommendation of the DSMB. 
The Board Chair and the PIs will determine meeting logistics based upon clinical urgency and the availability of 
DSMB members. 
The PIs will submit reports to the DSMB one week prior to the scheduled meeting. These reports will include all 
reported data up to and including 14 days prior to the reporting deadline (except for SAEs, which are to be 
reported within 24 hours of an event). For each meeting at which the study is to be considered or monitored, 
the PIs will present an overall progress statement. This brief statement will contain the assurance that the 
study investigators have considered the clinical trial’s progress and that there is/is not evidence of safety 
issues that should be addressed by the DSMB. The reports will also include information related to problems in 
recruitment, biases in attrition, biases with respect to AEs, or other operational problems that affect the integrity 
of the study. At the additional request of the DSMB, interim analyses will be conducted to determine whether 
the emerging pattern of findings may lead to deliberation of prematurely discontinuing the trial. 
The DSMB will be kept apprised of all SAE’s and AE’s on an ongoing basis and will serve as the final arbiters 
of whether individual patients should be removed from the protocol. Although research staff, under the 
supervision of the PIs and co-Is, are empowered to take whatever immediate action is necessary to safeguard 
the welfare of individual patients, the DSMB will be called upon whenever possible to render judgments in the 
advent of a serious adverse event. We acknowledge that there may be rare instances where some emergent 
situation occurs that was unanticipated regarding the welfare of the participant. In these situations, the VA IRB 
or the DSMB may be contacted to help resolve the situation. 
Data Security and Privacy Review 
All signed informed consents will be stored separately from any paper data that has been de-identified and 
under double locked conditions at the Center of Excellence or Center for Integrated Healthcare research 
offices at the Syracuse VA Medical Center (D409), Rochester Outpatient Clinic (rm 3809), Canandaigua CoE 
(Building 37), and CIH offices in Buffalo VAMC (building 20, room 128 or room 108) during the time that 
participants are active in the study. Paper and electronic data will be de-identified using random identification 
numbers. Paper data will also be stored separately under double locked conditions at the same offices 
identified above. Electronic data will be stored on a VISN 2 secure server in password protected files only IRB 
approved staff have access to. Electronic data obtained from VINCI will be stored on VA secure VINCI servers 
until it is downloaded to the local VA secure server (T: drive). Although there is no plan to have identifiable data 
on the audio recordings, we will collect audio recordings using a Philips DPM-8000 pocket memo digital 
dictation recorder (FIPS compliant) at the VA location and immediately upload to a VISN 2 secure server and 
delete them from the recording devices. The only transporting of identifiable paper data (e.g., consent forms) 
will occur from research rooms within the VA facility into the CIH and CoE research offices at the same facility. 
Identifiable electronic data may be downloaded from/uploaded to the secure VINCI server or sent to IRB-
approved staff via encrypted VA email for data analysis. All study staff are up to date with VA Privacy and 
Information Security and Rules of Behavior training. No identifiable nor de-identified data is being removed 
from the VA protected environment. All data will be stored following the current Federal Records Control 
Schedule. 
 

 
Reference List 
 
 
(1)  Department of Veterans Affairs. Report of the Blue Ribbon Work Group on Suicide Prevention in the Veteran 
Population. Washington D.C.: US Department of Veterans Affairs; 2008.  
 
(2)  U.S.Department of Defense. Department of Defense task Force on the Prevention of Suicide by the Members of 
the Armed Services. Washington D.C.: Department of Defense; 2010.  
 
(3)  Katz I. Lessons learned from mental health enhancement and suicide prevention activities in the Veterans Health 
Administration. Am J Public Health 2012;102 Suppl 1:S14-S16. 
 
(4)  Knox KL, Bossarte RM. Suicide prevention for veterans and active duty personnel. Am J Public Health 2012;102 
Suppl 1:S8-S9. 
 
(5)  Pigeon WR, Pinquart M, Conner K. Meta-analysis of sleep disturbance and suicidal thoughts and behaviours. J 
Clin Psychiatry 2012;73(9):1160-1167. 
 
(6)  Kessler RC, Borges G, Walters EE. Prevalence of and risk factors for lifetime suicide attempts in the National 
Comorbidity Survey. Archives of General Psychiatry 1999;56:617-625. 
 
(7)  Mann JJ, Apter A, Bertolote J et al. Suicide prevention strategies: a systematic review. JAMA 2005;294:2064-
2074. 
 
(8)  Preuss UW, Schuckit MA, Smith TL et al. Predictors and correlates of suicide attempts over 5 years in 1,237 
alcohol-dependent men and women. American Journal of Psychiatry 2003;160:56-63. 
 
(9)  Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and Suicidal Behavior. Epidemiologic 
Reviews 2008;30(1):133-154. 
 (10)  Kessler RC, Nock MK. Prevalence of and risk factors for suicide attempts versus suicide gestures: analysis of the 
National Comorbidity Survey. Journal of Abnormal Psychology 2006;115:616-623. 
 (11)  Cavanagh JTO, Carson AJ, Sharpe M, Lawrie SM. Psychological autopsy studies of suicide: a systematic review. 
Psychol Med 2003;33(3):395-405. 
 (12)  Ilgen MA, Bohnert ASB, Ignacio RV et al. Psychiatric diagnoses and risk of suicide in veterans. Arch Gen 
Psychiatry 2010;67(11):1152-1158. 
 (13)  Britton P, Ilgen MA, Valenstein M, Knox KL, Claasen C, Conner KR. DIfferences between veteran suicides with 
and without psychiatric symptoms. Am J Public Health 2012;102:S125-S130. 
 (14)  Lemaire C, Graham DP. Factors associated with suicidal ideation in OEF/OIF veterans. Journal of Affective 
Disorders 2011;130:231-238. 
 (15)  Screening for Depression in Adults: US Preventive Services Task Force Recommendation Statement. Ann Intern 
Med 2009;151(11):784-W256. 
 (16)  VA/DoD Clinical Practice Working Group. Management of post-traumatic stress. Washington D.C.: Department of 
Veterans Affairs and Health Affairs, Department of Defense, Office of Quality and Performance Publication; 2003.  
 (17)  Department of Veterans Affairs. Mental Health Quality Enhancement Research Initiative (MH-QUERI) Strategic 
Plan.  2006.  
Ref Type: Generic 
 (18)  Cuijpers P, de Beurs DP, van Spijker BA, Berking M, Andersson G, Kerkhof AJ. The effects of psychotherapy for 
adult depression on suicidality and hopelessness: A systematic review and meta-analysis. J Affect Disord 
2013;144(3):183-190. 
 (19)  Shepardson RL, Funderburk JS, Pigeon WR, Maisto SA. Insomnia treatment experience and preferences among 
veterans affairs primary care patients. Mil Med 2014;179(10):1072-1076. 
 (20)  Possemato K, Ouimette P, Lantinga L et al. Treatment of Department of Veterans Affairs primary care patients 
with posttraumatic stress disorder. Psychological Services 2011;8(2):82-93. 
 (21)  Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a 
review of the evidence. Am J Psychiatry 2002;159(6):909-916. 
 (22)  Brockman L, Lsaky E, Mach J, Stephen J. Findings from the 2012 National PC-MHI Evaluation Survey. 
Washington DC; 2015.  
 (23)  Zeiss AM, karlin BE. Integrating mental health and primary care services in the Department of Veterans Affairs 
health care system. J Clin Psychol Med Settings 2008;15(1):73-78. 
 (24)  Funderburk JS, Shepardson RL. Recent advances in primary care behavioral  
 health. Current Opinion in Psychology 2015;5:37-41. 
 (25)  Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to 
practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003;93(8):1261-1267. 
 (26)  Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining 
elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 
2012;50(3):217-226. 
 (27)  Proctor E, Silmere H, Raghavan R et al. Outcomes for implementation research: conceptual distinctions, 
measurement challenges, and research agenda. Adm Policy Ment Health 2011;38(2):65-76. 

 (28)  Malik S, Kanwar A, Sim LA et al. The association between sleep disturbances and suicidal behaviors in patients 
with psychiatric diagnoses: a systematic review and meta-analysis. Syst Rev 2014;3:18. 
 (29)  Bishop TM, Britton P, Knox KL, Pigeon WR. Cognitive behavioral therapy for insomnia and imagery rehearsal in 
combat veterans with comorbid posttraumatic stress: A case series. Military Behavioral Health 2016;4(1):58-64. 
 (30)  Pigeon W.R., Funderburk J. Delivering a brief insomnia intervention to depressed VA primary care patients. 
Cognitive Behavioral Practice 2014;21:252-260. 
 (31)  Posner K, Brown GK, Stanley B et al. The Columbia-Suicide Severity Rating Scale: Initial Validity and Internal 
Consistency Findings From Three Multisite Studies With Adolescents and Adults. Am J Psychiatry 
2011;168(12):1266-1277. 
 (32)  Koons CR, Robins CJ, Tweed LJ et al. Efficacy of dialectical behavior therapy in women veterans with borderline 
personality disorders. Behavior Therapy 2001;32:371-390. 
 (33)  Weitz E, Hollon SD, Kerkhof A, Cuijpers P. Do depression treatments reduce suicidal ideation? The effects of 
CBT, IPT, pharmacotherapy, and placebo on suicidality. J Affect Disord 2014;167:98-103. 
 (34)  Cross W, Pisani A, Schmeelk-Cone K, Xia Y. Fidelity assessment of the transfer of training in a suicide prevention 
program for crisis hotlines. Crisis 2014;35(3):202-212. 
 (35)  Cross W, West J, Wyman PA, Schmeelk-Cone K. Observational measures of implementer fidelity for a school-
based preventive intervention: development, reliability and validity. Prev Sci 2015;16(1):122-132. 
 (36)  Schoenwald SK, Garland AF. A review of treatment adherence measurement methods. Psychol Assess 
2013;25(1):146-156. 
 (37)  Kroenke K, Spitzer RL, Williams JBW. The PHQ-9 - Validity of a brief depression severity measure. J Gen Intern 
Med 2001;16(9):606-613. 
 (38)  Bastien C, Vallieres, Morin CM. Validation of the Insomnia Severity Index as an outcome measure for Insomnia 
research. Sleep Medicine 2001;2:297-307. 
 (39)  Bovin MJ, Marx, BP, Weathers, FW et al. Psychometric properties of the PTSD Checklist for Diagnostic and 
Statistical Manual of Mental Disorders - Fifth Edition (PCL-5) in veterans. Psychol Assess 2016; 28(11): 1379-
1391. 
 (40)  Chung F, Subramanyam R, Liao P, Sasaki E, Shapiro C, Sun Y. High STOP-Bang score indicates a high 
probability of obstructive sleep apnoea. Br J Anaesth 2012;108(5):768-775. 
 (41)  Manber R, Carney CE, Edinger J et al. Dissemination of CBT-I to the non-sleep specialist:  Protocol development 
and training issues. J Clin Sleep Med 2012;8(2):209-218. 
 (42)  Sheehan DV, Lecrubier Y, Sheehan KH et al. The validity of the Mini International Neuropsychiatric Interview 
(MINI) according to the SCID-P and its reliability. European Psychiatry 1997;12(5):232-241. 
 (43)  Beck AT, Kovacs M, Weissman A. Assessment of suicidal intention: the Scale for Suicide Ideation. J Consult Clin 
Psychol 1979;47(2):343-352. 
 (44)  Brown GK, Beck AT, Steer RA, Grisham JR. Risk factors for suicide in psychiatric outpatients: A 20-year 
prospective study. Journal of Consulting and Clinical Psychology 2000;68:371-377. 
 (45)  Bell JB, Nye EC. Specific symptoms predict suicidal ideation in Vietnam combat veterans with chronic post-
traumatic stress disorder. Mil Med 2007;172(11):1144-1147. 
 (46)  Beck AT, Steer RA. Manual for the Beck Hopelessness Scale. San Antonio: Psychological Corporation; 1988. 
 (47)  Linehan MM, Comtois KA, Brown MZ, Heard HL, Wagner A. Suicide Attempt Self-Injury Interview (SASII): 
development, reliability, and validity of a scale to assess suicide attempts and intentional self-injury. Psychol 
Assess 2006;18(3):303-312. 
 (48)  Fischer EH, Farina A. Attitudes toward seeking professional psychological help: a shortened form and considerations for 
research. Journal Study Development 1995;36:368-373. 
 (49)  Kushner MG, Sher KJ. Fear of psychological treatment and its relation to mental health service avoidance. Professional 
Psychology: Research and Practice 1989;20(251):257. 
 (50)  Smith GT, McCarthy DM. Methodological considerations in the refinement of clinical assessment instruments. 
Psychological Assessment 1995;7(3):300-308. 
 (51)  Streiner DL. Being inconsistent about consistency: When coefficient alpha does and doesn't matter. Journal of Personality 
Assessment 2003;80(3):217-222. 
 (52)  Hedeker D, Gibbons RD, Waternaux, C. Sample size estimation for longitudinal designs with attrition:  Comparing 
time-related contrasts between two groups. J Educ Beh Statistics 1999;24(1):70-93. 
 (53)  Rast P, Hofer SM. Longitudinal design considerations to optimize power to detect variances and covariances 
among rates of change: simulation results based on actual longitudinal studies. Psychol Methods 2014;19(1):133-
154. 
 (54)  macCallum RC, Browne MW, Sugawara HM. Power analysis and determination of sample size for covariance 
structure modeling. Psychological Methods 1996;1(2):130-149. 
 (55)  Feingold A. Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for 
classical analysis. Psychol Methods 2009;14(1):43-53. 
 (56)  Hennessy M, Greenberg J. Bringing it all together: Modeling intervention processes using structural equation 
modeling. American Journal of Evaluation 1999;20(3):471-480. 

 (57)  Anderson JC, Gerbing DW. Structural equation modeling in practice: A review and recommended two-step 
approach. Psychol Bull 1988;103(3):411-423. 
 (58)  Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new 
alternatives. Structural Equation Modeling 1999;6(1):1-55. 
 (59)  MacKinnon DP, Lockwood CM, Williams J. Confidence Limits for the Indirect Effect: Distribution of the Product 
and Resampling Methods. Multivariate Behav Res 2004;39(1):99. 
 (60)  Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15(9):1277-
1288. 
 (61)  Allison PD. Missing Data. Newbury Park: Sage; 2001. 
 (62)  Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York: John Wiley & Sons; 1987. 
 (63)  Graham JW. Missing data analysis: making it work in the real world. Annu Rev Psychol 2009;60:549-576. 
 (64)  Hedeker D, Gibbons RD. Application of random-effects pattern-mixture models for missing data in longitudinal 
studies. Psychological Methods 1997;2(1):64-78. 
 (65)  Arbuckle JL. Full information estimation in the presence of incomplete data. In: Marcoulides GA, Schumacker RE, 
editors. Advanced structural equation modeling: Issues and techniques.Mahwah, NJ: Erlbaum; 1996. 243-277. 
 (66)  Benjamini Y, Hochberg Y. Controlling the False Discovery Rate - A Practical and Powerful Approach to Multiple 
Testing. Journal of the Royal Statistical Society Series B-Methodological 1995;57(1):289-300. 
 (67)  Williams VSL, Jones LV, Tukey JW. Controlling error in multiple comparisons, with examples from state-to-state 
differences in educational achievement. Journal of Educational and Behavioral Statistics 1999;24:42-69.