Improving Adherence to Evidence-Based Practice Using an Innovative and Easy-to-Use Health IT Solution
Phase: NA
Conditions: Anxiety, Depression, Depression/Anxiety
Sponsor: University of South Florida

 
 
 
 
 
PROTOCOL TITLE: 
Improving adherence to evidence-based practice using an innovative and easy-to-use 
health IT solution 
 
PRINCIPAL INVESTIGATOR: 
Brian E. Bunnell 
Department of Psychiatry and Behavioral Neurosciences 
813-974-8607 
bbunnell@usf.edu 
 
VERSION NUMBER/DATE: 
V10. February 13, 2024 
 
REVISION HISTORY 
*This table should only be used during submission of a Modification application to the IRB.  
 
Revision 
# 
Version Date 
Summary of Changes 
Consent 
Change? 
1 
July 8, 2021 
We will conduct 90-minute focus groups 
instead of 60-minute individual interviews for 
Aim 1, resulting in a change in sample size 
from 60 to 72 for this aim; we revised the 
STTR Protocol, STTR Aim 1 Interviews 
Verbal Consent form, and STTR Aim 1 Focus 
Group Recruitment Email to reflect these 
changes; we updated the Aim 1 interview for 
focus groups; we uploaded Aim 2 trial 
questionnaires as separate word documents; we 
renamed documents to include the version 
number. 
Yes 
2 
August 24, 
2021 
In addition to audio recording the focus groups 
for Aim 1, we will also screen-record. We 
revised the STTR protocol and STTR Aim 1 
Focus Group Verbal Consent form to reflect 
these changes. 
Yes 
3 
December 10, 
2021 
We revised the protocol to include new focus 
group recruitment strategy 
No 
4 
January 13, 
2022 
We revised the protocol to include additional 
focus group strategy. We uploaded STTR Aim 
1 Facebook Info Card for Patient and STTR 
Aim 1 Focus Group Facebook Recruitment Ad 
for Facebook recruitment. 
No 

 
Page 2 of 30 
Revised: 2/13/2024 
5 
January 24, 
2022 
Revised protocol to increase provider sample 
size to 30 and patient sample size to 60 
Revised the following consent forms to update 
sample sizes: STTR Aim 2 Patients Trial 
Consent, STTR Aim 2 Providers Trial Consent 
Yes 
6 
July 18, 2022 
Revised protocol to include the re-consent 
procedure if a study client changes therapists, 
include option for families to be able to opt out 
of audio recording their therapy sessions and 
the Assessment of Core CBT Skills (ACCS) to 
assess provider’s CBT delivery; increase 
provider sample size to 40 and client referral 
request to 3; and update provider compensation 
to be sent after family enrollment. We revised 
the STTR Aim 2 Patients Trial Consent, STTR 
Aim 2 Providers Trial Consent to reflect these 
changes; we updated STTR Aim 2 Info card for 
Providers, STTR Aim 2 Provider Patient 
Referrals Email, and STTR Aim 2 Audio 
Recordings Upload Email; we uploaded Study 
Introduction Script and STTR Aim 2 Trial 
ACCS. 
 
Yes 
7 
September 20, 
2023 
Revised protocol to increase provider sample 
size to 50. We revised the STTR Aim 2 
Patients Trial Consent and the STTR Aim 2 
Providers Trial Consent to reflect these 
changes. 
Yes 
8 
January 22, 
2024 
Uploaded STTR Aim 2 Trial Post-
Implementation Client Interview Guide and 
STTR Aim 2 Trial Post-Implementation 
Therapist Interview Guide for post-
implementation interviews. 
No 
9 
February 13, 
2024 
Revised protocol to increase provider’s post-
implementation interview time to 45 minutes. 
Currently enrolled provider participants have 
already consented to the interview when they 
first consented/enrolled in the study, so they 
will not be reconsented, but study staff will 
verbally update them on the new time 
commitment. Therapists can choose to 
decline/withdraw if they want. Revised the 
following consent form to update new time 
commitment for newly enrolled participants: 
STTR Aim 2 Providers Trial Consent 
Yes 
 
 

 
Page 3 of 30 
Revised: 2/13/2024 
Table of Contents 
1.0 
Study Summary ....................................................................................................... 4 
2.0 
Objectives ............................................................................................................... 5 
3.0 
Background ............................................................................................................. 5 
4.0 
Study Intervention ................................................................................................... 8 
5.0 
Procedures Involved .............................................................................................. 11 
6.0 
Data and Specimen Storage for Future Research ................................................. 15 
7.0 
Sharing of Results with Subjects .......................................................................... 15 
8.0 
Study Timelines .................................................................................................... 15 
9.0 
Inclusion and Exclusion Criteria ........................................................................... 16 
10.0 
Vulnerable Populations ......................................................................................... 17 
11.0 
Local Number of Subjects .................................................................................... 17 
12.0 
Recruitment Methods ............................................................................................ 17 
13.0 
Withdrawal of Subjects ......................................................................................... 18 
14.0 
Risks to Subjects ................................................................................................... 18 
15.0 
Potential Benefits to Subjects or Others ............................................................... 18 
16.0 
Data Management and Confidentiality ................................................................. 19 
17.0 
Provisions to Monitor the Data to Ensure the Safety of Subjects ......................... 22 
18.0 
Provisions to Protect the Privacy Interests of Subjects ......................................... 22 
19.0 
Compensation for Research-Related Injury .......................................................... 22 
20.0 
Subject Costs and Compensation .......................................................................... 22 
21.0 
Consent Process .................................................................................................... 23 
22.0 
Setting ................................................................................................................... 24 
23.0 
References ............................................................................................................. 24 
 
 

 
Page 4 of 30 
Revised: 2/13/2024 
1.0 
Study Summary  
1.1 
Study Title 
Improving adherence to evidence-based practice using an 
innovative and easy-to-use health IT solution 
Study Design 
Randomized trial methodology with assessments 
Primary 
Objective/Purpose 
The purpose of this study is to expand Adhere.ly–a web-
based platform to improve provider implementation and 
patient engagement in homework (i.e., between-session 
practice of skills learned during therapy) during mental 
health treatment–to include new features and therapeutic 
exercises for adult patients, develop a plan for its 
implementation and sustainment, and preliminarily evaluate 
it by conducting a feasibility Optimization, Effectiveness, 
and Implementation (OEI) Hybrid trial. 
Secondary 
Objective(s)/Purposes 
N/A 
Research 
Intervention(s)  
Therapy 
ClinicalTrials.gov 
NCT # 
In progress. 
Study Population 
Masters-level mental health providers; Treatment seeking 
adults with anxiety and/or depression 
Sample Size 
N=282 
Aim 1 focus groups: n=24 patients, 24 providers, 24 
administrators  
Aim 1 survey: n=100 providers 
Aim 2 trial: n=50 providers; 60 adult patients 
Study Duration for 
individual subjects 
Aim 1 focus groups: 90-minutes each 
Aim 1 survey: 15 minutes 
Aim 2 trial providers: 28 hours over 9 months 
Aim 2 trial patients: 17 hours over 3 months 
Study Specific 
Abbreviations/ 
Definitions  
OEI=Optimization, Effectiveness, and Implementation 
I/S=Implementation/sustainment 
CBT=Cognitive Behavioral Therapy 
EBT=Evidence-Based Mental Health Treatment 
mCFIR=Modified Consolidated Framework for 
Implementation Research 
 
 

 
Page 5 of 30 
Revised: 2/13/2024 
2.0 
Objectives 
2.1  
The purpose of this study is to expand Adhere.ly (https://adhere.ly/)–a web-based 
platform to improve provider implementation and patient engagement in homework (i.e., 
between-session practice of skills learned during therapy) during mental health 
treatment–to include new features and therapeutic exercises for adult patients, develop a 
plan for its implementation and sustainment, and preliminarily evaluate it by conducting 
a feasibility Optimization, Effectiveness, and Implementation (OEI) Hybrid trial. 
 
Aim 1. Understand and prioritize key stakeholder goals, challenges, 
affordances, and constraints  
We will use an exploratory sequential (qual→quant) mixed-methods design to 
conduct semi-structured focus groups (qual) with mental health patients, 
providers, and administrators and a survey (quant) with providers. We will 
conduct 4 focus groups per stakeholder group (patient, provider, administrator) 
and each focus group session will include up to 6 participants. We will enroll 100 
providers to complete the survey. Analyses will inform the design of new 
Adhere.ly features and exercises and an implementation/sustainment (I/S) 
blueprint. 
 
Aim 2. Optimize, preliminarily evaluate, and implement Adhere.ly in 
community practice settings 
We will conduct a small-scale feasibility OEI Hybrid trial with 30 community mental 
health providers randomized to Cognitive Behavioral Therapy (CBT) alone (n=25) vs. 
CBT+Adhere.ly (n=25), and 60 treatment seeking and provider-referred adult patients 
with clinically elevated anxiety and/or depression (outcomes), which we will assess at 
baseline and 3-months post-baseline. We will assess homework use by providers and 
adherence of patients (targets) weekly via session audio recordings and patient-report. 
We will optimize new Adhere.ly features and exercises, and our I/S blueprint during the 
trial and assess implementation outcomes post-implementation. 
 
2.2  
This is a feasibility trial and is not powered for hypothesis testing. 
 
3.0 
Background 
3.1  
Homework is one of the most integral components of high-quality mental health 
treatment 
Mental health disorders affect 1 in 6 youth and 1 in 5 adults in the U.S. and are associated 
with costly physical and behavioral health problems. The quality of services these 
patients receive vs. should receive is highly variable and characterized as a “quality 
chasm” by the IOM.1 Homework, or between-session practice of skills learned during 
therapy, is one of the most integral, yet underutilized components of high-quality, 
evidence-based mental health treatments (EBTs) such as Cognitive Behavioral Therapy 
(CBT).2,3 Homework exercises are assigned by providers in-session and completed by 
patients between sessions with the goal of practicing therapeutic skills in the environment 

 
Page 6 of 30 
Revised: 2/13/2024 
where they will be needed most.4 Homework enables the generalization of skills and 
behaviors learned during therapy, facilitates treatment processes, provides continuity 
between sessions, allows providers to better grasp patients’ learning, and strengthens that 
learning, leading to improved maintenance of treatment gains.4–6 Meta-analytic and 
systematic reviews have shown that homework use by providers and adherence by 
patients predict increased treatment engagement, decreased treatment dropout, and 
medium-to-large effects on improvement in clinical outcomes (Cohen’s d=.45-.77).7–11 
Simply put, 68% vs. 32% of patients can be expected to improve when therapy involves 
homework.9 
 
Innovative solutions are needed to address barriers to the implementation of 
homework 
Despite its many benefits, homework is implemented with variable effectiveness in “real 
world” clinical settings. Only 68% of general mental health providers and ~55% of 
family providers report using homework “often” to “almost always”.12,13 Providers report 
using homework in an average of 57% of sessions and only 25% of providers report 
using expert-recommended systematic procedures for implementing homework (i.e., 
specifying frequency, duration, and location; writing down homework assignments for 
patients).14 A national survey revealed that 93% of mental health providers estimate rates 
of patient adherence to homework to be low to moderate,12 and studies generally report 
low to moderate rates of patients’ homework adherence.15,16 There are many barriers to 
successful homework implementation. For example, many providers struggle to 
consistently develop, assign, and assess homework exercises with their patients, and 
many patients have difficulty remembering to practice skills in an correct and timely way 
that fosters adequate learning.12–21 
 
Technology is ubiquitous and can address homework barriers, but more research is 
needed 
Between 92-96% of adults aged 18-49 years own a smartphone and 99% own a 
cellphone.22,23 Many health IT resources are effective, practical, desired by patients and 
providers, and available at low cost.24 Prior work by our team and others suggests that 
health IT solutions have tremendous potential to positively affect homework use and 
adherence and as a result, the quality of mental health treatment.21,25,26 Some existing 
health IT resources include features to support homework implementation (e.g., voice and 
SMS reminders and feedback, self-monitoring, assessment),25,26 and some mHealth apps 
with homework-specific resources have been developed with positive preliminary 
effects.27–33 However, these resources are generally age-, disorder-, and treatment 
protocol-specific, and solely native app- based with limited interaction between patient 
and provider interfaces. Further, more data are needed to ensure that these resources have 
the ability to target homework use by providers and adherence by patients and be 
implemented in community practice settings. 
 
Adhere.ly can address homework barriers and limitations of existing health IT 
resources  
To address these limitations, our team of experts at Adhere.ly, LLC, USF, and the 
Medical University of South Carolina (MUSC), and Doxy.me, LLC developed Adhere.ly, 

 
Page 7 of 30 
Revised: 2/13/2024 
an innovative, state-of-the-art, and user-friendly solution to improve provider 
implementation and patient engagement in homework during EBTs. Adhere.ly is a free 
web-based platform with optional provider login, patient management features, and built-
in therapeutic exercises (e.g., self-monitoring, relaxation, cognitive coping, emotion 
regulation, exposure therapy) for providers to introduce, practice, and assign as 
homework to patients. Patients receive SMS or email reminders with links to practice 
those exercises during the week on the days/times specified by providers, and providers 
are able to review patients’ homework adherence and relevant data (e.g., self-monitoring 
ratings) on their dashboard. Adhere.ly was developed as a resource to improve homework 
implementation by providers and adherence in children and caregivers during treatment 
for childhood PTSD (F32 MH108250; K23 MH118482; PI Bunnell). One of the 
objectives of this NIMH Phase I STTR project, conducted in partnership with Adhere.ly, 
LLC, is to expand Adhere.ly to include additional features (e.g., more guidance for 
providers in-session) and therapeutic exercises that support a range of youth and adult 
EBTs. 
 
3.2  
Perspectives on homework barriers and mHealth solutions (NIMH F32 MH108250) 
Adhere.ly’s initial conceptualization was informed by semi-structured qualitative 
interviews with 21 national trainers in Trauma Focused Cognitive Behavioral Therapy 
(TF-CBT)34 and 15 youth/caregiver TF-CBT patients. These interviews explored 
potential mHealth solutions to barriers to implementing homework during youth mental 
health treatment. Results suggested that many providers struggle to consistently develop, 
assign, and assess homework exercises with their patients, many of whom have difficulty 
remembering to practice skills in a correct and timely way that fosters adequate learning. 
Trainers and patients were generally positive about the potential for mHealth to improve 
the implementation of homework and provided suggestions for mHealth solutions in 
terms of functionality and user experience.21  
 
Ongoing development and evaluation of Adhere.ly (NIMH K23 MH118482) 
The current version of Adhere.ly was developed as a resource to improve homework 
implementation by providers and adherence in children and caregivers during treatment 
for childhood anxiety, depression, and PTSD. As such, the therapeutic exercises are 
mostly child focused.  
 
Survey among community mental health providers 
We recently conducted a survey among 277 community mental health providers who use 
telemedicine for about 25% of their caseload. The majority of respondents were master’s- 
(64%) and doctoral-level (25%) providers working in individual practice (70%) and small 
clinic (18%) settings. Most providers reported treating adults (98%) with anxiety (95%), 
depression (87%), and PTSD (77%), and using cognitive-behavioral (82%) and 
interpersonal (52%) treatment approaches in their general practice. Almost half (41%) of 
providers reported assigning homework “never” to “sometimes,” and the most common 
exercises assigned were mindfulness (78%), coping/emotion regulation (73%), relaxation 
(70%), interpersonal skills (56%), self-monitoring (56%), problem-solving (54%), and 
cognitive flexibility/reappraisal (51%).35 

 
Page 8 of 30 
Revised: 2/13/2024 
 
4.0 
Study Intervention  
4.1 
We designed Adhere.ly as a simple, HIPAA compliant, web-based application to help 
mental health providers implement homework during CBT with patients with elevated 
PTSD, anxiety, and/or depression. The three major components of Adhere.ly help 
providers to (1) Practice interactive, digitized CBT exercises with patients in-session; (2) 
Remind patients to practice CBT exercises for homework, and (3) Review homework 
during the next session. These functions are accessible via tabs on the left side of the site, 
as displayed in Figure 1. 
 
 
Figure 1 
 
The Practice component includes several brief interactive, digitized CBT exercises for 
providers to practice with patients in-session within each of the following areas: 
relaxation, affect and emotion, cognitive coping, exposure, parent-child activities, 
enhancing safety, and parenting videos (Figures 2-4). 
 
Figure 2 
 

 
Page 9 of 30 
Revised: 2/13/2024 
 
Figure 3 
 
 
Figure 4 
 
The Remind component enables providers to set automated text message/email 
reminders with editable default days/times, for patients to practice exercises between-
session. This includes all exercises found in the Practice component as well as automated 
customizable reminders, encouraging messages, self-monitoring, and parenting exercises 
(Figures 5-6). Patients receive automated text message (or email) reminders during the 
following week on the specified days and times that contain links to the digitized 
exercises, which are opened and completed in the patients’ smartphone or computer 
browser (i.e., no downloads or logins are required). 
 
 

 
Page 10 of 30 
Revised: 2/13/2024 
Figure 5 
 
 
Figure 6 
 
The Review component enables providers to view patients’ homework completion and 
relevant data for certain exercises (e.g., self-monitoring ratings, anxiety ratings during 
exposure exercises; Figure 7).  
 
 
Figure 7 
 
Adhere.ly is accessed by computer, tablet, and/or smartphone browser so it does not 
require any downloads by patients or providers. Patients do not create accounts or log in, 
and providers who choose not to create an account are still able to use the site with 
limited functionality. This functionality is limited in that it does not allow providers to 
save patient contact information (i.e., it must be entered every time a reminder is 
scheduled), does not allow providers to review homework results, and limits reminders to 
customizable reminders, encouraging messages, and self-monitoring and relaxation 
exercises (Figure 8). 
 

 
Page 11 of 30 
Revised: 2/13/2024 
 
Figure 8 
 
5.0 
Procedures Involved 
5.1  
Aim 1. Understand and prioritize key stakeholder goals, challenges, 
affordances, and constraints 
Aim 1 of this study will use an exploratory sequential (qual→quant) mixed-
methods design to conduct semi-structured focus groups (qual) with mental health 
patients, providers, and administrators and a survey (quant) with providers. We 
will conduct 4 focus groups per stakeholder group (patient, provider, 
administrator) and each focus group session will include up to 6 participants. Half 
of the focus groups will consist of stakeholders from public healthcare institutions 
and the other half will consist of stakeholders from private healthcare institutions.  
We then will enroll 100 providers to complete the survey. Analyses will inform 
the design of new Adhere.ly features and exercises and an 
implementation/sustainment (I/S) blueprint. 
 
Aim 2. Optimize, preliminarily evaluate, and implement Adhere.ly in 
community practice settings 
Aim 2 of this study will use a randomized controlled OEI Hybrid trial design. The 
trial will include 50 community mental health providers randomized within their 
respective practice locations to administer CBT+Adhere.ly (n=25) vs. CBT alone 
(n=25) to a total of 60 treatment-seeking adult mental health patients with 
clinically elevated anxiety and/or depression–referred and treated by providers. 
Study staff will conduct pre- and post- implementation assessments remotely with 
providers and will facilitate web-based administration of baseline and 3-month 
post-baseline assessments with patients via REDCap. Treatment sessions may be 
audio recorded and uploaded to REDCap by providers and coded by study staff to 
assess provider assignment and assessment of homework. Clients will have the 
option to allow providers to audio-record their therapy sessions through the 
consent. Patient homework adherence will be assessed weekly the day of their 
next session using a REDCap survey, a link to which will be sent via automated 
text-message and/or email reminders, or by telephone after 1 day of no response. 
 
5.2  

 
Page 12 of 30 
Revised: 2/13/2024 
☒ Audio/Video Recording 
☐ Psychophysiological Recording 
☒ Behavioral Interventions 
☐ Record Review - Educational 
☒ Behavioral Observations and Experimentations 
☐ Record Review - Employee 
☐ Deception 
☐ Record Review- Medical  
☒ Focus Groups 
☐ Record Review - Other 
☐ Interviews 
☐ Specimen Collection or Analysis 
☐ Investigational Device – Non-Significant Risk 
(e.g. Mobile Applications) 
☒ Surveys and/or Questionnaires 
☐Psychometric Testing 
☐ Other Social-Behavioral Procedures 
 
Aim 1. Understand and prioritize key stakeholder goals, challenges, affordances, 
and constraints 
Study staff will invite up to 24 adult mental health patients, 24 providers, and 24 
administrators to participate in remote (i.e., video conference or telephone), ≤90-minute, 
screen and audio-recorded, semi-structured focus groups to inform the design of new 
Adhere.ly features and exercises and an I/S blueprint. Study staff will email and/or call 
mental health providers and administrators registered with Doxy.me–who will be asked 
to refer adult patients–with information about the study and an invitation to participate 
(see attached Letter of Support from Dr. Welch at Doxy.me). Study staff will also post 
recruitment flyer on Facebook and submit research study into the Institute of 
Translational Health Sciences (ITHS) to recruit adult patients. Study staff will schedule 
an appointment to obtain verbal informed consent and begin focus groups, which will 
include basic demographic questions, an overview and brief demonstration of Adhere.ly, 
and will utilize the Modified Consolidated Framework for Implementation Research 
(mCFIR) domains relevant to Adhere.ly features and exercises and this implementation 
(i.e., intervention characteristics, outer settings, inner settings, end-user characteristics, 
and process of implementation).36,37 Next, we will draw upon insights obtained from 
these focus groups and the literature on homework and processes-based CBT to develop a 
quantitative web-based survey that we will administer to 100 mental health providers, 
recruited via emails to providers registered with Doxy.me, to prioritize certain goals, 
challenges, features, and exercises.38,39  
 
Aim 2. Optimize, preliminarily evaluate, and implement Adhere.ly in 
community practice settings 
 
Recruitment of providers 
We will leverage research-practice partnerships established in Aim 1 to recruit 50 
master’s-level mental health providers and will recruit additional providers via emails to 
mental health providers registered with doxy.me, if needed. Providers will be contacted 
by study staff via telephone and/or email to inform them about the study and inquire 
about their interest in participating. Providers who express interest in participating in the 
study will be scheduled for a 60-minute, televideo-based consent and training process 
with study staff to discuss the study, referral procedures, audio recording sessions, and 
uploading recordings to REDCap. Signed informed consent from providers will be 
obtained by study staff via REDCap e-Consent. Afterwards providers will complete a 

 
Page 13 of 30 
Revised: 2/13/2024 
demographics questionnaire and the Attitudes Toward Homework Questionnaire 
(ATHQ).12 
 
Recruitment of patients 
We will recruit 60 treatment-seeking adults over 18 with clinically elevated anxiety 
and/or depression. Patients will be referred and by providers following their initial intake 
session with that provider. We will ask providers to refer 3-4 patients to maximize the 
likelihood of treatment completion with at least 3 patients. Providers will receive 
automated weekly emails with reminders to make referrals and links to a REDCap 
referral form. Providers will also be able to provide referral information via telephone or 
email if preferred. If patients express interest in the study, providers will assist them in 
completing the referral or obtain verbal consent to submit their referral information.  
 
Upon receiving a referral, study staff will contact patient within one working day to 
provide study information, assess eligibility, obtain signed informed consent from 
patients via REDCap e-Consent, and assist patient in completing baseline assessments. 
Study staff will assess eligibility by: (1) administering a phone screen to the patient; and 
(2) emailing or texting patient a link through the REDCap system to a survey that will 
include a brief demographic questionnaire, the Generalized Anxiety Disorder-7 (GAD-7) 
or Patient Health Questionnaire-8 (PHQ-8).40–42 Patients who are eligible, as indicated by 
a score ≥ 10 on the GAD-7 or PHQ-8, will then provide informed consent via REDCap e-
Consent, and complete the remaining study baseline questionnaires. 
 
Treatment 
Providers will be randomly assigned to administer either CBT alone (n=25) or 
CBT+Adhere.ly (n=25). All providers will be asked to administer CBT as they usually 
would for each patient, either in-person or via telemedicine, with CBT+Adhere.ly 
providers integrating Adhere.ly into treatment with their patients. To minimize risk of 
CBT alone (i.e., “control”) providers being less motivated to refer cases, providers will 
be informed of their assignment only after their first study-eligible patient has been 
enrolled. Each provider will be asked to treat 3 patients over the course of 3 months. 
Providers will be asked to audio record their treatment sessions using either audio 
recorders provided by the study team, or audiorecording software on their computer, if 
preferred. Clients will have the option to audio record the treatment sessions through their 
consent. Providers will be asked to upload those recordings to REDCap either weekly or 
bi-weekly. Providers will receive automated weekly emails with a reminder to upload 
audio recordings and a link to a REDCap form for uploading recordings. This form will 
ask the provider to enter their name, the name of the patient being treated, the session 
number, and whether the session was conducted in-person or over telemedicine. 
Providers who have not uploaded a recording after 2 weeks will receive a follow-up 
phone call from study staff to provide reminders and assistance where needed. 
 
Assessment Strategy and Measures  
Trial questionnaires are shown in Table 1. Baseline and 3-month post-baseline 
questionnaires will be completed by patients via REDCap surveys. Study staff will email 
or text patients a link to the survey through the REDCap system and will be available to 

 
Page 14 of 30 
Revised: 2/13/2024 
remotely assist patients in completing questionnaires. As stated previously, GAD-7 and 
PHQ-8 will be administered to screen potential participants for eligibility criteria and will 
be administered prior to obtaining informed consent. The informed consent document 
will specify that these data will be included with participants’ study data if they are 
eligible and agree to participate or will not be included if they are ineligible or decide not 
to participate. 
 
Provider homework use, patient homework adherence, and treatment fidelity 
Patient homework adherence will be assessed weekly on the day of their next session using 
a REDCap survey, a link to which will be sent via automated text-message and/or email 
reminders, or via telephone by study staff after 1 day of no response. The first 4 items of 
the Homework Rating Scale II (HRS II) will be used to measure patient homework 
adherence.46 Session audio recordings will be observationally coded by independent coders 
blinded to study aims. Provider adherence and competence in reviewing, designing, and 
assigning homework (i.e., homework use) will be assessed using the Homework Adherence 
and Competence Scale (HAACS).47 Provider general therapeutic and CBT-specific skills 
to appropriately deliver CBT will be assessed by the Assessment of Core CBT Skills 
(ACCS). 48  
 
Optimization 
We will iteratively optimize Adhere.ly and our implementation strategies during the 
feasibility trial based on user data, error reports, support requests, and data from routine 
feedback interviews conducted by Adhere.ly, LLC. We will address unanticipated 
problems and opportunities encountered during implementation, remove unnecessary 
components and I/S strategies, and improve processes and functionality. We will develop 
a clear set of procedures for optimization and keep a log of all changes made for 
reporting purposes, consistent with the Trials of Intervention Principles outlined by Mohr 
and colleagues.49  
 
Implementation 
Pre-implementation assessments will take place during Aim 1. Study staff will conduct 
remote, ≤45-minute individual post-implementation interviews with ≤10 patients and ≤10 
providers who participate in the CBT+Adhere.ly condition. Participants will be recruited 
via telephone by study staff and scheduled for an interview. Informed consent from 
providers and patients to participate in these interviews will have been obtained by study 
staff during the initial consent process for the trial. These interviews will use the mCFIR 
tool to assess perceptions about Adhere.ly’s implementation and performance within each 
Table 1. Assessment Measures for Trial 
Domain 
Informant 
Measure 
Time Point 
B 
3M 
Demographics 
Provider 
Patient 
Provider Demographics Questionnaire 
Patient Demographics Questionnaire 
P 
 
Attitudes Toward Homework Provider 
Attitudes Toward Homework Questionnaire (ATHQ)12 
P 
P 
Anxiety Severity 
Patient 
General Anxiety Disorder-7 (GAD-7)40,41 
P 
P 
Depression Severity 
Patient 
Patient Health Questionnaire-8 (PHQ-8)42 
P 
P 
Quality of Life 
Patient 
Health-Related Quality of Life (CDC HRQOL-14)43 
P 
P 
Therapeutic Alliance 
Patient 
Working Alliance Inventory- Short Revised (WAI-SR)44 
 
P 
Treatment Satisfaction 
Patient 
Client Satisfaction Questionnaire (CSQ-8)45 
 
P 
Note. B=Baseline; 3M=3-Month Follow-Up. 
 
 

 
Page 15 of 30 
Revised: 2/13/2024 
of the mCFIR domains. To complement this evaluation of implementation processes, we 
will evaluate implementation outcomes with respect to Reach, Effectiveness, Adoption, 
Implementation, and Maintenance guided by the RE-AIM framework.50,51 These data will 
be added to the robust mCFIR data to contribute to an overall summative evaluation.  
 
5.3  
The standard of care procedures for adults with clinically elevated anxiety and/or 
depression is traditional CBT from provider. 
 
5.4  
There are no additional foreseeable risks to the above procedures in need of further 
mitigation beyond those ordinarily incurred in working with this population. Standard 
operational procedures of clinics specify responsibilities for handling dangers to self and 
others and safety planning in the event of domestic violence. 
 
5.5  
N/A 
 
5.6 
N/A 
 
5.7 
N/A 
 
6.0 
Data and Specimen Storage for Future Research 
6.1 
N/A 
 
6.2 
N/A 
 
6.3 
N/A 
 
7.0 
Sharing of Results with Subjects 
7.1 
Total scores on study questionnaires will be shared verbally by study staff with patients 
upon request. Total scores will also be shared with providers upon request from providers 
and when permission to do so is granted by patient as indicated on their informed consent 
form. 
 
8.0 
Study Timelines 
8.1 
Aim 1. Adult mental health patients, providers, and administrators will participate in one 
≤90-minute focus group. Providers will be invited to complete a ≤15-minute survey. 
 

 
Page 16 of 30 
Revised: 2/13/2024 
Aim 2. Providers will participate in one ≤60-minute consent and training process and 
twelve ~60-minute therapy sessions per study case. Study referral procedures will take an 
additional 5 minutes per patient. An additional 5 minutes per patient, per session will be 
required to upload audio recordings. Providers who participate in post-implementation 
interviews will spend an additional 45 minutes in the study. In all, the total amount of 
time spent in the study will be approximately 28 hours over the course of 9 months, only 
4 hours of which will be spent engaging in study procedures beyond their everyday 
practice. 
 
Patients will participate in a ≤60-minute eligibility screening, consent, and baseline 
assessment process. Patients will participate in twelve ≤60-minute therapy sessions, 
engage in twelve ~10-minute homework assignments, and spend ≤5 minutes each week 
completing homework assessments. 3-month follow-up assessments will take ≤30 
minutes. Patients who participate in post-implementation interviews will spend an 
additional 30 minutes in the study. In all, the total amount of time spent in the study will 
be approximately 17 hours over the course of 6 months, only about 5 hours of which will 
be spent engaging in study procedures beyond the time they would have otherwise been 
receiving therapy from their provider. 
 
9.0 
Inclusion and Exclusion Criteria 
9.1 
Aim 1. English-speaking, adult (≥18 years old) mental health patients, providers, and 
administrators. 
 
Aim 2. 
Providers: English-speaking, mental health providers who have obtained at least a 
master’s degree in social work, counseling, clinical psychology, or related field; carry 
active adult mental health treatment caseloads; and have a laptop, tablet, or smartphone 
with internet access.  
Patients: English-speaking, treatment seeking adults ≥18 years with clinically elevated 
anxiety and/or depression as indicated by a score ≥10 on the GAD-7 and PHQ-8; and 
have a laptop, tablet, or smartphone with internet access. 
 
9.2 
Patients with self-reported (1) active psychotic symptoms (e.g., hallucinations, delusions) 
or (2) significant cognitive disability, based on the phone screen. 
 
9.3 
Patients will be ineligible to continue if there is a discontinuation or interruption of CBT 
treatment with participating provider. If a patient changes therapists to a provider 
enrolled in the study, they will be able to participate with the new provider but will be re-
consented to confirm that they still want to participate with the new provider and will be 
notified that their baseline questionnaire data will be maintained to avoid the unnecessary 
burden of collecting those data a second time. 
 
 

 
Page 17 of 30 
Revised: 2/13/2024 
9.4 
We will be including employees as participants. 
 
10.0 Vulnerable Populations 
10. 
N/A 
 
11.0 Local Number of Subjects 
11.1 
N=282 
Aim 1 focus groups: n=24 patients; 24 providers; 24 administrators 
Aim 1 survey: n=100 providers 
Aim 2 trial: n=50 providers; 60 adult patients 
 
12.0 Recruitment Methods 
12.1 
 
12.2 
Aim 1. Study staff will email and/or call mental health providers and administrators 
registered with Doxy.me–who will be asked to refer adult patients–with information 
about the study and an invitation to participate (see attached Letter of Support from Dr. 
Welch at Doxy.me). Study staff will also post recruitment flyer on Facebook and submit 
research study into the Institute of Translational Health Sciences (ITHS) to recruit adult 
patients. Study staff will then schedule an appointment to obtain verbal informed consent 
and conduct focus groups. We will recruit 100 mental health providers registered with 
Doxy.me via email to complete the web-based survey. 
 
Aim 2. Providers will be recruited by leveraging research-practice partnerships 
established in Aim 1 and will recruit additional providers via emails to mental health 
providers registered with doxy.me, if needed. Providers will be contacted by study staff 
via telephone and/or email to inform them about the study and inquire about their interest 
in participating. Patients will be referred and by providers following their initial intake 
session with that provider. We will ask providers to refer 3-4 patients to maximize the 
likelihood of treatment completion with at least 3 patients. Providers will receive 
automated weekly emails with reminders to make referrals and links to a REDCap 
referral form. Providers will also be able to provide referral information via telephone or 
email if preferred. Providers will be given informational cards with a QR code to the 
referral form. If patients express interest in the study, providers will assist them in 
completing the referral or obtain verbal consent to submit their referral information. 
Upon receiving a referral, study staff will contact patient within one working day to 
☒ Email 
☒ Online/Social Media Advertisement 
☒ Flyer 
☐ Record Review 
☐ Letter 
☐ SONA 
☐ News Advertisement 
☐ Other  

 
Page 18 of 30 
Revised: 2/13/2024 
provide study information, assess eligibility, obtain signed informed consent from 
patients via REDCap e-Consent, and assist patient in completing baseline assessments. 
 
12.3 
N/A 
 
13.0 
Withdrawal of Subjects 
13.1 
If a participating provider decides to withdraw participation from the study, patients 
being treated by that provider will also be withdrawn from the research without their 
consent and will revert to standard treatment. If a patient changes therapists to a provider 
enrolled in the study, they will be able to participate with the new provider but will be re-
consented to confirm that they still want to participate with the new provider and will be 
notified that their baseline questionnaire data will be maintained to avoid the unnecessary 
burden of collecting those data a second time. 
 
 
13.2 
Subjects who withdraw will continue to receive the same standard of care from their 
mental health provider. 
 
14.0 Risks to Subjects 
14.1 
Physical, psychological, social, cultural, financial, and legal risks, and risks to privacy 
and/or confidentiality associated with this research are minimal because the probability 
and magnitude of harm or discomfort anticipated in the proposed research are not greater, 
in and of themselves, than those ordinarily encountered in daily lives of the general 
population or during the performance of routine physical or psychological examinations 
or tests. 
 
These include: 
1. Possible breach of patient privacy and/or confidentiality 
2. Possible breach of the security of patient and provider online data 
3. Possible patient discomfort due to completing psychological questionnaires 
4. Possible patient and provider discomfort due to having therapy sessions audio 
recorded 
 
14.2 
N/A 
 
14.3 
N/A 
 
15.0 Potential Benefits to Subjects or Others 
15.1 

 
Page 19 of 30 
Revised: 2/13/2024 
Potential direct benefits to participants assigned to the CBT+Adhere.ly condition include 
access to a clinically useful health IT resource, gains in patient knowledge and skill, and 
better patient outcomes. 
  
15.2 
Researchers and the general scientific community will benefit from the knowledge gained 
from the study. This includes knowledge about how to design and implement a user-
centered and stakeholder-informed program that supports patient and provider adherence 
to evidence-based practice as well as effective vs. less effective strategies for 
implementing health IT solutions that promote this adherence in mental health practice 
settings in the community. 
 
16.0 Data Management and Confidentiality 
16.1 
Aim 1 Data Analysis 
Focus group transcripts will be coded by trained study staff in NVivo using a hybrid 
inductive-deductive, consensus-based content analysis,52–54 and the mCFIR tool.36,37,55 
Coders will meet regularly to ensure consistency with code definitions and resolve 
inconsistencies via discussion to achieve consensus. Data aggregation queries will be 
used to create case memos and assign ratings for mCFIR constructs. Survey data will be 
analyzed to describe provider goals, challenges, affordances, and constraints. We will use 
the Expert Recommendations for Implementing Change (ERIC) matching tool to 
generate a list of potential implementation strategies for refinement.56,57 We will 
synthesize qualitative and quantitative findings to prioritize new features, exercises, and 
I/S strategies. 
 
Aim 2 Data Analysis 
Feasibility 
We will assess feasibility of the proposed trial methodology using the following 
benchmarks, which were informed by the sample size needed for hypothesis testing in 
Phase II and expert recommendations.58–61 
 
Benchmarks: 50 providers will be enrolled in months 1-2 [20/month]); 60 patients will be 
enrolled in months 2-6 [8/month]); 70% of providers and patients will use Adhere.ly; 
70% of session recordings will be uploaded; 70% of weekly HRS II assessments will be 
completed; 70% of patients will be retained at 3-month follow-up. 
 
Evaluation 
The small sample size prevents any conclusions about effectiveness; however, ANCOVA 
will be used to preliminarily assess-between group differences in clinical outcomes while 
co-varying for pre-treatment scores. ANOVA will be used to compare homework use and 
adherence. We also will preliminarily examine relations among study variables 
(therapeutic alliance, satisfaction). 
 
Implementation 
Interview transcripts will be coded by trained study staff in NVivo using the mCFIR 

 
Page 20 of 30 
Revised: 2/13/2024 
coding scheme developed during Aim 1. Data aggregation queries will be used to create 
case memos and assign ratings for mCFIR constructs, which we will examine between 
pre- and post-implementation using a modified CFIR matrix template. We will further 
synthesize qualitative with quantitative (i.e., mCFIR, RE-AIM) results to confirm and/or 
explain summative evaluation findings. 
 
Table 2. RE-AIM outcomes 
Dimension 
Results 
Reach 
Number of potential providers and patients available for recruitment; percentage eligible/ineligible, invited, and 
enrolled; sample characteristics; reasons for ineligibility and varying levels of engagement; ambiguities regarding 
eligibility criteria 
Effectiveness 
Attrition and related patient characteristics; adverse events; satisfaction; impact 
Adoption 
Estimated number of settings for future trials; settings and stakeholder characteristics 
Implementation 
Percentage of providers and patients who use Adhere.ly; average number of exercises used by providers and 
patients; percentage of session recordings uploaded; percentage of weekly HRS II completed; degree of use and 
adherence; provider/patient time costs; patient retention; assessment time burden; completed assessments; missing 
items; broken/lost/stolen equipment; technical issues 
Maintenance 
Clinical outcomes at 3-months post-baseline 
 
16.2  
All data collected during the trial will be securely stored in a REDCap database, 
including informed consent/assent documents, questionnaire data, and audio recordings. 
Data obtained from Adhere.ly platform for the purposes of this study will include names, 
phone numbers, and email addresses, and usage data relating to providers practicing, 
assigning, and assessing exercises, and patients completing those exercises. These data 
will be downloaded from Adhere.ly and merged with the data in REDcap. Only IRB-
approved and trained study personnel will have access to the REDCap project and access 
will be limited to information and modules that are required for them to complete their 
assigned study-related tasks. All identifiers will be marked as such in REDCap and will 
not be included in the final exported dataset, which will instead include assigned ID 
numbers. 
 
We will use the following security measures to protect data sources: 
1. all research data exported from REDCap will include ID numbers only 
2. the codes that link the name of the participant and the study ID will be kept 
confidential in REDCap 
3. computers and servers containing data will be password-protected to prohibit 
unauthorized access 
4. online survey data collection will be accessible only by IRB approved study staff 
with secure logins 
5. Adhere.ly includes state-of-the-art technical infrastructure–including encryption 
and other software, security practices, and business operational practices to ensure 
compliance with all major governing legislation, including HIPAA 
6. all study data will be kept on a secure, USF server 
 
16.3 
We will use the following quality assurance measures for subject recruitment, enrollment, 
enrollment targets, and for the validity and integrity of the data: 
1. study staff will complete and maintain up-to-date CITI and GCP training 
2. study staff will be trained and supervised weekly by the PI 

 
Page 21 of 30 
Revised: 2/13/2024 
3. Standard Operating Procedures (SOPs) will be used to train all study staff 
4. a Manual of Procedures (MOP) and study checklists will be used to ensure 
fidelity to the study protocol 
5. informed consent will be obtained and documented by study staff to provide an 
audit trail 
6. any contact with study participants or potential study participants will be 
documented to provide an audit trail 
7. participant screening, recruitment, enrollment and enrollment targets, and data 
collection will be tracked to provide weekly updates to the PI 
8. assessment data will be entered directly by participants into REDCap 
9. automated validity checks will be in place for any data collection 
10. data checks for ranges, cross-validity, and completion will be completed proximal 
to data collection 
11. collection of any study data will be documented by study staff to provide an audit 
trail 
12. coders will be trained and appropriate measures will be used to assess interrater 
reliability 
13. coders will be blinded to study aims and hypotheses 
14. the USF Conflict of Interest (COI) Office will maintain and monitor study 
progress according to ongoing conflict of interest management plans to ensure 
compliance with all requirements 
 
16.4  
Identifiable information in this study will include provider, patient names, phone 
numbers, email addresses, and physical addresses. Human subjects research records, 
including the original signed and dated consent documents, will be stored for at least 5 
years after study completion. Signed and dated HIPAA authorizations and consent 
documents that include HIPAA authorizations will be stored for at least 6 years after 
study completion. After this time data will be deleted from REDCap. Patient assessment 
data will be shared with providers upon request, as stated in the consent. 
 
16.5 
 
Patient PHI that will be disclosed to providers will include total scores on questionnaires 
if patients indicate their permission to do so on the consent form. Patient PHI that will be 
obtained from providers will include names, session dates, audio recordings of sessions, 
telephone numbers, and email addresses. We will document in REDCap events where 
PHI is disclosed or obtained. 
 
☒Obtaining Signed Authorization 
☐Waiver of HIPAA Authorization for 
Recruitment/Screening Purposes Only 
☐Obtaining Online or Verbal Authorization 
(Alteration of HIPAA Authorization) 
☐Waiver of HIPAA Authorization for 
Entire Study 
☐Data Use Agreement 
☐Business Associate Agreement 

 
Page 22 of 30 
Revised: 2/13/2024 
17.0 Provisions to Monitor the Data to Ensure the Safety of Subjects 
17.1  
N/A 
 
17.2 
N/A 
 
18.0 Provisions to Protect the Privacy Interests of Subjects 
18.1 
All research activities will be conducted over web-based surveys or telephone by IRB 
approved study staff located in a secure and private location. Subjects’ assessment results 
will only be shared with their mental health provider if they indicate their permission 
when providing consent. 
 
18.2 
Study participants will provide informed consent. We will not access any previously 
existing records. 
 
19.0 Compensation for Research-Related Injury 
19.1 
N/A 
  
20.0 Subject Costs and Compensation 
20.1  
Providers and patients who participate in this study will have internet and smartphone 
access, so we anticipate that they will have data and SMS plans that will allow them to 
participate without any additional costs. However, there is the small risk that some 
participants may exceed their monthly data or SMS limits during this study, resulting in 
additional costs for which they will be responsible. This risk will be discussed during the 
informed consent process. 
 
20.2 
 
Aim 1 participants will receive $50 and $15 eGift cards for completing focus groups and 
surveys, respectively. 
 
Aim 2 providers will receive a $50 eGift card after each referred client enrolls/consents 
into the study, in compensation of their time. Patients will receive a $30 eGift card after 
completing baseline assessments and an additional $30 eGift card after completing 3-
month post-baseline follow-up assessments, with a possibility of receiving a total of $60 
☐No Compensation 
☐Tokens (pens, food items, etc.) 
☒Financial Compensation (cash, gift cards) 
☐Other  
☐Course Credit (i.e. extra credit, SONA 
points) 
 

 
Page 23 of 30 
Revised: 2/13/2024 
in eGift cards. Participants who participate in post-implementation interviews will 
receive $50 eGift card following completion of the interview. 
 
21.0 Consent Process 
21.1  
 
21.2 
 
Aim 2. The project coordinator will obtain signed informed consent from providers via 
REDCap e-Consent during a ≤60-minute, televideo-based consent and training process. 
We will ensure that providers understand all aspects of the study and consent form, and 
adequate time will be provided for questions relating to referral procedures, audio 
recording sessions, and uploading recordings to the study server. The consent form will 
also inform providers that they may be asked to participate in post-implementation 
interviews if they participate in the CBT+Adhere.ly condition. To minimize the 
possibility of coercion or undue influence on providers we will emphasize that declining 
to participate in the study will not influence their employment. If a patient changes 
therapists to a provider enrolled in the study, they will be able to participate with the new 
provider but will be re-consented to confirm that they still want to participate with the 
new provider and will be notified that their baseline questionnaire data will be maintained 
to avoid the unnecessary burden of collecting those data a second time. 
 
The project coordinator will obtain signed informed consent from patients via REDCap e-
Consent during a 30-minute televideo-based consent process. We will ensure that 
providers understand all aspects of the study and consent form, and adequate time will be 
provided for questions relating to the assessment process over the course of the study. 
The consent form will include a field for patients to indicate their consent to have their 
total scores from questionnaires shared with their providers if requested. The consent 
form will also inform providers that they may be asked to participate in post-trial 
qualitative interviews if they participate in the CBT+Adhere.ly condition. To minimize 
the possibility of coercion or undue influence we will emphasize that declining to 
participate in the study will not influence the quality of care that they would have 
otherwise received from their provider. 
 
21.3  
Aim 1. Study staff will obtain verbal informed consent from patients, providers, and 
administrators prior to conducting focus groups. Providers will indicate their consent to 
☒Obtaining Signed Consent (Subject or 
Legally Authorized Representative) 
☒Obtaining Consent Online (Waiver of 
Written Documentation of Consent ) 
☐Obtaining Signed Parental Permission 
☒Obtaining Verbal Consent (Waiver of 
Written Documentation of Consent) 
☐Obtaining Signed Assent for Children 
or Adults Unable to Consent 
☐Waiving Consent and/or Parental Permission 
(Waiver of Consent Process) 
☐Obtaining Verbal Assent for Children or 
Adults Unable to Consent 
☐Waiving Assent/Assent is Not Appropriate 

 
Page 24 of 30 
Revised: 2/13/2024 
participate in the survey by checking a box indicating their consent on the online consent 
form prior to completing the survey. To minimize the possibility of coercion or undue 
influence on providers we will emphasize that declining to participate in the study will 
not influence their employment. 
 
21.4 
N/A 
 
21.5 
N/A 
 
21.6 
N/A 
 
22.0 Setting 
22.1 
The research will be conducted via telephone, online, and in the USF Department of 
Psychiatry and Behavioral Neurosciences. Providers will practice and audio record 
sessions at their own clinic site, as normal. 
 
23.0 References 
 
1.  Committee on Crossing the Quality Chasm: Adaptation to Mental Health and 
Addictive Disorders, Board on Health Care Services, Institute of Medicine, National 
Academy of Sciences. Improving the quality of health care for mental and 
substance-use conditions. Washington, D. C. , DC: National Academies Press; 2006. 
2.  Ronan KR, Kazantzis N. The use of between-session (homework) activities in 
psychotherapy: Conclusions from the Journal of Psychotherapy. J Psychother Integr 
[Internet]. American Psychological Association (APA); 2006;16(2):254–259. 
Available from: http://dx.doi.org/10.1037/1053-0479.16.2.254 
3.  Kazantzis N, Ronan KR. Can between-session (homework) activities be considered 
a common factor in psychotherapy? J Psychother Integr [Internet]. American 
Psychological Association (APA); 2006;16(2):115–127. Available from: 
http://dx.doi.org/10.1037/1053-0479.16.2.115 
4.  Kazantzis N, Deane FP, Ronan KR, L’Abate L. Using Homework Assignments in 
Cognitive Behavior Therapy. Routledge; 2005. 
5.  Hudson JL, Kendall PC. Showing you can do it: homework in therapy for children 
and adolescents with anxiety disorders. J Clin Psychol [Internet]. Wiley; 2002 
May;58(5):525–534. Available from: http://dx.doi.org/10.1002/jclp.10030 PMID: 
11967878 
6.  Scheel MJ, Hanson WE, Razzhavaikina TI. The process of recommending 
homework in psychotherapy: A review of therapist delivery methods, client 

 
Page 25 of 30 
Revised: 2/13/2024 
acceptability, and factors that affect compliance. Psychotherapy (Chic) [Internet]. 
American Psychological Association (APA); 2004;41(1):38–55. Available from: 
http://dx.doi.org/10.1037/0033-3204.41.1.38 
7.  Kazantzis N, Lampropoulos GK. Reflecting on homework in psychotherapy: what 
can we conclude from research and experience? J Clin Psychol [Internet]. Wiley; 
2002 May;58(5):577–585. Available from: http://dx.doi.org/10.1002/jclp.10034 
PMID: 11967882 
8.  Kazantzis N. Homework assignments in cognitive and behavioral therapy: A meta-
analysis. Clin Psychol [Internet]. American Psychological Association (APA); 2000 
May 1;7(2):189–202. Available from: 
https://doi.org/10.1093/clipsy.7.2.189?locatt=mode:legacy 
9.  Kazantzis N, Whittington C, Dattilio F. Meta-analysis of homework effects in 
cognitive and behavioral therapy: A replication and extension. Clin Psychol (New 
York) [Internet]. Wiley; 2010 Jun 8;17(2):144–156. Available from: 
http://dx.doi.org/10.1111/j.1468-2850.2010.01204.x 
10.  Mausbach BT, Moore R, Roesch S, Cardenas V, Patterson TL. The Relationship 
Between Homework Compliance and Therapy Outcomes: An Updated Meta-
Analysis. Cognit Ther Res [Internet]. Springer Nature; 2010 Oct;34(5):429–438. 
Available from: http://dx.doi.org/10.1007/s10608-010-9297-z PMCID: 
PMC2939342 
11.  Kazantzis N, Whittington C, Zelencich L, Kyrios M, Norton PJ, Hofmann SG. 
Quantity and quality of homework compliance: A meta-analysis of relations with 
outcome in cognitive behavior therapy. Behav Ther [Internet]. 2016 Sep;47(5):755–
772. Available from: http://dx.doi.org/10.1016/j.beth.2016.05.002 PMID: 27816086 
12.  Kazantzis N, Lampropoulos GK, Deane FP. A national survey of practicing 
psychologists’ use and attitudes toward homework in psychotherapy. J Consult Clin 
Psychol [Internet]. American Psychological Association (APA); 2005 
Aug;73(4):742–748. Available from: http://dx.doi.org/10.1037/0022-006X.73.4.742 
PMID: 16173863 
13.  Dattilio FM, Kazantzis N, Shinkfield G, Carr AG. A survey of homework use, 
experience of barriers to homework, and attitudes about the barriers to homework 
among couples and family therapists. J Marital Fam Ther [Internet]. Wiley; 2011 
Apr;37(2):121–136. Available from: http://dx.doi.org/10.1111/j.1752-
0606.2011.00223.x PMID: 21457279 
14.  Kazantzis N, Deane FP. Psychologists’ use of homework assignments in clinical 
practice. Prof Psychol Res Pr [Internet]. 1999 Dec;30(6):581–585. Available from: 
https://psycnet.apa.org/fulltext/1999-01568-007.pdf 
15.  Gaynor ST, Lawrence PS, Nelson-Gray RO. Measuring homework compliance in 
cognitive-behavioral therapy for adolescent depression: review, preliminary 

 
Page 26 of 30 
Revised: 2/13/2024 
findings, and implications for theory and practice. Behav Modif [Internet]. SAGE 
Publications; 2006 Sep;30(5):647–672. Available from: 
http://dx.doi.org/10.1177/0145445504272979 PMID: 16894234 
16.  Helbig S, Fehm L. Problems with homework in cbt: Rare exception or rather 
frequent? Behav Cogn Psychother [Internet]. Cambridge University Press (CUP); 
2004 Jul;32(3):291–301. Available from: 
http://dx.doi.org/10.1017/s1352465804001365 
17.  Kazantzis N, Shinkfield G. Conceptualizing patient barriers to nonadherence with 
homework assignments. Cogn Behav Pract [Internet]. Elsevier BV; 2007 
Aug;14(3):317–324. Available from: http://dx.doi.org/10.1016/j.cbpra.2006.08.003 
18.  Simons AD, Marti CN, Rohde P, Lewis CC, Curry J, March J. Does homework 
“matter” in cognitive behavioral therapy for adolescent depression? J Cogn 
Psychother [Internet]. Springer Publishing Company; 2012;26(4):390–404. 
Available from: http://connect.springerpub.com/lookup/doi/10.1891/0889-
8391.26.4.390 
19.  Coon. Homework compliance in CBT effects outcomes [Internet]. PsycEXTRA 
Dataset. American Psychological Association (APA); 2003. Available from: 
http://dx.doi.org/10.1037/e490262004-001 
20.  Deane FP, Ronan KR, L’Abate L, Kazantzis N, editors. Using homework 
assignments in cognitive behavior therapy. London, England: Routledge; 2005. 
21.  Bunnell BE, Nemeth LS, Lenert LA, Kazantzis N, Deblinger E, Higgins KA, 
Ruggiero KJ. Barriers associated with the implementation of homework in youth 
mental health treatment and potential mobile health solutions. Cognit Ther Res 
[Internet]. Springer Science and Business Media LLC; 2020 Mar 16; Available 
from: http://link.springer.com/10.1007/s10608-020-10090-8 
22.  Demographics of mobile device ownership and adoption in the United States 
[Internet]. 2021 [cited 2021 Apr 14]. Available from: 
https://www.pewresearch.org/internet/fact-sheet/mobile/ 
23.  Anderson M. Teens, social media & technology 2018 [Internet]. 2018 [cited 2021 
Apr 14]. Available from: https://www.pewresearch.org/internet/2018/05/31/teens-
social-media-technology-2018/ 
24.  Ruggiero KJ, Saunders BE, Davidson TM, Lewsky Cook D, Hanson R. Leveraging 
Technology to Address the Quality Chasm in Children’s Evidence-Based 
Psychotherapy. Psychiatr Serv [Internet]. 2017 Jul 1;68(7):650–652. Available from: 
http://dx.doi.org/10.1176/appi.ps.201600548 PMID: 28617212 
25.  Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental Health Smartphone Apps: 
Review and Evidence-Based Recommendations for Future Developments. JMIR 

 
Page 27 of 30 
Revised: 2/13/2024 
Ment Health [Internet]. JMIR Publications Inc.; 2016 Mar 1;3(1):e7. Available 
from: http://dx.doi.org/10.2196/mental.4984 PMCID: PMC4795320 
26.  Tang W, Kreindler D. Supporting Homework Compliance in Cognitive Behavioural 
Therapy: Essential Features of Mobile Apps. JMIR Ment Health [Internet]. JMIR 
Publications Inc.; 2017 Jun 8;4(2):e20. Available from: 
http://dx.doi.org/10.2196/mental.5283 PMCID: PMC5481663 
27.  Reger GM, Hoffman J, Riggs D, Rothbaum BO, Ruzek J, Holloway KM, Kuhn E. 
The “PE coach” smartphone application: an innovative approach to improving 
implementation, fidelity, and homework adherence during prolonged exposure. 
Psychol Serv [Internet]. 2013 Aug;10(3):342–349. Available from: 
http://dx.doi.org/10.1037/a0032774 PMID: 23937084 
28.  Miloff A, Marklund A, Carlbring P. The challenger app for social anxiety disorder: 
New advances in mobile psychological treatment. Internet Interventions [Internet]. 
2015 Nov 1;2(4):382–391. Available from: 
http://www.sciencedirect.com/science/article/pii/S2214782915300014 
29.  Watts S, Mackenzie A, Thomas C, Griskaitis A, Mewton L, Williams A, Andrews 
G. CBT for depression: a pilot RCT comparing mobile phone vs. computer. BMC 
Psychiatry [Internet]. 2013 Feb 7;13:49. Available from: 
http://dx.doi.org/10.1186/1471-244X-13-49 PMCID: PMC3571935 
30.  Ly KH, Trüschel A, Jarl L, Magnusson S, Windahl T, Johansson R, Carlbring P, 
Andersson G. Behavioural activation versus mindfulness-based guided self-help 
treatment administered through a smartphone application: a randomised controlled 
trial. BMJ Open [Internet]. 2014 Jan 9;4(1):e003440. Available from: 
http://dx.doi.org/10.1136/bmjopen-2013-003440 PMCID: PMC3902198 
31.  Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, Deleeuw 
W. Mobile therapy: case study evaluations of a cell phone application for emotional 
self-awareness. J Med Internet Res [Internet]. 2010 Apr 30;12(2):e10. Available 
from: http://dx.doi.org/10.2196/jmir.1371 PMCID: PMC2885784 
32.  Dagöö J, Asplund RP, Bsenko HA, Hjerling S, Holmberg A, Westh S, Öberg L, 
Ljótsson B, Carlbring P, Furmark T, Andersson G. Cognitive behavior therapy 
versus interpersonal psychotherapy for social anxiety disorder delivered via 
smartphone and computer: a randomized controlled trial. J Anxiety Disord 
[Internet]. 2014 May;28(4):410–417. Available from: 
http://dx.doi.org/10.1016/j.janxdis.2014.02.003 PMID: 24731441 
33.  Pramana G, Parmanto B, Lomas J, Lindhiem O, Kendall PC, Silk J. Using Mobile 
Health Gamification to Facilitate Cognitive Behavioral Therapy Skills Practice in 
Child Anxiety Treatment: Open Clinical Trial. JMIR Serious Games [Internet]. 2018 
May 10;6(2):e9. Available from: http://dx.doi.org/10.2196/games.8902 PMCID: 
PMC5968217 

 
Page 28 of 30 
Revised: 2/13/2024 
34.  Cohen JA, Mannarino AP, Deblinger E. Treating Trauma and Traumatic Grief in 
Children and Adolescents, Second Edition [Internet]. Guilford Publications; 2017. 
Available from: https://play.google.com/store/books/details?id=sZvUDAAAQBAJ 
35.  Brian E. Bunnell, Nikolaos Kazantzis, Samantha R. Paige, Janelle Barrera, Rajvi N. 
Thakkar, Dylan Turner, Brandon M. Welch. Provision of Care By “Real World” 
Telemental Health Providers. Front Psychol. 
36.  Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. 
Fostering implementation of health services research findings into practice: a 
consolidated framework for advancing implementation science. Implement Sci 
[Internet]. 2009 Aug 7;4:50. Available from: http://dx.doi.org/10.1186/1748-5908-4-
50 PMCID: PMC2736161 
37.  Tilahun B, Smillie K, Bardosh KL, Murray M, Fitzgerald M, Cook V, Poureslami I, 
Forrest J, Lester R. Identifying Barriers and Facilitators of 13 mHealth Projects in 
North America and Africa: Protocol for a 5-Year Implementation Science Study. 
JMIR Res Protoc [Internet]. 2018 Jul 3;7(7):e162. Available from: 
http://dx.doi.org/10.2196/resprot.9633 PMCID: PMC6053607 
38.  Hayes SC, Hofmann SG. Process-Based CBT: The Science and Core Clinical 
Competencies of Cognitive Behavioral Therapy. 
39.  Kazantzis N, Dattilio FM. Definitions of homework, types of homework, and ratings 
of the importance of homework among psychologists with cognitive behavior 
therapy and psychoanalytic theoretical orientations. J Clin Psychol [Internet]. 2010 
Jul;66(7):758–773. Available from: http://dx.doi.org/10.1002/jclp.20699 PMID: 
20527055 
40.  Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing 
generalized anxiety disorder: the GAD-7. Arch Intern Med [Internet]. 2006 May 
22;166(10):1092–1097. Available from: 
http://dx.doi.org/10.1001/archinte.166.10.1092 PMID: 16717171 
41.  Johnson SU, Ulvenes PG, Øktedalen T, Hoffart A. Psychometric Properties of the 
General Anxiety Disorder 7-Item (GAD-7) Scale in a Heterogeneous Psychiatric 
Sample [Internet]. Frontiers in Psychology. 2019. Available from: 
http://dx.doi.org/10.3389/fpsyg.2019.01713 
42.  Kroenke K, Strine TW, Spitzer RL, Williams JBW, Berry JT, Mokdad AH. The 
PHQ-8 as a measure of current depression in the general population. J Affect Disord 
[Internet]. 2009 Apr;114(1–3):163–173. Available from: 
http://dx.doi.org/10.1016/j.jad.2008.06.026 PMID: 18752852 
43.  for Disease Control C, Prevention, Others. CDC HRQOL-14 “Healthy Days 
Measure.” Centers for Disease Control and Prevention. 2016; 

 
Page 29 of 30 
Revised: 2/13/2024 
44.  Hatcher RL, Gillaspy JA. Development and validation of a revised short version of 
the working alliance inventory. Psychother Res [Internet]. Routledge; 2006 Jan 
1;16(1):12–25. Available from: https://doi.org/10.1080/10503300500352500 
45.  Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of 
client/patient satisfaction: development of a general scale. Eval Program Plann 
[Internet]. 1979;2(3):197–207. Available from: http://dx.doi.org/10.1016/0149-
7189(79)90094-6 PMID: 10245370 
46.  Kazantzis N, Deane FP, Ronan KR. The homework rating scale II—Client version. 
Using homework assignments in cognitive behavior therapy New York, NY: 
Routledge. 2005; 
47.  Kazantzis N, Wedge P, Dobson KS. Homework adherence and competence scale 
(HAACS). From the team research project Cognitive Behavior Therapy Homework 
Project Albany, New Zealand: Massey University. 2005; 
48.  Muse K, McManus F, Rakovshik S, Thwaites R. Development and psychometric 
evaluation of the Assessment of Core CBT Skills (ACCS): An observation-based 
tool for assessing cognitive behavioral therapy competence. Psychol Assess. 
American Psychological Association (APA); 2017 May;29(5):542–555. 
49.  Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, Kwasny MJ, 
Stiles-Shields C, Cheung K. Trials of Intervention Principles: Evaluation Methods 
for Evolving Behavioral Intervention Technologies. J Med Internet Res [Internet]. 
2015 Jul 8;17(7):e166. Available from: http://dx.doi.org/10.2196/jmir.4391 PMCID: 
PMC4526981 
50.  RE-AIM [Internet]. [cited 2021 Apr 19]. Available from: http://www.RE-AIM.org 
51.  Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, Ory MG, 
Estabrooks PA. RE-AIM Planning and Evaluation Framework: Adapting to New 
Science and Practice With a 20-Year Review [Internet]. Frontiers in Public Health. 
2019. Available from: http://dx.doi.org/10.3389/fpubh.2019.00064 
52.  Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services 
research: developing taxonomy, themes, and theory. Health Serv Res [Internet]. 
2007 Aug;42(4):1758–1772. Available from: http://dx.doi.org/10.1111/j.1475-
6773.2006.00684.x PMCID: PMC1955280 
53.  Shenton AK. Strategies for ensuring trustworthiness in qualitative research projects. 
Education for information [Internet]. IOS press; 2004;22(2):63–75. Available from: 
https://content.iospress.com/articles/education-for-information/efi00778 
54.  Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual 
Health Res [Internet]. SAGE Publications; 2005 Nov;15(9):1277–1288. Available 
from: http://dx.doi.org/10.1177/1049732305276687 PMID: 16204405 

 
Page 30 of 30 
Revised: 2/13/2024 
55.  Damschroder LJ. CFIR codebook template. Ann Arbor, MI. 2015; 
56.  Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, 
Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results 
from the Expert Recommendations for Implementing Change (ERIC) project. 
Implement Sci [Internet]. 2015 Feb 12;10:21. Available from: 
http://dx.doi.org/10.1186/s13012-015-0209-1 PMCID: PMC4328074 
57.  Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, 
Proctor EK, Kirchner JE. Use of concept mapping to characterize relationships 
among implementation strategies and assess their feasibility and importance: results 
from the Expert Recommendations for Implementing Change (ERIC) study 
[Internet]. Implementation Science. 2015. Available from: 
http://dx.doi.org/10.1186/s13012-015-0295-0 
58.  Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, 
Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. 
BMC Med Res Methodol [Internet]. 2010 Jan 6;10:1. Available from: 
http://dx.doi.org/10.1186/1471-2288-10-1 PMCID: PMC2824145 
59.  Abbott JH. The Distinction Between Randomized Clinical Trials (RCTs) and 
Preliminary Feasibility and Pilot Studies: What They Are and Are Not. J Orthop 
Sports Phys Ther [Internet]. Journal of Orthopaedic & Sports Physical Therapy; 
2014 Aug 1;44(8):555–558. Available from: https://doi.org/10.2519/jospt.2014.0110 
60.  Tickle-Degnen L. Nuts and bolts of conducting feasibility studies. Am J Occup Ther 
[Internet]. 2013 Mar;67(2):171–176. Available from: 
http://dx.doi.org/10.5014/ajot.2013.006270 PMCID: PMC3722658 
61.  Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in 
clinical research. J Psychiatr Res [Internet]. 2011 May;45(5):626–629. Available 
from: http://dx.doi.org/10.1016/j.jpsychires.2010.10.008 PMCID: PMC3081994