Testing of a Patient-centered E-health Implementation Model in Addiction Treatment
Phase: NA
Conditions: Substance Use Disorders
Sponsor: University of Wisconsin, Madison

1  
  
Testing of a patient-centered e-health implementation model in addiction treatment 
 
NCT03954184 
 
Todd Molfenter, Ph.D.  
University of Wisconsin - Madison  
 
February 15, 2021 
 
 
 

2  
  
  
  
Testing of a patient-centered e-health 
implementation model in addiction treatment  
  
Principal Investigators:  
  
  
Todd Molfenter, Ph.D.  
University of Wisconsin - Madison  
Industrial Engineering  
1513 University Avenue, ME Bldg. Suite 4103  
Madison, WI 53706  
608-262-1685; todd.molfenter@wisc.edu  
  
David H. Gustafson, Ph.D. 
University of Wisconsin - Madison  
Industrial Engineering  
1513 University Avenue, ME Bldg. Suite 4109  
Madison, WI 53706  
608-263-4882; Dhgustaf@wisc.edu  
  
Co-Investigators:     
  
Jee-Seon Kim, Ph.D.  
University of Wisconsin - Madison  
Educational Psychology  
1025 W. Johnson Street, 1057 Educational Sciences  
Madison, WI 53706  
608-262-0741; jeeseonkim@wisc.edu  
  
  
  
Stephan Arndt, Ph.D.  
University of Iowa  
100 MTP4 Room 126  

3  
  
Iowa City, IA 52242  
319-335-4488; Stephan-arndt@uiowa.edu  
  
DeAnn Decker, B.A.  
Bureau Chief  
Iowa Department of Public Health  
Lucas State Office Building  
321 E. 12th Street  
Des Moines, IA 50319  
515-281-7689; DeAnn.Decker@idph.iowa.gov  
  
Study Coordinator:   
  
Julie Horst, MBA  
Industrial Engineering  
1513 University Avenue, ME Bldg. Suite 4121 
Madison, WI 53706  
608-265-6322 ; jhorst@wisc.edu  
  
Sponsor: National Institute on Drug Abuse  
  
Protocol ID: 2018-0997 
Protocol Version:  February 15, 2021 
 
   
  
 
 

4  
  
Contents  
Study Summary ......................................................................................................................................... 5 
1. BACKGROUND AND SIGNIFICANCE....................................................................................................... 5 
2. STUDY OBJECTIVES ............................................................................................................................... 7 
3. SELECTION OF SUBJECTS ...................................................................................................................... 8 
4. REGISTRATION PROCEDURES ............................................................................................................... 9 
5. TREATMENT PLAN  ............................................................................................................................. 10 
6. POTENTIAL RISKS ................................................................................................................................ 21 
7. MEASUREMENT OF EFFECT ................................................................................................................ 24 
8. STUDY PARAMETERS .......................................................................................................................... 25 
9. STATISTICAL CONSIDERATIONS .......................................................................................................... 25 
10. RECORDS TO BE KEPT ....................................................................................................................... 27 
Appendix A: CHESS DATA SECURITY PLAN .............................................................................................. 29 
10. ACRONYM LIST.................................................................................................................................. 31 
11. REFERENCES CITED ........................................................................................................................... 32 
 
  
 
 

5  
  
Study Summary  
Patient-centered e-health has failed to achieve its promise despite considerable consumer 
interest in technology and research supporting its potential. E-health adoption rates in 
healthcare are poor, with specialty substance use disorder (SUD) treatment having the lowest 
technology adoption rate of any sector.1 Implementation science can address this emerging gap 
in the e-health field by augmenting existing models, that explain organizational and individual 
ehealth behaviors retrospectively, with prospective models that can guide implementation. The 
organizational planning discipline, with its decades of research, could provide a cross-
disciplinary “jump start” to developing an e-health implementation model for health 
organizations.2, 3 Henry Mintzberg, a respected pioneer in this field, describes 2 beneficial 
approaches to planning: the deliberate approach, which is grounded in pre-implementation 
planning, and the emergent approach that is grounded in adapting to the environment as the 
plan is implemented.4 The proposed e-health implementation model, called the Network for the 
Improvement of Addiction Treatment–Technology Implementation (NIATx-TI) Framework, 
incorporates both approaches.  
NIATx-TI was piloted in the Iowa Rural Health Information Technology Initiative (IRHIT) with 14 
of Iowa’s 105 SUD treatment sites and resulted in a 2-fold increase in patients receiving 
distance treatment. The framework’s deliberate component includes using an organizational 
technology assessment5 and patient simulation.6 These tools identify and address assets and 
barriers to incorporate into the technology’s implementation protocol. The framework’s 
emergent component includes using a project team to uncover and prioritize implementation 
barriers as they arise, develop changes to address identified barriers, and monitor selected 
adoption measures, while receiving monthly coaching.  
This project, based in Iowa, will compare a control condition (using a typical product training 
approach to software implementation that includes user tutorials and instruction on  
administrative and clinical protocols, followed by access to on-line support) to the typical 
product training combined with NIATx-TI. While e-health spans many modalities and health 
disciplines, this project will focus on the implementing Addiction Comprehensive Health 
Enhancement Support System (A-CHESS), an evidence-based SUD treatment recovery app 
developed by our Center for a disease that affects 21.5 million and kills 136,000 Americans 
annually: substance use disorder.7 A mobile app was selected, as opposed to another e-health 
technology, because of the near ubiquitous daily use of mobile technology and because mobile 
e-health adoption requires supportive participation of both health centers and patients.  
  
  
In response to the COVID-19 pandemic, we have added a study component focused on describing how 
patients are responding to receiving remote treatment (e.g., telehealth). We will also seek to understand 
how using A-CHESS mitigates COVID-19 associated anxiety and loneliness among those with substance 
use disorders. 
 
1. BACKGROUND AND SIGNIFICANCE  
This project addresses an issue of substantial public health significance: enhancing the uptake 
of the growing list of patient-centered evidence-based addiction treatment e-health 
technologies.  
Substance Use Disorders (SUDs) Impact on Society and Health. Every day on average, 96 
Americans die due to opioid overdoses and another 6 die due to alcohol poisoning, making 
SUDs the third-leading cause of preventable death in the United States.8 Many injuries and 

6  
  
diseases (e.g., cancer, diabetes, cardiovascular problems, cirrhosis, and HIV/AIDS) are caused 
or exacerbated by alcohol abuse.9 Despite the health and societal consequences of SUDs, just 
9.2% of those who need treatment receive it7 and, for those treated, 75% fail to achieve 
longterm sobriety.10 The gap between the need for and the effectiveness of SUD treatment 
services calls for implementing fundamentally different treatment modalities to improve 
SUD.   
Patient-centered Mobile E-health: A Potential Answer? Patient-centered mobile e-health 
technologies offer innovative ways to improve treatment and recovery supports for SUDs.11 
Seventy-seven percent of the adult population in United States currently have a smartphone,12 
as do 74% of adults accessing Iowa SUD treatment clinics. By 2020, over 95% of the US 
population is expected to have smartphone access.12 Meta-analyses of mobile behavioral 
ehealth interventions have found superior treatment outcomes when compared to treatment as 
usual.13,14 The combination of consumer interest, research evidence, and the number of 
research dollars being dedicated to evaluating these technologies will continue to place positive 
pressure on the use of e-health. Despite the promise of e-health, its potential benefits are 
far from being realized.15-17 A prominent example of this is in SUD care. Despite the 
availability of evidence-based e-health tools such as the Drinker’s Check-up,18,19 Therapeutic 
Education System (TES),20 CBT4CBT,21 and A-CHESS, none of these technologies are used by 
>1% of SUD treatment providers.22 This proposal tests the premise that an evidence-based 
technology implementation framework can reduce the gap between patient-centered 
ehealth evidence and practice.  
Technology Adoption Models: Strengths and Needs. Extensive research supports building 
models to speed diffusion and sustainability of technology adoption.23 Technology adoption 
research began with models, grounded in behavioral change,24 explaining why individual users 
abandon traditional practices in favor of new technologies.25 Over time, these individual-focused 
models have had to expand because of the necessary role organizational practices and climate 
play in individual decisions to adopt and continue to use a technology.26,27 Explanatory models 
emerged based on 2 prominent frameworks: the diffusion of innovation and the technology, 
organization, and environment framework.24 These frameworks include the key organizational 
factors of management support, clinician satisfaction, clinical workflow, and financial resources 
for technology purchase, implementation, and use.28,29 Financial resources such as start-up 
grants and reimbursement supporting technology use are important factors in technology 
adoption. However, reviews by Brooks28 and Jeyaraj29 demonstrate that reimbursement cannot 
be the only issue considered. Two health technology implementation projects conducted by our 
Center in 7 different states during 2012–2015 confirmed that patient, clinician, and 
organizational as well as financial barriers can limit technology adoption. In seeking approaches 
to address these multi-faceted barriers, we were surprised to find that most models described 
the barriers,30,31 but not how to address them. In seeking technology implementation 
models, we realized the organizational planning field could provide the frameworks 
needed to address the competencies and processes that remained unexplored in 
previous technology implementation research.   
Planning Model Theory as a Conceptual Foundation for Network for the Improvement of 
Addiction Treatment (NIATx)- Technology Implementation (TI): Two important evolutions 
have created a planning field that has remained relatively stable in theory and approach for the 
past 2 decades.32 The first was Porter’s (1979)33 5 competitive forces framework, which created 
a foundation for deliberate planning. This “look before you leap” planning approach relies on a 
centralized planning function and development of “the plan.” Concerns with this methodology 
led Mintzberg and Waters (1985)4 to assert that Porter’s deliberate (sometimes called 

7  
  
prescriptive) planning for organizational strategy and change was necessary, but not sufficient. 
Simply too much occurred once a plan was hatched. Their emergent (sometimes called 
descriptive) planning approach used data that emerged from the implementation process to 
improve the plan. This moved planning from a purely centralized (top-down) process to include 
a more emergent (staff engaged) approach.34-36 Using these concepts, our Center, The Center 
for Health Enhancement Systems Studies (CHESS), searched through our field experiences37-40 
and the literature, for pre-implementation (deliberate) as well a post-implementation (emergent) 
strategies capable of quickly adjusting to environmental feedback to construct the NIATx-TI 
ehealth implementation framework.   
  
Mobile Technology as a Focus was selected because of its pervasive use, the evidence base 
for the CHESS platform that supports A-CHESS, and our familiarity with A-CHESS.     A-
CHESS is based on the CHESS platform, which has been studied in 16 randomized control 
trials (RCTs) over 16 years, using several technology media.41-43 These RCTs, each involved 
hundreds of people, found that the CHESS platform improved key dimensions of quality of 
life,41,  
44 health behavior adherence (e.g., smoking cessation and risky drinking),45-47 and outcomes 
(e.g., lung cancer death rates and sobriety). A-CHESS is consistent with respected health 
behavior change models such as self-determination theory (SDT)48 and Marlatt’s Cognitive 
Behavior model49 of relapse prevention. In a randomized trial (N=349), patients using A-
CHESS, compared to those in the control group, reported 51% fewer risky drinking days in the 
prior 30 days at the 4-, 8-, and 12-month follow-ups (mean of 1.39 days in A-CHESS vs. 2.75 
days per month in TAU, p = .003). Rates of alcohol abstinence were significantly higher among 
A-CHESS patients at 8 months (78% vs. 67%) and 12 months (79% vs. 66%).50 The A-CHESS 
app includes educational materials, discussion groups, links to SUD resources and weekly self-
monitoring for patients. The CHESS platform has also been an effective method for clinicians to 
interact with their patients.51 A-CHESS provides a report for clinicians that delivers to the 
clinician team information about worrisome changes in symptoms collected from the patient 
during self-monitoring of risk-related items (i.e., sleeping problems, depression, urge, risky 
situation and relationship troubles).52 A-CHESS contacts the clinical team by email, phone, or 
fax (according to clinician preference) whenever a threshold symptom is reported. The alert was 
intended to quickly bring clinician attention to emerging symptoms.  
Technology like A-CHESS is a potential game changer: it can provide recovery supports 24 
hours a day, 7 days a week and can be customized to individual needs. This research will test 
a technology adoption framework to increase use of the A-CHESS smartphone app. A 
supplemental study will help understand how A-CHESS may help patients manage increased 
levels of anxiety and loneliness during the COVID-19 pandemic. 
  
2. STUDY OBJECTIVES  
The specific aim of this study is to compare the effectiveness of NIATx-TI (Arm 2) and Product 
Training (Arm1) in implementing an evidence-based innovation (A-CHESS). Impact will be 
measured using the RE-AIM framework:53  
• 
Reach or Participation (Primary Outcome): a) percent and representativeness, based on 
age, gender, and ethnicity by organizational site, and b) A-CHESS use per each 
counselor, by days of use;  
• 
Effectiveness: retention rates of eligible patients overall 

8  
  
• 
Adoption: percent and representativeness, based on age, gender, race, ethnicity, and 
education level of counselors referring and not referring patients to A-CHESS;   
• 
Implementation: a) organizational readiness for, and b) fidelity of A-CHESS 
implementation; and c) qualitative and d) cost effectiveness analyses to develop a better 
understanding of the implementation processes for delivering NIATx-TI compared to 
product training alone; and   
• 
Maintenance: A-CHESS use (e.g., A-CHESS downloads and days of A-CHESS use) 
during the post-test phase.  
 
The specific aims of the COVID-19 supplemental study are 1) to understand whether using A-
CHESS may help patients manage anxiety and loneliness associated with social isolation, 
physical distancing, and other changes to daily life during the COVID-19 pandemic, and 2) to 
understand how patients have adapted to changes in treatment modality during the COVID-19 
pandemic. Impact will be measured by offering all patients who have ever signed up for an A-
CHESS account the chance to complete an online survey (administered through REDCap) as 
well as the opportunity for a qualitative interview with a member of our research team using a 
standardized interview guide. 
  
Study Coordination  
The UW-Madison Center for Health Enhancement Systems Studies (CHESS) is the 
coordinating site for this study. The UW study coordinator will oversee all recruitment activities 
which includes:   
• 
developing organizational site-specific recruitment and data collection processes that 
meet study objectives;  
• 
training staff on protocol procedures prior to start of recruitment and continuous 
monitoring to assure compliance with the protocol and human subjects regulation;  
• 
communicating with clinic site staff as needed via conference calls to monitor progress, 
inform of protocol changes/distribute new version of protocol, and address unanticipated 
issues or challenges;  
• 
and manage all study data.  
  
3. SELECTION OF SUBJECTS  
This study will engage SUD organizations, counseling and peer recovery support specialist staff 
at those organizations, and patients of those organizations.    
Organizations: Sixteen SUD organizations in Iowa, with a total of forty sites, have agreed to 
participate in the project.    
Counselors: The site coordinator will invite all outpatient counselors serving adult patients 
employed by the 16 SUD organizations to be part of the study as part of their role in the 
participating organizations.    Any counselor may formally decline to participate during the 
training or any time thereafter.   
Peer Recovery Support Specialists: Peer Recovery Support Specialists associated with the 
organization will be invited to be part of the study.  Any Peer Recovery Support Specialists may 
formally decline to participate during training or any time thereafter.  

9  
  
Patients: We will aim to enroll 1,800 outpatients from the 16 enrolled organizations.  Each 
organization is expected to have about 516 A-CHESS-eligible patients on average. We estimate 
that we will recruit a conservative 30% (or approximately 43 per organizational site) for a total 
n=1,800 patients in the study. We will recruit patients for the 18-month intervention period plus 6 
months into the maintenance period.  Patients are eligible for the study if they:   
• 
Are 18+ years old  
• 
Understand English  
• 
Have a SUD diagnosis  
• 
Have access to a smartphone  
 
All patients will be offered the opportunity to complete a survey through REDCap. To date, 
n=520 patients have created an A-CHESS account. We will also conduct in-depth interviews 
with 40 patients who have created an A-CHESS account.  
4. REGISTRATION PROCEDURES  
Recruitment and Consent Procedures  
Counselor and Peer Recovery Support Specialists Recruitment: 
Initially, Counselors and Peer Recovery Support Staff will be provided with the Counselor and 
Peer Recovery Support Specialists consent (092518 Version 2.0) when they go to log onto the 
App (in a similar fashion to how the patient consent is occurring in the study).  This will occur 
prior to the A-CHESS product training. In addition, during the A-CHESS product training, that 
will occur in both arms of the study, the ACHESS trainer will explain the Counselor Training 
consent form (10.1 Version 2.0) to the counselors as well as Peer Recovery Support Staff and 
give them the opportunity to sign the consent form should they want to participate in the study.  
The trainer will remind counselors and Peer Recovery Support Staff that are not required to 
participate in the study and can leave the study at any time. For those counselors and Peer 
Recovery Support Staff involved with the study that are unable to attend the product training, or 
are hired after the training, those counselors and Peer Recovery Support Staff will be sent an e-
mail with the consent form and will be given the opportunity to check the consent form (after 
reading it) and return it to the research team should they want to participate in the study. 
 
Patient Recruitment: 
Patients meeting the criteria outlined in section 3.0 will be introduced to A-CHESS during their 
first 3 group or individual treatment sessions at the participating clinic. Counselors in both arms 
of the study that choose to introduce patients to A-CHESS will follow a standard protocol that 
makes clear that clients should be offered the opportunity to download A-CHESS, but clients 
are not required to download the app. For group participants, A-CHESS will be offered during 
every group and anyone who is interested can pick up the A-CHESS URL information sheet, 
with login instructions, following the group session. A-CHESS will be available as an Internet 
download to patients in the study. The consent form will be part of the initial sign-on to the A-
CHESS application. The consent form will clearly state that use of ACHESS is voluntary and not 
a condition of receiving treatment services. Once patients download A-CHESS and agree to the 
consent form via checkbox, they are enrolled in the study.   
 

10  
  
Organizational and Staff Recruitment: 
The organizational survey will be completed by a member of the management team.  The staff 
surveys will be completed by staff that are invited to complete the survey by the site 
coordinator. (The site coordinator’s role in the clinic may be an administrative support position in 
the quality or staff training departments, a member of clinical supervision, or management.) The 
site coordinator will inform the staff person that survey completion is optional, and the consent 
form that precedes the survey in REDCap will also state that survey participation is optional.  
The site coordinator does not have access to the survey data nor will they receive any data 
feedback or summaries from the surveys.  The consent form will describe that completing the 
staff or organizational survey infers their consent. For those who do not complete the survey, 
the survey will not be recorded and no record of the person not wanting to participate will be 
made.  
 
Patient COVID-19 Supplemental Redcap Survey and Interview Recruitment:  
All patients who have signed up for an A-CHESS account will be contacted via email and invited 
to complete a supplemental survey about their experiences during the COVID-19 pandemic. It 
will be necessary to collect patient’s names and email addresses in order to link their survey 
responses to passively collected app usage data. Patients will then complete an approximately 
30-minute online survey through REDCap. Participants will be remunerated $25 for their time. 
At the end of the survey, participants will be asked whether they are interested in being 
contacted by a member of the research team to participate in a follow-up, in-depth interview. A 
research assistant will contact interested participants and set up a time to conduct an 
approximately 30-minute interview. Research assistants will go over an informed consent 
document, and participants will need to consent to interviews being recorded. Participants will 
be remunerated an additional $25 for their time. 
  
5. TREATMENT PLAN  
Treatment as Usual (TAU)/Control: Product Training/On-line Support – SUD organizational 
sites in the TAU or Control arm will participate in 1. day product implementation (or training) 
period sessions during Ms 11 - 15. On-line technical support will occur for the 18 months 
following product training.   
Product training consists of two components: A-CHESS Demo and A-CHESS Product Training.  
During the A-CHESS Demo with the organization’s IT, clinical leadership, and the A-CHESS 
site coordinator, the A-CHESS Product Trainer will:   
a) describe the features of A-CHESS (Tutorial)  
b) provide an A-CHESS organizational case example on how A-CHESS is implemented  
(including clinical protocols)  
c) provide examples of how to recruit and prepare clinicians and patients for A-CHESS  
 
During the 1-hour A-CHESS Product Training, the organization’s A-CHESS Site Coordinator 
and the A-CHESS Product Trainer will train the counselors and Peer Recovery Support 
Specialists at each site. Prior to the training, the site coordinator will present the Counselor and 
Peer Recovery Consent Form to each counselor as well as Peer Recovery Support Specialists 
and give them the opportunity to consent or decline to participate.  

11  
  
For those counselors that agree to participate, the training will include:  
a) demonstrating A-CHESS features and how A-CHESS allows patients to self-manage 
their recovery in conjunction with their counselor.  
b) role-playing (or patient simulation) live A-CHESS enrollment with counselors, where they 
can use the A-CHESS patient description sheet to guide conversations with patients   
c) demonstrating how to use the A-CHESS counselor dashboard.   
Following the product training session, on-line technical support will be available. Patients and 
counselors can call or e-mail questions using the A-CHESS portal.  
  
Counselor and Peer Recovery Support Specialist’s role (same for Control and Intervention) 
– The counselors will learn about A-CHESS during the A-CHESS Product training.  Counselors 
and Peer Recovery Support Specialists may use A-CHESS to view comments in discussion 
groups.  Counselors and Peer Recovery Support Specialists will also have access to the 
counselor dashboard which contains data from his/her patients’ weekly reports.  The counselor 
may access the dashboard at any time and can use the information from the dashboard in 
counseling sessions.   Following the demo and training, counselors will identify potentially 
eligible patients and ask them if they are interested in the study. If yes, the counselor will 
provide a study overview at a regularly scheduled appointment.   
 
Patients will learn:  
• 
the benefits and potential risks of participation  
• 
how their privacy will be protected.  
  
The counselor will provide a brief introduction to A-CHESS and provide the client with the 
information necessary to enroll in A-CHESS.  If the patient is interested in participating, 
informed consent will be documented via an IRB-approved consent form on the A-CHESS app, 
containing the above information regarding benefits, risks and privacy protections. Record of 
their consent will be stored on a DoIT server at UW Madison.  Patients declining participation 
will receive treatment as usual.  
  
. Staff at the clinic site are responsible for:  
• 
Identifying potential subjects  
• 
Providing patients with the information on how to sign-up for ACHESS  
  
Some (5 or fewer), but not all, counselors and Peer Recovery Support Specialists at each site 
will be asked to voluntarily participate in the study’s annual staff survey. Also, a few counselors 
and Peer Recovery Support Specialists in the sites who are invited to participate in the 
quantitative analysis may also be invited to participate in qualitative interviews.  
 
Patient use of A-CHESS app (same for Control and Intervention) – At the first use of the A-
CHESS app, the patient will use an automated A-CHESS set up.  The set-up tool will train 
patients to use A-CHESS and customize it (e.g. by sources of support [e.g. family], and 
contacts who detract from recovery [e.g. friends who use illegal drugs]). A-CHESS will be 
updated monthly with activities for the healthy event calendar, changes to therapeutic goals and 
recovery plan (e.g., self-help groups, medication, home-work-education responsibilities), and 
high-risk locations to avoid. Patients can use video tutorials to practice A-CHESS services and 
experience how A-CHESS can help them 
 

12  
  
With subsequent use of the A-CHESS app patients will be asked how they are doing and about 
their recent alcohol and drug use. These questions will be sent weekly on the A-CHESS system 
and will take about 2-4 minutes to complete. All questions are voluntary. Patients are free to 
refuse to answer any questions they are uncomfortable with. However, by answering the weekly 
questions about their recent alcohol and drug use A-CHESS can send the patient information to 
help them in their recovery. In addition, data based on their responses will be accessible to 
his/her clinical team via the Counselor Dashboard so the clinic team can provide needed 
support. This sensitive health information is being collected and shared via the Counselor 
Dashboard to allow clinicians to be better informed and provide treatment more responsively 
based on their patients’ needs as they deal with addiction and other serious health issues. All 
patients will be reminded that they are under no obligation to participate in this study, can 
withdraw from the study at any time, and in no way will their health care be affected by their 
participation in this study.   
  
Intervention: NIATx-TI with Product Training/On-line Support – For SUD organizational sites in 
the Intervention arm, the NIATx TI framework includes the product training as well as a pre-
implementation (planning) phase and a post-implementation (problem-solving) phase.  A 
NIATxTI coach (who is also an A-CHESS product trainer) will provide the training for this arm. 
The coach will also assist the organizations with applying the NIATx-TI framework. Figure 1 
presents the steps in the NIATx-TI Framework.   
Pre-implementation (Planning) Phase  
Step1. Define Aims: The NIATx-TI coach will provide an executive briefing on project setup and 
developing strategic and implementation A-CHESS aims for the organization. (This occurs 
before Product Training.)    
Step 2.  Organizational Pre-work & Assessments of assets and barriers, organizational 
readiness (using OCM).   
Step 3. Protocol Development: Develop clinical and administrative protocols that integrate 
assessment findings. Steps 3 – 5 are part of two partial day NIATx-TI Implementation/Product 
Training meetings.    
Post-implementation (Problem-solving) Phase  

13  
  
Step 4. Change Leader & Change Team 
Training/Planning on how to modify 
organizational processes to address barriers 
to A-CHESS implementation. This is 
combined with the Product Training 
visit/Implementation meeting.    
Step 5: Pilot test A-CHESS implementation 
enhancement changes. During 
implementation meetings, via a series of Plan 
Do Study Act (PDSA) cycles as staff and 
patients are being recruited and trained, 
unstructured feedback on the progression of 
A-CHESS implementation will be collected. 
Step 5 begins at the Implementation meeting 
and is also part of the iterative PDSA change 
cycle process with steps 6 and 7.    
  
Step 6. Monitor A-CHESS Implementation by 
reviewing progress on Aims and gaining real-
time user feedback. The A-CHESS app will 
provide monthly de-identified enrollment 
reports to the site coordinator or his/her 
designee on the number of patients enrolled in A-CHESS for the month.    
 
May return to Step 5 again - Change Cycles or pilot tests to overcome barriers to implementing 
A-CHESS.  Monthly calls from the NIATx coach support the organization through the change 
cycles that occur as part of steps 5 – 7.    
Step 7. Sustain Changes: Implement plans to institutionalize gains and incorporate use of 
ACHESS in organizational processes.    
  
Data collection  
Surveys: See table 1 for a complete list of Measures, Sources, and Tool/Frequency.  Contacts 
at organizational sites will complete two main surveys: organizational and staff. Clinic staff 
participants will be sent links to complete the survey via REDCap (Research Electronic Data 
Capture managed by ICTR).  
 
Organizational Surveys: Will be completed by a member of the management team two times 
during the project (organizational baseline and end of the intervention.)  To increase survey 
completion rates for the organizational surveys, we conduct follow-up phone calls and e-mails 
with the change leaders at Week 1 of post-survey distribution to assure the survey receipt, then 
at Weeks 4 and 12-post survey distribution if surveys are not complete. 
 
Staff Surveys: The full organizational and staff surveys (A-CHESS Implementation Fidelity, 
Organizational Change Manager, and Financial Resource Availability) will be administered two 
times during the project organizational baseline and end of the intervention).  A ‘generic’ version 
of the Organizational Change Manager (OCM) will be administered prior to training.  This 

14  
  
version of the survey will include questions that focus on prior change projects or innovations in 
the organization.  During the second administration the OCM (as part of the full Staff Survey, 
the OCM questions will focus on the planned implementation of the e-health app.  Each survey 
(A-CHESS Implementation Fidelity, Organizational Change Manager and Financial Resource 
Availability) is expected to take approximately 15 minutes or less to complete.   Up to seven 
staff including a clinical supervisor, an administrative staff person, and up to 5 randomly 
selected counselors and Peer Recovery Support Specialists will complete the staff surveys.  
Staff completing surveys will receive consent documents when they are sent the survey link.  
Each organizational site will receive a $1000 stipend once all organizational and staff surveys, 
have been completed.  
The trial will use 4 cohorts that will have staggered starts in order to have sufficient training 
resources for the trial (Figure 2). Each cohort will have 2 matched pairs with a control and 
treatment site in each pair, resulting in 4 organizations per cohort and n=16 organizations total.   
  
  
Figure 2  
  
 

 
Table 1 - Measures, Sources, and Tool/Frequency  
Measurement   
Measure Source  
Data Source/Frequency (for Cohort 1)  
Descriptive Statistics  
 
 
Organizational traits: Admissions, Rural v. Urban             
Patient traits: Age, Gender, Ethnicity, and Poverty Level  
N-SSATS55  
TEDS56   
IDPH – Ms 11, 22,34, 46IDPH – Ms 11, 
22,34, 46 
Counselor traits: Age, Gender, Ethnicity, Race, Certification or education 
level 
  
Organizational Survey – Ms  12, 30  
 
 
 
 
 
 
Reach  
 
 
% who download A-CHESS app and frequency of A-CHESS use in days  
Gustafson et al.50  
A-CHESS Server - Monthly (Ms 11 - 47)  
IDPH – Monthly (Ms11 - 47)  
Adoption  
 
 
% Counselor Using A-CHESS  
Knight et al.59  
Organizational Survey & A-CHESS Logs  
Effectiveness  
 
 
Retention (Length of Stay)  
Simpson et al.61  
IDPH- Ms 11 - 50 
Maintenance  
 
 
Finance Resource Availability  
Klein62   
Organizational Survey – Ms 12, 30 
13  
  

 
Measurement   
Measure Source  
Data Source/Frequency (for Cohort 1)  
A-CHESS Implementation Fidelity  
Organizational Change Manager (OCM)  
 
Developed for this trial   
Gustafson et al.63  
 
Staff Survey – Ms 12, 30 
Staff Survey – Ms  11, 12, 30 
 
Patient COVID-19 Survey 
Patient socio-demographic and descriptive characteristics: age, 
race/ethnicity, sex, gender, sexual orientation, education, marital status, 
income, housing, family size, employment status, health insurance 
 
Criminal justice history 
 
 
 
Alcohol Use Disorder Severity 
 
 
 
Substance Use Disorder Severity 
 
Loneliness 
 
Anxiety 
 
Substance use disorder recovery strengths 
 
COVID-19 risk and protective behaviors and impact on daily living 
PhenX repository 
 
Justice Community 
Opioid Involvement 
Network 
Saunders et al.; Babor 
et al. 
Skinner 
Hays & DiMatteo 
Cella et al. 
Vilsant et al. 
Adapted from several 
COVID-19 surveys 
Patient COVID-19 Survey – once during 
March 2021 
14  
  

17  
  
Qualitative Interviews: Qualitative data will be gathered from organizational staff at four points. 
Qualitative interviews with clinicians and peer recovery support staff will occur during the 
intervention phase.  The goal of the interviews is to gain a better understanding of patterns with 
app adoption and use.  Phone interviews of up to 30 minutes will be conducted with 
approximately 15 people at various sites. The interviews will be audiotaped (with participant 
permission), but the staff person’s identity will be protected and in no way be linked to results.   
 Additionally, the study, interviews with staff from a randomly selected site in all organizations 
will occur at the end of intervention phase and again at the same site at the end of sustainability 
phase. The intent of the surveys is to gain a retrospective understanding of what occurred 
during implementation. Phone interviews of up to 30 minutes will be conducted with up to four 
people in at each site. The interviews will be audiotaped (with staff participant permission), but 
the staff person’s identity will be protected and in no way be linked to results.   
 
Use data: A-CHESS use data will be collected monthly to assess the percentage of eligible 
patients who download the A-CHESS app per month per site and the frequency of use by app 
user.  For the percentage who download A-CHESS, data will be collected during the 18-month 
intervention period and 6 months of post-intervention period for that organization. Frequency of 
use data for an individual user will be collected during the 12-month period following initial use 
of A-CHESS. The app automatically collects use data by placing “cookies’ on patients’ 
smartphones (with patients’ consent) when they enroll in the study.  Data are stored on a central 
server at CHESS.  Electronic data, such as A-CHESS use data, will be stored on a secure DoIT 
server which is backed up according to a standard protocol.  Access to this data will be limited 
by granting individuals access via their UW user log-in.   
The Iowa Department of Public Health (IDPH) will also provide data available through existing 
IDPH administrative databases.  A preliminary analysis of organizational data will compare 
baseline characteristics between the NIATx-TI and Product Training alone organizations, as 
well as organizations enrolled in the project and eligible organizations not participating 
(Admissions, Rural v. Urban, and Gender/Age/Ethnicity, and Poverty Level), using the National 
Survey of Substance Abuse Treatment Services (N-SSATS)55 and Treatment Episode Data Set 
(TEDS)56 data definitions.  
During the study, IDPH will provide patient demographic data (gender, age, and ethnicity, using 
TEDS definitions) for all patients by organizational site.  As part of the organizational survey, we 
will include questions about counselors’ gender, ethnicity, and race.  
Retention data will be collected by IDPH and provided monthly using existing data.  
We will use UW Box to store data received from IDPH.  
 
Supplemental COVID-19 Study – Patient Surveys and In-Depth Interviews 
All patients who have signed up for an A-CHESS account will be invited to complete an online, 
standardized survey through REDCap. Protected Health Information will be collected only as 
necessary; patients who consent to participate will provide their name (on the informed 
consent) and their email address, phone number, and zipcode (on the survey). All participants 
will be assigned a unique study ID number that will be stored with their survey data (names 
will not be collected on the survey) and the informed consent document will provide the only 
link between the patient’s name and study ID number. Informed consents and survey data will 

18  
  
be stored securely and separately to minimize any risk of compromising patient privacy. 
Patients will be asked to report their socio-demographic information, level of loneliness and 
anxiety, criminal justice system involvement history, substance use disorder and treatment 
history, and COVID-19 risk and protective behaviors. On the informed consent, patients will be 
informed that they can skip or stop the survey at any time. A total of $25 remuneration will be 
provided. 
At the end of the survey, patients will be asked if they would like to be contacted about the 
possibility of completing an in-depth interview about their treatment and A-CHESS experience 
during COVID-19. If they are interested, a research study team member will reach out to them 
by email and/or phone, explain the study and informed consent document, and conduct an in-
depth interview. Again, a unique study ID number will be used instead of the patient’s name. 
During transcription of interview recording, any remaining identifying information stated by the 
patient will be removed prior to analysis. Recordings and transcriptions will be stored securely 
and separately from informed consent documents.  
Unanticipated events  
Should any unanticipated problems arise, they will be reported to the IRB reviewing the site and 
the UW-IRB per the appropriate IRB’s guidance on unanticipated problems and reportable 
events.  
  
Privacy and Confidentiality  
Since substance use disorder (SUD) treatment sites will play a role in the testing of the NIATx 
Technology Implementation (TI) and Product Training implementation strategies, we view both 
consumers (patients) and staff members as human subjects. For the project’s evaluation plan, 
we will address human subjects’ protection from both the staff and consumer levels. CHESS 
(The Center) has addressed human subjects’ protection during the development and testing of 
A-CHESS, the technology, the NIATx-TI Framework and product training approaches we will 
attempt to implement. The Center has successfully handled sensitive personal health 
information in numerous studies of CHESS modules, including cancer, asthma, and the A-
CHESS app.   
The data to be collected in the evaluation will come from existing consumer data collected in 
state administrative databases, user data automatically collected by the A-CHESS App server 
(based on the activities of the unique users), organizational and staff surveys, and staff 
interviews. Table 2 (below) briefly describes the individuals referred to in the protection plan 
who will have access to the project data and their level of access.   
 
Table 2 Research participant overview   
Title  
Affiliation  
Access  
Clinic A-CHESS Site 
Coordinator  
Participating Clinics  
Consumer A-CHESS App server  
data only.  They will not receive or 
view any administrative, survey, or 
interview data  

19  
  
Clinic staff member(s) - 
clinicians only  
Participating Clinics  
Consumer A-CHESS App server 
data for his/her patients only. They 
will not receive or view any 
administrative, survey, or interview 
data  
State database administrator  
Iowa Dept. of Public 
Health  
Consumer administrative data only. 
They will not receive any A-CHESS 
App server, survey, or interview 
data  
Scientific Director (PI)  
UW-Madison  
Limited Data Set consumer 
administrative data, A-CHESS App 
server, survey, and interview data   
UW project team  
(researchers/students)  
UW-Madison   
Limited Data Set consumer 
administrative data, A-CHESS App 
server, survey, and interview data  
  
Study Design  
Staff level: The A-CHESS Site Coordinator within each treatment site will handle recruitment 
for employees to complete interviews and surveys.  Information on the implementation process 
will be gathered from clinic staff members during surveys and interviews. Staff participation will 
be voluntary. The PIs (Dr. Molfenter and Dr. Gustafson) are formally trained on human subjects’ 
protection.  
The Organizational Change Manager (OCM), and ACHESS Implementation Fidelity will be 
assessed by staff survey at months 11 – 14 and 30 – 33 for the four cohorts. Each survey cycle, 
at months listed above  , ill include 280 people per cycle. The 280 staff will include up to 7 
participants from the 40 sites and will likely include a supervisor, a member of administrative 
staff, and up to 5 counselors and Peer Recovery Support Specialists. Since many of the same 
staff will likely complete surveys in successive cycles, 350 unique staff are expected for survey 
completion across all cycles. The total is not 560 unique people (e.g., 280 x 2 rounds of 
surveys) because many of the same people are expected to complete the survey in successive 
rounds. The 350 unique people projected assumes 25% or 70 new people per round during 
rounds that occur post-intervention. Hence, there will be 280 unique people in the early 
intervention – and 70 new or unique people in the subsequent round (post-intervention), for a 
total of 350.  
  
During the intervention as well as at the end of the intervention phase and at the end of the 
sustainability phase we will conduct qualitative interviews.  We will conduct an interview with 
staff at an organizational site from each of the 16 organizations in the study. During the 
intervention we will interview approximately 15 clinicians or peer recovery support staff 
regarding success with getting patients on the app.  We will assess the implementation process 
in both the product training alone (Arm 1) and NIATx-TI sites (Arm 2).  
 
Staff participation will be voluntary. Jacobson, Horst, and a staff person from the Iowa 
Consortium for Substance Abuse Research and Evaluation will conduct telephone interviews 

20  
  
with up to 4 staff members (the organization’s Executive Director, the site’s clinical director, the 
A-CHESS site coordinator at each clinic, and a randomly selected counselor) at 16 sites. This 
will result in 64 interviews per period for a total of 80 interviews (assuming 25% turnover for 
between the rounds of interviews). During the interviews, participants will reflect on their 
participation, issues that arose in using NIATx-TI and A-CHESS, factors that influenced 
acceptance, and perceptions of short- and long-term effects, as well as unintended 
consequences.   
  
The grand total of staff surveys for the main study, at n=350, and staff interviews, at n=80 
participants, is 430.  
  
We will create and distribute an information sheet for staff giving them the option to decline to 
participate in the trial or A-CHESS distribution.  
 
 
Organizational Data: We will assess Financial Resource Availability Inventory, A-CHESS 
Implementation Fidelity, and List of Counselors (for counselor A-CHESS participation rates) at 
two points during the study - at organizational baseline (M 11 - 14) and end of the intervention 
period (M30 - 33) (M=months are in a range due to the 4 study periods) The Iowa Department 
of Public Health (IDPH) will provide, by site, aggregate data on Admissions, and Rural v. Urban. 
This data will be collected from IDPH databases at months  11, 22, 34, and 42 
  
Consumer level: Eligible consumers will be identified in real time during their first three 
outpatient appointments with a participating SUD treatment site. Site counselors will provide 
information about the study.  If the patient is interested in the study, the counselor will provide a 
brief introduction to A-CHESS and provide information to the client with the URL to download A-
CHESS.  If the patient chooses to participate, informed consent will be documented via an IRB-
approved consent form on the A-CHESS app.   
 
To be eligible for A-CHESS, SUD consumers must: 1) be >18 years old; 2) understand and 
agree to an English language consent form via the A-CHESS app; 3) have access to a mobile 
smartphone, and 4) have an SUD diagnosis. 
  
We anticipate enrolling consumers to use A-CHESS (approximately 43 per organizational per 
site, for a n=1,800 for the study). The state (IDPH) will provide de-identified data including 
gender, age, ethnicity, poverty level, and units of service for each site.  
 
During approximately March 2021, all patients who have signed up for an A-CHESS account 
(to-date n=520) will be invited to participate in a one-time, online, self-report survey through 
REDCap. 
  
Sources of Materials   
  
All data will be coded before reaching evaluation staff.  These include data collected by A-
CHESS App server on A-CHESS use as well as organization and staff interview and qualitative 
survey data. Coding of consumer and counselor/staff data will take place, where necessary, by 
replacing identifying information (such as consumer and staff names) with code numbers prior 
to data analysis. A unique study ID number will be assigned to all patient surveys and in-depth 
interviews. Names collected through the informed consent process will be stored separately. 

21  
  
 
Site-level aggregate data will be received from the Iowa Department of Public Health (IDPH).  
The individual de-identified consumer data from the state database will be transmitted to the 
University of Wisconsin database administrator (Wright). The evaluation team will monitor 
overall use and effectiveness of A-CHESS through consumer data directly from the A-CHESS 
App server.   
 
Consumer level:  
A-CHESS usage based on A-CHESS App server tracking of consumer use of their 
smartphones (for initial log-in, duration and frequency of use).   
Surveys and in-depth interviews about COVID-19 associated anxiety and loneliness. 
 
Staff level:  
The OCM Survey will collect data on the organizational philosophy. The A-CHESS 
Implementation Fidelity will document the fidelity to the NIATx TI implementation process.    
 
Counselor level:  
The organizational survey will be used to document all active counselors per site. The following 
information will be gathered about each counselor: title or certification, age, gender, ethnicity, 
and race.   The Iowa Consortium for Substance Abuse Research and Evaluation will check for 
any changes in counselor employment during quarterly check-ins. This will be used to create 
and maintain a counselor list to be compared to the counselors listed during A-CHESS log-ins 
to develop the % of Counselors Using A-CHESS Measure. This same data will not be collected 
for the Peer Recovery Support Specialist. 
 
Organization level:  
IDPH database will provide program characteristics and organizational descriptive 
characteristics and will collect data on the nature and scope of the individual SUD treatment 
sites. Site-level aggregate data on adults with SUD diagnosis by organization by site, including 
gender, age, and ethnicity.  Retention data will also be obtained from IDPH. 
 
The organizational survey will also collect the names and demographics of all counselors and 
Finance Resource Availability assessment. The Finance Resource Availability assessment 
assesses the resources the organization dedicates to adoption of the targeted technology.   
 
 
6. POTENTIAL RISKS 
Staff level:  
Staff members could feel pressure to participate in the study. It will be made clear, through 
written materials and oral instruction, that staff participation in the project’s evaluation is 
completely voluntary. The cost and implementation analyses and their purpose in understanding 
the feasibility of implementing A-CHESS will be explained. Staff responses to interviews will be 

22  
  
coded and responses will be accessible only by evaluation team members (survey and 
interview data will not be accessible to other clinic staff members, including the A-CHESS Site 
Coordinator). If staff members are asked to complete the interview or survey, but refuse, the A-
CHESS Site Coordinator will attempt to identify replacement staff respondents. In addition, 
members of management and clinical supervision will be notified of the importance of staff not 
feeling coerced to participate the study, A-CHESS recruitment, or survey completion.   
 
The potential risk for staff is that the information they provide could be offensive to members of 
management or their peers. Hence, protecting staff confidentiality is an important element of our 
protection plan. Researchers (Jacobson, Horst) trained in protecting consumer confidentiality 
will conduct interviews. Data collected in the interviews and surveys will have a code number 
assigned (as previously defined) attached prior to storing in the project dataset. In this way, we 
can ensure that the interviewer (in the case of interviews) and the University of Wisconsin 
database administrator (in the case of surveys) will be the only person who can identify the 
interviewee’s responses. Also, managers or other members of the organization will not see 
transcripts of interviews.  
 
Consumer level:  
There is a risk that patient subjects may feel uncomfortable sharing personal information or 
answering weekly survey questions through A-CHESS or questions about COVID-19 on the 
supplemental survey and in-depth interviews, especially about sensitive topics. Participants will 
be asked personal questions related to past behaviors and experiences, such as illicit drug use, 
that could produce psychological stress.  
 
The on-line consent process will inform consumers of the need to provide the A-CHESS app 
with information on an ongoing basis to determine whether treatment goals are being met. 
Consumers must agree to allow monitoring of their A-CHESS usage. We will ask consumers to 
consent to share information with their SUD provider. Consent to share information will also be 
provided through the consent form and will comply with 42CFR regulations, specifying what 
information will be shared with the SUD provider and how long the information will be shared. 
The consumer will be given the right not to share specific information with specific people and 
retain the right to revoke the permission granted at any point. Informed consent and assurance 
of confidentiality, with a description of privacy protection measures, should assist with lowering 
participant anxiety.   
 
An additional and similar consent process will be used for the COVID-19 supplemental patient 
survey and in-depth interviews. Individual survey and in-depth interview results will not be 
shared with anyone outside the research team, including with counselors or others at the 
treatment organization. 
 
Consumers can turn on must consent GPS monitoring of their location via A-CHESS, which 
could pose a privacy risk. GPS monitoring is included in A-CHESS to trigger a system response 
when consumers approach high-risk locations. These locations are identified and entered by the 
consumer. GPS monitoring will be used for no other purpose than avoidance of high-risk 
locations. The GPS data is strictly a feature of the A-CHESS app and will not be used for any 
type of analysis as part of this study.  Users are able to turn the GPS monitoring on or off (and 
similarly can choose whether or not to participate in other aspects of A-CHESS such as the 

23  
  
discussion group). The COVID-19 surveys do not involve GPS monitoring, but the patient 
survey will ask about patients’ zipcode to help understand their COVID-19 risk and community 
spread. All protected health information will be stored securely and only with a unique study ID 
number (not the patient’s name).  
 
There is also a risk of breach of confidentiality. Dissemination of some of the more sensitive 
information could result in risk to reputation.  To mitigate the risk of consumer breaches of 
confidentiality, all subjects will be assigned a code number. A list of subject code numbers will 
be maintained by the project director (Horst) and stored in a password-protected spreadsheet.   
 
There is a risk that information and resources consumers may divulge in A-CHESS will be used 
to the detriment of the subjects. Particular sources of risk include A-CHESS communications 
written within the discussion groups or personal profiles. Consumers will be instructed not to 
use their real names as a code name and will be warned of the potential dangers of divulging 
confidential information (e.g., real names or telephone numbers). An explicit statement 
regarding security and lack of confidentiality will be provided each time users enter discussion 
groups. These data will be collected by subjects' codename and will not be attached to real 
names or identities.  
 
Additionally, numerous methods are taken to assure the security of ACHESS data, COVID-19 
survey data, and patient in-depth interview recordings and transcriptions. The data from various 
components of A-CHESS will be housed on the secure CHESS servers at the University of 
Wisconsin. A database administrator (Wright) will be the only member of the evaluation team 
who will have direct access to this database. The CHESS Center, Dr. Molfenter, and Wright 
have extensive experience with HIPAA and IRB compliant practices for accessing, storing, and 
using personal health information. Furthermore, the A-CHESS registry is equipped with a log 
that records who accessed what data, at what time, and for how long as an added layer of 
security. Consumer data will be identified using a unique A-CHESS ID number (based on the A-
CHESS log-in). Similarly, survey and in-depth interview data will be stored securely as indicated 
above.   
 
In summary, during the on-line consent, potential subjects will be informed of (1) the nature and 
purpose of the study, (2) the types of data that will be collected from smart phone use tracking, 
(3) measures taken to insure the confidentiality of data collected (4) their right to leave the study 
at any time, and (5) the timeline of the study. Consent will be documented by obtaining online 
IRB-approved consent forms containing all of the above information. Consent forms will be 
digitally based. The A-CHESS system will store the consent form at the University Wisconsin – 
Madison on a secure server.  The consumer can download a PDF of the consent form. Upon 
signing the consent forms, the A-CHESS Site will provide a unique login name for the system, a 
mandatory requirement of HIPAA. A similar online consent will be provided to participants 
invited to complete the COVID-19 patient survey. Verbal consent with a trained research 
assistant will be used for in-depth interviews related to COVID-19. Copies of all consents will be 
provided to participants and stored securely.  
 
A Certificate of Confidentiality has automatically been granted by virtue of this study being 
funded through the National Institutes of Health.  This provides an additional level of protection 
for participant data. 
 
There is a risk that inaccurate or harmful information could be provided to consumers; to 

24  
  
mitigate this risk, all A-CHESS content has been screened by experts from our A-CHESS 
Consumer Advisory Committee, which has existed at the Center for 7 years. Despite the 
rigorous review process that occurs with all CHESS content, we will encourage study 
participants during training to contact their counselor or A-CHESS Site Coordinator if they are 
uncomfortable with the information provided by A-CHESS.  
 
Features such as a “panic button” could produce over-reliance on a system such as A-CHESS 
to provide support during periods of intense craving and when the perceived risk for relapse is 
high. We don’t expect that such incidences would be frequent. It is possible that a counselor 
and peers may not be immediately available to answer a call for help from an A-CHESS user. 
Study participants will be told that every effort will be made for their pre-designated support 
team to follow-up on their call for help. Additionally, if nobody in their immediate support network 
is able to respond, the individual calling for help will be forwarded to the central desk of their 
treatment agency and referred to other support people as needed. Study participants will be told 
during training that the panic button is for the most serious situations where relapse is imminent, 
and not to be used when other modes of communication (e.g., phone call, text message) will 
suffice.   
 
Study data collected at both the staff and consumer levels will be input to a SQL Server 
database housed at CHESS for analysis purposes. The University of Wisconsin database 
administrator will manage access to appropriate levels of study data for the evaluation team via 
password-protected accounts.   
  
7. MEASUREMENT OF EFFECT  
All scales used in this study have good psychometric properties with similar populations. Listed 
below are the factors to be measured and measurement instruments with references to 
validation studies.  
Descriptive Statistics - Organizational traits such as admission and rural v. urban will be 
obtained via IDPH data at four points during the study (N-SSATS55). Patient traits such as age, 
gender, ethnicity, and poverty level will also be obtained at four points during the study via IDPH 
data (TEDS56). Counselor traits such as gender, ethnicity, race, and certification or education 
level will be obtained via the organizational survey.  
REACH - The percentage of patients that download the A-CHESS app and frequency of use in 
days will be obtained monthly during the study via the A-CHESS server and IDPH data.50 
Adoption - The percentage of counselors using A-CHESS will be assessed via the 
organizational survey and A-CHESS logs.59    
Effectiveness - Retention (length of stay) will be assessed using IDPH data.61    
Maintenance - The organizational survey, which will be administered at two points during the 
study, will assess Financial Resource Availability.62 The staff survey will assess A-CHESS 
Implementation Fidelity (developed for trial) and Organizational Change Manager (OCM)63.The 
OCM and A-CHESS Implementation Fidelity will be administered at two points during the study.  
 
Patient COVID-19 Surveys – Surveys will assess how A-CHESS use may have lessened the 
impact of COVID-19 on anxiety and loneliness. We will collect several patient socio-

25  
  
demographic traits as recommended by NIH, including age, race, self-identified gender, sex at 
birth, approximate income level, educational attainment, sexual orientation, marital status, 
housing type, family size, health insurance. We will use several validated measures to assess 
patients’ substance use severity, loneliness, anxiety, and substance use disorder recovery 
strengths. Where validated measures are not available due to the rapidly emerging COVID-19 
situation, we have drawn measures from existing population-based studies recommended by 
the National Insititutes of Health.  
  
8. STUDY PARAMETERS  
The 16 participating Iowa SUD organizations will be assigned to the   
• 
Control arm (typical Product Training/On-line Support alone, n=8 organizations, n=20 
sites) or   
• 
Treatment arm (NIATx-TI framework plus Product Training/On-line Support, n=8 
organizations, n=20 sites)   
We will use a stratified randomization (or blocking with matched pairs). The matching criteria, 
using organizational characteristics and technology adoption traits, are:   
a) Innovation score per organization  
b) # of rural sites (due to the digital divide in these areas)  
c) # of outpatient sites per organization.   
 
The innovation score per organization will be calculated based on the level of “success “ (i.e. 
volume of patient enrollment) the organization had with implementing an online gambling portal 
in 2014. The implementation of the online gambling portal was in partnership with the Iowa 
Department of Public Health, who will provide gambling project enrollment numbers. This is an 
objective and relevant measure for innovativeness of the organizations as we will be 
implementing a mobile technology that will require the organizations to disperse the technology 
to their patients.  
 
 
We will apply urn randomization to achieve arm balance, using these 3 traits (e.g., # 
above/below mean for each trait or # of outpatient sites). An organization from each matched 
pair will be assigned to a study arm. Organizations (and their staff and patients) will have 
access to A-CHESS for the duration of the study regardless of which arm they are randomized 
to. Organizations and staff in the treatment arm will have access to NIATx-TI for 36 months.    
  
9. STATISTICAL CONSIDERATIONS  
Sample Size, Power & Data Analysis Plan: We will fit a mixed-effects model with the 
appropriate link function (e.g., linear, log) to the monthly results for the performance measures, 
% of eligible A-CHESS users and frequency of A-CHESS use, for each of the participating 
organizational sites, clustered by organization, and estimate the NIATx-TI framework and 

26  
  
product training effects. A mixed-effect model offers 2 advantages over conventional regression 
models: 1) The mean outcomes for a given treatment organization are expected to be 
correlated from month to month. A mixed-effect model allows for auto-correlated error terms. 2) 
Many unobservable treatment organizational site characteristics are likely to influence the key 
measures. Random effects at the treatment organization site level can be employed within the 
mixed-effect model to capture the correlation introduced by such characteristics. Proper 
reflection of auto-correlation and random effects allows for more reliable estimates of the fixed 
effects, starting at baseline, then again during the 18-month test and post-test periods. The 
power of the study design is determined by the anticipated standardized effect size based on 
effects experienced in the IRHIT project. The standardized effect size is calculated following 
Cohen’s definition of mean effect size divided by effect size standard error.65 The higher the 
standardized effect size, the more powerful the study design. Historically, cluster-randomized 
trials portray the standardized effect sizes of 0.2, 0.5 and 0.8 as small, medium, and large 
respectively. The IRHIT pilot data found that NIATx-TI increased client technology use rates by 
12.4% (increase of intervention success), which can be transformed into Cohen’s d = 0.352.66 
Intraclass correlation (ICC) among sites affects the power of cluster-randomized trials.67 An 
estimate of ICC is around 0.10.50 With a total of 8 organizations, with 20 organizational sites in 
each arm, where each site has an average 62 eligible patients, the study will achieve a power of 
.90 with a type I error rate of 0.05.   
Mediational (and Maintenance) analysis: We will examine the mediating effects of 
organizational factors of: % of counselors using A-CHESS, organizational readiness (via OCM), 
financial resource availability (via Klein’s inventory), and A-CHESS implementation readiness 
on A-CHESS use through a causal mediational analysis, using data from end of intervention 
periods. These factors will be analyzed using mixed-effects models. Through a mediation 
analysis,68,69 we can estimate the direct (A-CHESS use with NIATx TI and Product Training) and 
indirect effects (Adoption and Implementation factors) of each implementation arm on A-CHESS 
use. We will test the mediation effect of each potential mediator at each time point as well as 
across time. The R package ‘mediation,’ will estimate the causal mediation effects, examine 
moderated mediation effects, and conduct sensitivity analysis.70   
Effectiveness analysis: We will compare a retention rates in the 2 different arms using 
hierarchical linear models, where arm membership is used as the treatment assignment. We will 
examine the distribution of this outcome measure and will implement an appropriate 
transformation or nonlinear link function when appropriate (e.g., logistic, Poisson, exponential). 
Monthly data collection of the Effectiveness as well as Reach (A-CHESS Use) measures, allows 
for analysis after each stage of NIATx-TI.  
Qualitative Analysis: The qualitative portion of the study will support the specific aims by 
exploring how NIATx-TI and product training are implemented, how implementation affects 
ACHESS use, and how staff users react to NIATx-TI, product training, and A-CHESS. During 
the study, interviews with a randomly selected site in all organizations n=16 will occur at Ms 26-
30 (at the end of intervention phase) and 44-48 (at the end of sustainability phase), with the 
same sites in each phase, to gain a retrospective understanding of what occurred during 
implementation. The interviews will include closed and open-ended questions designed to 
explore participants’ experiences within each arm and the perceived effectiveness of each 
approach in assisting with implementing A-CHESS. The closed-ended questions will be based 
on elements of CFIR, developed by Damschroder et al. (2013),71 that will address: 1) 
intervention characteristics; 2) outer setting; 3) inner setting; and 4) process. The additional 
open-ended questions will allow for discovery of other factors affecting NIATx-TI, product 
training, and A-CHESS implementation. The information collected will inform our understanding 

27  
  
of implementation assets, barriers, and potential revisions for each approach. Interviews of up 
to 30 minutes will be conducted with the organization’s Executive Director, the site’s clinical 
director, the A-CHESS site coordinator, and a randomly selected full-time site counselor. Four 
interviews/organization will result in n=64 interviews per phase and up to n=80 interviews 
overall (assuming 25% turnover between phases).  
The qualitative analysis will be conducted by a team consisting of a qualitative methodologist 
(Jacobson), an outreach specialist from the Iowa Consortium for Substance Abuse Research 
and Evaluation, and Project Director from NIATx/CHESS. Jacobson will train the research team 
to assure that proper and consistent interviewing and analysis techniques are used. All 
interviews will be audio recorded and transcribed verbatim. Using a coding scheme derived 
from CFIR concepts, project staff will conduct a directed content analysis71 of these data. To 
assure coding consistency, every fourth interview will be double-coded. Significant 
inconsistencies will be discussed within the team and recoded. In addition, an inductive 
approach based on grounded theory and dimensional analysis72-74 will be used to identify and 
explore additional contextual and process factors that affect NIATx-TI and product training, as 
well as A-CHESS use. The inductive analysis will emphasize open and axial coding; coding 
divergences will provide opportunities for further reflection and exploration during team analysis 
meetings. Both directed and inductive analyses will include within-site, cross-site, and cross 
organization comparisons. Qualitative analysis will begin with the first interview and will continue 
in tandem with data collection, allowing the investigators to use later interviews to delve into 
factors that are identified in early interviews.  
The qualitative analysis will produce a description of the technology implementation process. It 
will encompass grounded explications of the theoretical concepts that influenced development 
of the NIATx-TI framework. This description will enhance our understanding of how the NIATxTI 
framework works, promoting valuable insights that can be applied to future dissemination of 
mobile e-health applications.  
 
Analysis of Supplemental COVID-19 Patient Survey and In-Depth Interview Data: Patient 
survey data will be matched with longitudinal app usage data. We anticipate using linear 
regression models to quantify the association between app usage (modeled continuously and 
binary as an indicator of meaningful use of A-CHESS) and several continuous outcomes based 
on validated measure – anxiety (PROMIS 29) and loneliness (UCLA Loneliness Scale-8). We 
will adjust for potentially confounding variables, including socio-demographic characteristics, 
recovery capital, criminal justice involvement, and substance use disorder severity. Further, we 
will use logistic or log-binomial regression models to assess how app usage is associated with 
substance use disorder treatment participation (e.g., AA/NA/self-help group participation, 
attendance at group therapy). 
  
10. RECORDS TO BE KEPT  
• 
Organizational traits  
• 
Patient/counselor traits  
• 
Survey results (Organizational and staff surveys, COVID-19 surveys)  
• 
A-CHESS Server data specific to project  

28  
  
• 
IDPH data  
 
  
  
 
 

29  
  
Appendix A: CHESS DATA SECURITY PLAN   
In accordance with the Health Insurance Portability and Accountability Act (HIPAA), CHESS has 
adopted the following Data Security and Monitoring Plan:  
Data Security Officer. Matthew Wright is the primary security officer for CHESS. Responsibilities 
of the security officer includes: developing information technology (IT) security policies, 
increasing security awareness for all CHESS staff, providing virus protection for IT resources, 
maintaining security patches on computing equipment, developing and implementing back-up 
procedures, performing periodic vulnerability scanning on IT equipment, reviewing and updating 
firewall strategies and policies, and enhancing the physical security of IT resources.  
Training. Before gaining access to our data, all of our students, faculty and staff must provide 
our Data Security Officer a copy of their certificates of completion of the UW-Madison Human  
Subjects online training  
(http://info.gradsch.wisc.edu/research/compliance/humansubjects/tutorial/) and the online 
HIPAA Privacy Rule training: http://www.wisc.edu/hipaa/ResearchGuide/index.html.  
Furthermore, they are required to complete training on CHESS security procedures and policies 
and sign a “CHESS Data Security Policy Certification” upon completion of this training.   
Workstation. All workstations will require login with a unique user name and password. Users 
will log-out from or lock workstations when leaving them unattended. Screen savers will be 
configured to require a password to activate after 10 minutes of workstation inactivity. Users 
requiring virtual private network (VPN) access to the CHESS network will only do so with 
computers specifically certified by a CHESS Data Security Officer.  
Passwords. Users require a password to access any computer connecting to the CHESS 
network. Passwords may not contain all or part of the user’s account name and must be at least 
7 characters long. Passwords may contain characters from 3 of the following four categories:  
English uppercase characters (A through Z); English lowercase characters (a through z); base 
10 digits (0 through 9); non-alphanumeric characters (e.g., !, $, #, %). User accounts will be 
locked after 5 failed login attempts and will reactivate automatically after 30 minutes of no failed 
attempts. Passwords may not be stored in proximity to the workstation and may not be shared 
by others.  
Data created by subjects as they use ACHESS. Data are not stored on smartphones, so there 
is no danger if a subject’s phone is stolen. Hackers could obtain data from the server, but 
useridentifying data are not stored on servers. We do not ask for identifying information (e.g. 
addresses, phone numbers) on ACHESS. Users receive code names and passwords. Users 
could walk away from their phone and not log out, leaving sensitive information on screen. We 
warn users (on enrollment, during training, and when they use the services) to not share 
personal data in the on-line discussions or ask-an-expert service and to close the browser if 
they leave ACHESS. All dynamic pages are configured so the browser stores a local copy only 
when they appear on screen. Cookies last only for the duration of the session. We use 
encryption software on our system so the website will be secure.  
Storage, Retrieval, and Disposal of Protected Information. Any computer-based Consumer 
Identifiable, Confidential, or Personnel Data will be stored on secure servers only and may not 
be stored on individual workstations. Currently, the only place such information can be stored is 
a secured section of the CHESS Intranet. Servers within CHESS will be located in locked rooms 
with limited access. The CHESS Data Security Officer will be responsible for assigning and 
restricting access to shared resources on the CHESS servers. Consumer Identifiable,  

30  
  
Confidential, and Personnel Data may not be copied to or stored on CHESS publicly accessible 
servers at any time. Remote access to files on secure CHESS servers may be provided only 
through a virtual private network (VPN) on a certified CHESS Workstation (See CHESS 
Workstation policy above for details). Storage media will be rendered unusable before disposal. 
All back-up media will be stored in locked rooms with limited access.  
Sending data by fax. All correspondence via fax must be stripped of consumer identifiers. The 
sender must call the receiver to ensure the receiver is available to receive the fax.  
  
  
  
  
 

31  
  
  
10. ACRONYM LIST  
  
ACHESS  
  
Addiction Comprehensive Health Enhancement Support  
 
System  
CBT4CBT  
  
Computer Based Training for Cognitive Behavioral Therapy  
  
CFIR  
  
Consolidated Framework for Implementation Research  
  
CHESS  
  
Center for Health Enhancement System Studies  
  
ICC  
  
Intraclass correlation  
  
IDPH  
  
Iowa Department of Public Health  
  
IRHIT  
  
Iowa Rural health Information technology Initiative   
  
NIATx-TI  
  
Network for the Improvement of Addiction Treatment- 
 
Technology Implementation  
N-SSATS  
  
National Survey of Substance Abuse Treatment Services  
  
OCM  
  
Organizational Change Manager  
  
PDSA  
  
Plan-Do-Study-Act  
  
RCT  
  
Randomized Control Trials  
  
RE-AIM  
  
Reach, Effectiveness, Adoption, Implementation, Maintenance     
RIS  
  
Readiness for Implementation (of Technology)  
  
SDT  
  
Self-Determination Theory  
  
SUD  
  
Substance Use Disorder  
  
TAM  
  
Technology Acceptance Model  
  
TAU  
  
Treatment as Usual  
  
TEDS  
  
Treatment Episode Data Set  
  
TES  
  
Therapeutic Education System  
  
  
 
 

32  
  
 
  
11. REFERENCES CITED   
  
1. 
Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. 
New England Journal of Medicine. 2010;363(6):501-4.  
2. 
Mintzberg H. Rethinking strategic planning part I: pitfalls and fallacies. Long Range 
Planning. 1994;27(3):12-21.  
3. 
Porter ME. How competitive forces shape strategy.  Readings in strategic management. 
Palgrave, London: Springer; 1989. p. 133-43.  
4. 
Mintzberg H, Waters JA. Of strategies, deliberate and emergent. Strategic Management 
Journal. 1985;6(3):257-72.  
5. 
Wen KY, Gustafson DH, Hawkins RP, Brennan PF, Dinauer S, Johnson PR, Siegler T. 
Developing and validating a model to predict the success of an IHCS implementation: the 
Readiness for Implementation Model. Journal of the American Medical Informatics 
Association : JAMIA. 2010;17(6):707-13. PMCID: PMC3000759.   
6. 
Ford JH, 2nd, Green CA, Hoffman KA, Wisdom JP, Riley KJ, Bergmann L, Molfenter T. 
Process improvement needs in substance abuse treatment: admissions walk-through 
results. Journal of Substance Abuse Treatment. 2007;33(4):379-89. PMCID: PMC2151921. 
7.  Center for Behavioral Health Statistics and Quality. Behavioral health trends in the 
United States: Results from the 2014 National Survey on Drug Use and Health. (HHS 
Publication No. SMA 15-4927, NSDUH Series H-50). Rockville, MD: Center for Behavioral 
Health  
Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2015.  
8. 
Stahre M, Roeber J, Kanny D, Brewer RD, Zhang X. Contribution of excessive alcohol 
consumption to deaths and years of potential life lost in the United States. Preventing 
Chronic Disease. 2014;11:E109. PMCID: PMC4075492.  
9. 
National Drug Intelligence Center (NDIC). The economic impact of illicit drug use on 
American society. Washington, DC: United States Department of Justice; 2011. Available 
from http://www.justice.gov/archive/ndic/pubs44/44731/44731p.pdf.  
10. Miller WR, Walters ST, Bennett ME. How effective is alcoholism treatment in the United 
States? Journal of Studies on Alcohol. 2001;62(2):211-20.   
11. Budney AJ, Marsch LA, Bickel WK. Computerized therapies: towards an addiction 
treatment technology test. In: el-Guebaly N, Carra G, editors. Textbook of Addiction 
Treatment: International Perspectives: Springer; 2015. p. 987-1006.  
12. Rainie L, Zickuhr K. Americans' views on mobile etiquette. Pew Research Center, 2015, 
Aug.  
13. Lindhiem O, Bennett CB, Rosen D, Silk J. Mobile technology boosts the effectiveness of 
psychotherapy and behavioral interventions: a meta-analysis. Behavior modification. 
2015;39`(6):785-804. PMCID: PMC4633319.  
14. Ramsey A, Gerke D, Proctor E. Implementation of technology-based alcohol interventions 
in primary care: a systematic review.  Addiction Health Services Researchers Conference; 
Marina Del Ray, CA2015.  
15. Miller EA. Solving the disjuncture between research and practice: telehealth trends in the 
21st century. Health Policy. 2007;82(2):133-41.  
16. Zanaboni P, Lettieri E. Institutionalizing telemedicine applications: the challenge of 
legitimizing decision-making. Journal of Medical Internet Research. 2011;13(3).  
17. Hebert MA, Korabek B, Scott RE. Moving research into practice: a decision framework for 
integrating home telehealth into chronic illness care. International Journal of Medical 
Informatics. 2006;75(12):786-94.  

33  
  
18. Hester RK, Delaney HD, Campbell W, Handmaker N. A web application for moderation 
training: Initial results of a randomized clinical trial. Journal of Substance Abuse Treatment. 
2009;37(3):266-76.  
19. Squires DD, Hester RK. Using technical innovations in clinical practice: The Drinker's 
Check‐Up software program. Journal of Clinical Psychology. 2004;60(2):159-69.  
20. Marsch LA, Guarino H, Acosta M, Aponte-Melendez Y, Cleland C, Grabinski M, Brady R, 
Edwards J. Web-based behavioral treatment for substance use disorders as a partial 
replacement of standard methadone maintenance treatment. Journal of Substance Abuse 
Treatment. 2014;46(1):43-51.  
21. Carroll K, Ball S, Martino S, Nich C, Babuscio T, Nuro K, Gordon M, Portnoy G, Rounsaville 
B. Computer-assisted delivery of cognitive-behavioral therapy for addiction: a randomized 
trial of CBT4CBT. American Journal of Psychiatry. 2008;165(7):881-8; PMCID: 
PMC2562873.  
22. Molfenter T, Capoccia VA, Boyle MG, Sherbeck CK. The readiness of addiction treatment 
agencies for health care reform. Substance Abuse Treatment, Prevention, and Policy. 
2012;7(1):1-8; PMCID: PMC3439296.  
23. Venkatesh V, Davis FD, Morris MG. Dead or alive? The development, trajectory and future 
of technology adoption research. Journal of the Association for Information Systems. 
2007;8(4):267.  
24. Oliveira T, Martins MF, editors. Information technology adoption models at firm level: review 
of literature. European Conference on Information Management and Evaluation; 2010: 
Academic Conferences International Limited.  
25. Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health 
care. Journal of Biomedical Informatics. 2010;43(1):159-72.  
26. Novak LL, Holden RJ, Anders SH, Hong JY, Karsh BT. Using a sociotechnical framework to 
understand adaptations in health IT implementation. International Journal of Medical 
Informatics. 2013;82(12):e331-e44.  
27. Khoja S, Scott RE, Casebeer AL, Mohsin M, Ishaq AFM, Gilani S. e-Health readiness 
assessment tools for healthcare institutions in developing countries. Telemedicine and 
eHealth. 2007;13(4):425-32.  
28. Brooks E, Turvey C, Augusterfer EF. Provider barriers to telemental health: obstacles 
overcome, obstacles remaining. Telemedicine journal and e-health : the official journal of 
the American Telemedicine Association. 2013;19(6):433-7.   
29. Jeyaraj A, Rottman JW, Lacity MC. A review of the predictors, linkages, and biases in IT 
innovation adoption research. Journal of Information Technology. 2006;21(1):1-23.  
30. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach 
G, Seydel ER. A holistic framework to improve the uptake and impact of eHealth 
technologies. Journal of Medical Internet Research. 2011;13(4):e111. PMCID: 
PMC3278097.  
31. Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information 
technology implementation framework in behavioral science: a systematic analysis of the 
literature on IT use. Journal of Biomedical Informatics. 2003;36(3):218-27.   
32. Hutzschenreuter T, Kleindienst I. Strategy-process research: what have we learned and 
what is still to be explored. Journal of Management. 2006;32(5):673-720.  
33. Porter ME. How competitive forces shape strategy. Harvard Business Review. 1979:137.  
34. Finger S, Dixon JR. A review of research in mechanical engineering design. Part I: 
Descriptive, prescriptive, and computer-based models of design processes. Research in 
Engineering Design. 1989;1(1):51-67.  

34  
  
35. Tsang EW. Organizational learning and the learning organization: a dichotomy between 
descriptive and prescriptive research. Human Relations. 1997;50(1):73-89.  
36. Bryson JM. Strategic planning for public and nonprofit organizations: a guide to 
strengthening and sustaining organizational achievement. New York: John Wiley & Sons; 
2011.  
37. Molfenter TD, Boyle MG, Holloway D, Zwick J. Trends in telemedicine use in addiction 
treatment. Addiction Science & Clinical Practice. 2015;10(1):14. PMCID: PMC4636787.  
38. Ford JH, 2nd, Alagoz E, Dinauer S, Johnson KA, Pe-Romashko K, Gustafson DH. 
Successful organizational strategies to sustain use of A-CHESS: a mobile intervention for 
individuals with alcohol use disorders. Journal of Medical Internet Research. 
2015;17(8):e201. PMCID: PMC4642385.  
39. Johnson K, Richards S, Chih MY, Moon TJ, Curtis H, Gustafson DH. A pilot test of a mobile 
app for drug court participants. Substance Abuse : research and treatment. 2016;10:1-7. 
PMCID: PMC4755700.  
40. Dennis ML, Scott CK, Funk RR, Nicholson L. A pilot study to examine the feasibility and 
potential effectiveness of using smartphones to provide recovery support for adolescents. 
Substance abuse : official publication of the Association for Medical Education and 
Research in Substance Abuse. 2015;36(4):486-92. PMCID: PMC4395503.  
41. Gustafson DH, Hawkins RP, Boberg EW, McTavish F, Owens B, Wise M, Berhe H, Pingree 
S. CHESS: 10 years of research and development in consumer health informatics for broad 
populations, including the underserved. International Journal of Medical Informatics. 
2002;65(3):169-77.   
42. Gustafson DH, McTavish FM, Stengle W, Ballard D, Hawkins R, Shaw BR, Jones E, 
Julèsberg K, McDowell H, Chen WC. Use and impact of eHealth system by low-income 
women with breast cancer. Journal of Health Communication. 2005;10(S1):195-218.  
43. Gustafson DH, Hawkins R, McTavish F, Pingree S, Chen WC, Volrathongchai K, Stengle 
W, Stewart JA, Serlin RC. Internet‐based interactive support for cancer patients: are 
integrated systems better? Journal of Communication. 2008;58(2):238-57.  
44. Gustafson DH, Hawkins R, Boberg E, Pingree S, Serlin RE, Graziano F, Chan CL. Impact 
of a patient-centered, computer-based health information/support system. American 
Journal of Preventive Medicine. 1999;16(1):1-9.   
45. Patten CA, Croghan IT, Meis TM, Decker PA, Pingree S, Colligan RC, Dornelas EA, Offord  
KP, Boberg EW, Baumberger RK, Hurt RD, Gustafson DH. Randomized clinical trial of an 
Internet-based versus brief office intervention for adolescent smoking cessation. Patient 
Education and Counseling. 2006;64(1-3):249-58.   
46. Japuntich SJ, Zehner ME, Smith SS, Jorenby DE, Valdez JA, Fiore MC, Baker TB, 
Gustafson DH. Smoking cessation via the internet: a randomized clinical trial of an internet 
intervention as adjuvant treatment in a smoking cessation intervention. Nicotine & tobacco 
research : official journal of the Society for Research on Nicotine and Tobacco. 2006;8 
Suppl 1:S59-67.   
47. Gustafson DH, McTavish FM, Schubert CJ, Johnson RA. The effect of a computer-based 
intervention on adult children of alcoholics. Journal of Addiction Medicine. 2012;6(1):24-8.   
48. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, 
social development, and well-being. American Psychologist. 2000;55(1):68.  
49. Marlatt GA, George WH. Relapse prevention: introduction and overview of the model. 
British Journal of Addiction. 1984;79(3):261-73.   
50. Gustafson DH, McTavish FM, Chih MY, Atwood AK, Johnson RA, Boyle MG, Levy MS, 
Driscoll H, Chisholm SM, Dillenburg L, Isham A, Shah D. A smartphone application to 

35  
  
support recovery from alcoholism: a randomized clinical trial. JAMA Psychiatry. 
2014;71(5):566-72. PMCID: PMC4016167.  
51. Mathews D, Evaluation Consultant. Evaluation Report: Combatting addiction with 
technology for pregnant Appalachian women using smartphones. Hazard, KY: Kentucky 
River Community Care, Inc., 2014.  
52. Chih MY, Patton T, McTavish FM, Isham AJ, Judkins-Fisher CL, Atwood AK, Gustafson 
DH. Predictive modeling of addiction lapses in a mobile health application. Journal of 
Substance Abuse Treatment. 2014;46(1):29-35.  
53. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion 
interventions: the RE-AIM framework. American Journal of Public Health. 
1999;89(9):13227. PMCID: PMC1508772.   
54. Baker TB, Gustafson DH, Shaw B, Hawkins R, Pingree S, Roberts L, Strecher V. 
Relevance of CONSORT reporting criteria for research on eHealth interventions. Patient 
Education and Counseling. 2010;81 Suppl:S77-86. PMCID: PMC2993846.   
55. Substance Abuse and Mental Health Services Administration (SAMHSA). The N-SSATS 
Report: Trends in the Use of Methadone and Buprenorphine at Substance Abuse 
Treatment Facilities: 2003 to 2011. Rockville, MD: Center for Behavioral Health Statistics 
and Quality; 2013.  
56. Substance Abuse and Mental Health Services Administration (SAMHSA). Treatment 
Episode Data Set (TEDS): 2002-2012. State admissions to substance abuse treatment 
services. HHS Publication No. (SMA) 14-4889. Rockville, MD: Substance Abuse and 
Mental Health Services Administration; 2014.     
57. McConnell KJ, Hoffman KA, Quanbeck A, McCarty D. Management practices in substance 
abuse treatment programs. Journal of Substance Abuse Treatment. 2009;37(1):79-89. 
PMCID: PMC2755538.   
58. Gustafson DH, Brennan PF, Hawkins RP. Investing in e-health: what it takes to sustain 
consumer health informatics. In: Hannah KJ, Ball MJ, editors. Health Informatics. New 
York: Springer Science+Business Media, LLC; 2007.   
59. Knight DK, Broome KM, Simpson DD, Flynn PM. Program structure and counselor-client 
contact in outpatient substance abuse treatment. Health Services Research. 
2008;43(2):616-34. PMCID: PMC2442366.  
60. Sobell LC, Sobell MB. Timeline Follow-Back. In: Litten RZ, Allen JP, editors. Measuring 
alcohol consumption: psychosocial and biochemical methods. Totowa, NJ: Humana Press; 
1992. p. 41-72.  
61. Simpson DD, Joe GW, Broome KM, Hiller ML, Knight K, Rowan-Szal GA. Program diversity 
and treatment retention rates in the Drug Abuse Treatment Outcome Study (DATOS). 
Psychology of Addictive Behaviors. 1997;11(4):279.  
62. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational 
analysis. The Journal of Applied Psychology. 2001;86(5):811-24.  
63. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and 
testing a model to predict outcomes of organizational change. Health Services Research. 
2003;38(2):751-76.  
64. Maher L, Gustafson D, Evans A. Sustainability, model and guide. Leicester, England: 
British National Health Service Modernization Agency; 2004.  
65. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: 
Lawrence Erlbaum Associates; 1988.  
66. Rosenthal R, Rubin DB. A simple, general purpose display of magnitude of experimental 
effect. Journal of Educational Psychology. 1982;74(2):166.  

36  
  
67. Hsieh FY, Lavori PW, Cohen HJ, Feussner JR. An overview of variance inflation factors for 
sample-size calculation. Evaluation & the Health Professions. 2003;26(3):239-57.   
68. MacKinnon DP. Introduction to statistical mediation analysis. New York, NY: Lawrence 
Erlbaum Associates; 2008.  
69. Imai K, Keele L, Tingley D. A general approach to causal mediation analysis. Psychological 
Methods. 2010;15(4):309-34.  
70. Tingley D, Yamamoto T, Hirose K, Keele L, Imai K. Mediation: R package for causal 
mediation analysis. Journal of Statistical Software. 2014;59(5):1-39.  
71. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program 
using the consolidated framework for implementation research (CFIR). Implementation 
Science. 2013;8:51. PMCID: PMC3656778.  
72. Glaser BG, Strauss A. The discovery of grounded theory: strategies for qualitative 
research. New York: Aldine Transaction Publishing; 1967.  
73. Strauss AL. Qualitative research for social scientists. Cambridge: Cambridge University 
Press; 1987.  
74. Schatzman L. Dimensional analysis: notes on an alternative approach to the grounding of a 
theory in qualitative research. Strauss AL, Maines DR, editors. New York: Aldine De 
Gruyter; 1991.