Scaling Up Science-based Mental Health Interventions in Latin America
Phase: NA
Conditions: Depression, Problematic Alcohol Use
Sponsor: Dartmouth-Hitchcock Medical Center

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
1
Official Title: 
Scaling Up Science-based Mental Health Interventions in 
Latin America 
NCT number: 
NCT03392883 
Document Type:  
Study Protocol and Statistical Analysis Plan 
Date of the 
Document: 
04/30/2021 
 
 
 
 
 
 
Scaling Up Science-based Mental Health 
Interventions in Latin America 
 
Implementation Research Project 
 
 
Lead Investigators: Lisa A. Marsch, PhD; Carlos Gómez-Restrepo, MD, PhD 
 
 
Supported by the National Institute of Mental Health, NIH 
 
1U19MH109988 
 
[April 14, 2021] 
Version 15.4 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
2
 
Principal Investigators: 
 
Lisa A. Marsch, PhD 
Director, Center for Technology and Behavioral Health 
Andrew G. Wallace Professor 
Geisel School of Medicine, Dartmouth College 
Lisa.a.marsch@dartmouth.edu 
 
Carlos Gómez-Restrepo, MD 
Director and Professor, Department of Clinical Epidemiology and Biostatistics  
Pontificia Universidad Javeriana 
cgomez@javeriana.edu.co 
 
Co-Investigators at Dartmouth College: 
 
William Torrey, MD 
Professor of Psychiatry 
Geisel School of Medicine, Dartmouth College 
William.C.Torrey@dartmouth.edu 
 
Leonardo Cubillos, MD 
Department of Psychiatry 
Geisel School of Medicine, Dartmouth College 
Leonardo.Cubillos@hitchcock.org 
 
Pablo Martinez Camblor, PhD 
Research Scientist  
The Dartmouth Institute for Health Policy and Clinical Practice  
Pablo.Martinez.Camblor@dartmouth.edu 
 
Andrea Meier, MS 
Director of Operations, Northeast Node Clinical Trials Network 
Center for Technology and Behavioral Health 
Dartmouth College 
Andrea.L.Meier@Dartmouth.edu 
 
 
Co-Investigators at Pontificia Universidad Javeriana: 
 
Ana María Medina, PhD 
Professor-Investigator  
Department of Medicine  
Pontificia Universidad Javeriana  
medina.ana@javeriana.edu.co 
  
Sergio Castro, MD 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
3
Psychiatrist 
Pontificia Universidad Javeriana 
sergiomariocastro@gmail.com 
 
Magda Cepeda, MD, PhD 
Research Coordinator and Study Clinician  
Pontificia Universidad Javeriana 
mccepedag@gmail.com 
 
Fernando Suarez Obando, MD 
Research Associate in Biomedical Informatics 
PhD Candidate, Department of Clinical Epidemiology and Biostatistics  
Pontificia Universidad Javeriana 
fernando.suarez@javeriana.edu.co 
 
NIMH Project Scientists 
 
Makeda J. Williams, PhD, MPH, CHES 
Program Officer 
Office for Research on Disparities and Global Mental Health 
Office of the Director 
National Institute of Mental Health 
willimak@mail.nih.gov 
 
Beverly Pringle, PhD 
Deputy Director, Office for Research on Disparities and Global Mental Health (ORDGMH)  
National Institute of Mental Health, NIH 
bpringle@mail.nih.gov 
 
NIMH Program Official: 
 
Holly Campbell-Rosen, PhD 
Program Officer  
National Institute of Mental Health 
Holly.campbell-rosen@nih.gov  
 
Expert Consultants: 
 
Miguel Uribe, MD, MPH 
Independent Consultant, formerly at Pontificia Universidad Javeriana 
miguel.uribe@gmail.com 
 
David C. Mohr, PhD 
Northwestern University 
Chicago, IL 
d-mohr@northwestern.edu 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
4
 
Paulo Rossi Menezes, MD 
University of São Paulo, Brazil 
paulo.menezes@fm.usp.br 
 
Additional Research Partners (in Chile and Peru): 
 
Humberto Castillo-Martell, MD 
Universidad Peruana Cayetano Heredia 
Lima, Peru 
walter.castillo@upch.pe 
 
July Caballero, MD 
Peruvian National Institute of Mental Health 
Lima, Peru 
julycaballero@hotmail.com 
 
Paulina Bravo, PhD 
Assistant Professor of Nursing 
Pontificia Universidad Católica de Chile  
bravop17@gmail.com 
 
Aixa Contreras, MSc 
Associate Professor  
School of Nursing 
Pontificia Universidad Católica de Chile  
aixacont@gmail.com 
 
Jaime C. Sapag Muñoz, MD, MPH, PhD 
Departments of Public Health and Family Medicine 
Public Health and Family Medicine Divisions 
School of Medicine, Universidad Católica de Chile  
jsapag@med.puc.c 
 
Rubén Alvarado, PhD 
School of Public Health 
University of Chile 
ralvarado@med.uchile.cl 
 
Dr. Alvaro Aravena, MD 
Psychiatrist, Universidad de Santiago de Chile. 
Head of Mental Health Unit, Aconcagua Health Services, Chile 
dr.aaravena@gmail.com 
 
 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
5
TABLE OF CONTENTS 
1.0 
LIST OF ABBREVIATIONS ......................................................................................................... 8 
2.0 
PRÉCIS ............................................................................................................................................ 9 
2.1 
Overall Study Objective ........................................................................................................................... 9 
2.2 
Study Intervention .................................................................................................................................... 9 
2.3 
Study Design, Population, Outcomes, and Sample Size .......................................................................... 9 
2.4 
Potential Impact ...................................................................................................................................... 10 
3.0 
PROJECT ORGANIZATION, ROLES AND RESPONSIBILITIES ...................................... 10 
4.0 
INTRODUCTION ......................................................................................................................... 12 
4.1 
Background and Significance to the Field .............................................................................................. 12 
5.0 
OBJECTIVES ................................................................................................................................ 14 
5.1 
Primary Objectives ................................................................................................................................. 14 
Administrative Core ....................................................................................................................................... 14 
Scale-Up Core ................................................................................................................................................ 14 
Capacity Building Core .................................................................................................................................. 14 
5.2 
Secondary Objective(s) .......................................................................................................................... 14 
6.0 
FORMATIVE RESEARCH ......................................................................................................... 15 
6.1     Technology Usage/Access Survey in Colombia………………………………………………15 
6.2.     Focus Groups and In-Depth Interviews with Multiple Stakeholders…………………………16 
6.3.     Behavioral Health Integration in Medical Care (BHIMC) assessment………………………19 
 
7.0 
STUDY DESIGN ........................................................................................................................... 20 
7.1 
Summary Overview of Implementation Study Design........................................................................... 20 
8.0 
COMPONENTS OF MENTAL HEALTH CARE MODEL TO BE IMPLEMENTED ........ 21 
9.0 
STUDY POPULATION FOR IMPLEMENTATION STUDY ................................................. 26 
9.1 
Provider and administrative staff Inclusion Criteria: ............................................................................. 26 
9.2 
Participant Inclusion Criteria:................................................................................................................. 26 
9.3 
Participant Exclusion Criteria................................................................................................................. 26 
9.4 
Participant Recruitment, Screening and Informed Consent ................................................................... 27 
10.0 
SITE SELECTION........................................................................................................................ 30 
10.1 
Number of Sites ...................................................................................................................................... 31 
10.2 
Site Characteristics ................................................................................................................................. 31 
10.3 
Rationale for Site Selection .................................................................................................................... 32 
10.4 
Wi-Fi Expansion to Rural Sites .............................................................................................................. 32 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
6
11.0 
STUDY PROCEDURES ............................................................................................................... 33 
11.1 
Pilot Implementation Study .................................................................................................................... 33 
11.2 
Implementation Research ....................................................................................................................... 33 
11.3 
Participant Reimbursement .................................................................................................................... 35 
12.0 
STUDY ASSESSMENTS .............................................................................................................. 35 
12.1 
Quantitative Outcome Measures ............................................................................................................ 36 
12.1.1 Summary of Quantitative Outcome Measures..................................................................................... 36 
12.1.2 Quantitative Implementation Measures ............................................................................................... 37 
12.1.3 Cost of Implementing the Model of Care ............................................................................................ 39 
12.2 
Qualitative Outcome Measures .............................................................................................................. 43 
12.2.1. Administrative/provider Interviews .................................................................................................... 43 
12.2.2. Patient Utilization Interviews/observations ........................................................................................ 43 
13.0 
STATISTICAL DESIGN AND ANALYSES .............................................................................. 43 
13.1 
Statistical Analyses. ................................................................................................................................ 44 
13.2 
Rationale for Sample Size and Statistical Power.................................................................................... 44 
13.3 
Interim Analysis ..................................................................................................................................... 45 
13.4 
Exploratory Analysis .............................................................................................................................. 45 
13.5 
Missing Data and Dropouts .................................................................................................................... 45 
13.6 
Demographic and Baseline Characteristics ............................................................................................ 46 
13.7 
Safety Analysis ....................................................................................................................................... 46 
13.8 
Incidental Findings ................................................................................................................................. 46 
14.0 
INFRASTRUCTURE SUPPORT ................................................................................................ 46 
14.1 
Stakeholders ........................................................................................................................................... 46 
14.2 
Project Launch and Annual Team Meetings .......................................................................................... 47 
14.3 
Administrative Meetings ........................................................................................................................ 47 
14.4 
Learning Collaborative Meetings ........................................................................................................... 47 
14.5 
Research Capacity-building Meetings .................................................................................................... 47 
14.6 
Annual NIH Meetings ............................................................................................................................ 47 
14.7 
Partnership and Advisory Board Meetings ............................................................................................. 48 
15.0 
REGULATORY COMPLIANCE AND SAFETY ..................................................................... 48 
15.1 
Regulatory Compliance .......................................................................................................................... 48 
15.2 
Statement of Compliance ....................................................................................................................... 48 
15.3 
Institutional Review Board Approval ..................................................................................................... 48 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
7
15.4 
Informed Consent ................................................................................................................................... 49 
15.5 
Confidentiality ........................................................................................................................................ 52 
15.6 
Investigator Assurances .......................................................................................................................... 52 
15.7 
Financial Disclosure ............................................................................................................................... 52 
15.8 
Clinical and Quality Monitoring ............................................................................................................ 53 
15.9 
Inclusion of Women and Minorities ....................................................................................................... 53 
15.10 
Regulatory Files ................................................................................................................................. 54 
15.11 
Records Retention and Requirements ................................................................................................ 54 
15.12 
Reporting to Sponsor .......................................................................................................................... 54 
15.13 
Audits ................................................................................................................................................. 54 
15.14 
Study Documentation ......................................................................................................................... 54 
15.15 
Protocol Deviations ............................................................................................................................ 54 
15.16 
Safety Monitoring ............................................................................................................................... 55 
Data and Safety Monitoring Board (DSMB) .................................................................................................. 55 
Adverse Events (AEs) .................................................................................................................................... 55 
16.0 
POTENTIAL RISKS AND PROTECTIONS AGAINST RISK ............................................... 60 
17.0 
ANTICIPATED BENEFIT .......................................................................................................... 62 
17.1 
Potential Benefits of the Proposed Research to Human Subjects and Others ........................................ 62 
17.2 
Importance of the Knowledge to be Gained ........................................................................................... 62 
18.0 
REFERENCES .............................................................................................................................. 62 
19.0 
APPENDIX .................................................................................................................................... 69 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
8
1.0 
LIST OF ABBREVIATIONS  
This section contains a comprehensive, alphabetical list of any abbreviations used throughout the protocol 
document.  
 
AEs 
 
Adverse Events 
AUDIT 
Alcohol Use Disorder Identification Test 
BA 
 
Behavioral Activation  
BHIMC 
Behavioral Health Integration in Medical Care  
CFIR  
Consolidated Framework for Implementation Research 
CBT 
 
Cognitive Behavioral Therapy 
CRA  
Community Reinforcement Approach to Behavioral Therapy 
DALYs 
Disability-Adjusted Life Years 
DSMB  
Data and Safety Monitoring Board 
EPIS  
Exploration, Preparation, Implementation and Sustainment Framework 
GAD-7 
Generalized Anxiety Disorder screener 
GCP  
Good Clinical Practice 
IRB 
 
Institutional Review Board 
HPQ  
Health and Work Performance Questionnaire 
LMICs  
Low and Middle Income Countries 
mhGAP 
Mental Health Gap Action Program  
NIH 
 
U.S. National Institutes of Health 
NIMH  
National Institute of Mental Health, U.S. National Institute of Health 
NSMOS 
Non-Study Medical and Other Services 
PAHO  
Pan American Health Organization 
PHQ-9  
Patient Health Questionnaire-9 
PST 
 
Problem Solving Therapy 
QDS  
Quick Drinking Screen 
RE-AIM 
Reach, Effectiveness, Adoption, Implementation, and Maintenance framework 
SAEs  
Serious Adverse Events 
TDABC 
Time-driven Activity-based Costing Approach 
WHO  
World Health Organization 
WHODAS 
World Health Organization Disability Assessment Schedule 
 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
9
2.0 
PRÉCIS 
2.1 
Overall Study Objective 
Conduct systematic, multi-site mental health implementation research in both rural and urban primary 
care settings with a broad group of stakeholders in the US and Latin America. 
2.2 
Study Intervention  
We plan to launch and evaluate a new mental health service delivery model in Latin America.  
Specifically, in this new multi-component, mental health service delivery intervention, we propose to: 
(1) harness mobile behavioral health technology for mental health (with a primary focus on depression 
and a secondary focus on problematic alcohol and other substance abuse), (2) launch new workforce 
training and service delivery models (including the integration of technology into service delivery), (3) 
launch and evolve an integrated data management system for systematic data tracking and outcomes 
assessment, and (4) launch and grow a learning collaborative of organizations integrating mental health 
into primary care.  We will launch this project at multiple primary care sites in various parts of 
Colombia, with a plan to inform subsequent adoption in several other Latin American countries, 
including Chile and Peru. 
2.3 
Study Design, Population, Outcomes, and Sample Size 
We will initially pilot test this mental health service delivery model at a single primary care site and 
then refine the model based on pilot data. We will then expand implementation across 6 Colombia-
based healthcare sites in urban and rural communities (including at least 1200 participants).  Consistent 
with a modified stepped wedge design (multiple baseline design), we will implement the model across 
sites on a staggered basis and expand the number of sites in which we implement over time.  By 
conducting this multi-site implementation research project, we can assess the extent to which the 
implementation model and associated outcomes are replicable across sites and/or the extent to which 
the model needs to be modified for differing contexts. 
 
We will evaluate the ability of our proposed approach to accelerate the translation of evidence-based 
mental health services into practice and expand research capacity at multiple levels.   Specifically, we 
will measure implementation outcomes, including: (1) the ability of this model to accelerate the 
adoption of science-based mental health service delivery (e.g., increase provision of evidence-based 
resources to more individuals), (2) the acceptability of the model for healthcare service delivery (the 
ability of the technology-based service-delivery model to increase patient activation and engagement in 
their own self-management), and (3) the cost-evaluation of services in the model.  We will also collect 
patient level data outcomes as a secondary focus to assess (4) the model’s impact on public/population 
health (e.g., its effects on accelerating improved behavioral health and health outcomes, and improving 
patient quality of life, functioning, etc.). 
 
We will also use implementation context measures to evaluate barriers and facilitators to 
implementation in each of the following domains: intervention characteristics, organization 
characteristics (e.g., climate, readiness), individual characteristics (i.e., provider/staff attitudes, 
experiences; patient attitudes and experiences), and external influences (e.g., socio-political 
characteristics local technology infrastructure i.e., wireless in the community, state policy/regulations, 
and reimbursement models).  Strategies related to planning, facilitating provider and patient 
engagement in use of the novel mental health service delivery model, and potential sustainability issues 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
10
will also be explored.  We will additionally conduct exploratory moderator/mediator analyses to 
examine how implementation context variables relate to implementation outcomes.   
2.4 
Potential Impact  
Overall, this project will create new knowledge to inform unprecedented, science-based approaches to 
scaling-up mental health implementation research and building sustainable research capacity and 
science-based policies and programs in Latin America. This project brings together an outstanding 
expert team to test and refine an entirely new model for delivering widespread, science-based, mental 
health care in Latin America. This project may also serve as an important demonstration project to low 
resource settings globally as they tackle the significant burden of mental health disorders and scale-up 
access to evidence-based models of mental health service delivery. 
3.0 
PROJECT ORGANIZATION, ROLES AND RESPONSIBILITIES 
 
As illustrated in the Organizational chart, the Project Director’s Office will provide overall 
leadership for the project.  The Multiple Principal Investigators [PIs] (Lisa A Marsch, PhD and 
Carlos Gómez Restrepo, MD) will constitute the Project Director’s Office.  In their role in the Director’s 
Office, the Multiple PIs will collaborate to integrate efforts across sites, project collaborators, and 
project activities.  The PIs will be responsible for ensuring all project Aims are successfully realized.  
Dr. Marsch will serve as Contact PI and will be responsible for ensuring all NIMH, IRB and other 
reporting requirements are successfully met.  Dr. Marsch will also be primarily responsible for 
overseeing all proposed activities to be conducted out of Dartmouth College (e.g., data analyses; 
supervision of data collection; supervision of implementation of novel mental health service delivery 
model), while Dr. Gómez Restrepo will be responsible for overseeing all proposed activities to be 
conducted out of Pontificia Universidad Javeriana in Colombia (oversight of all local research staff and 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
11
local implementation research and capacity building activities).  The Multiple PIs will also coordinate 
project activities with NIMH staff and will coordinate the engagement of the Consultants throughout 
the life of the project.  The Director’s Office will also assume primary responsibility for facilitating 
activities designed to synthesize research findings and promote joint publications and presentations 
among project Investigators.   The Office will also work to resolve any potential disputes/concerns that 
may arise.  
 
Consultants and Investigators.  The Director’s Office will be advised by a team of expert consultants 
(Miguel Uribe, MD; David C. Mohr, PhD, Paulo Rossi Menezes, MD).  As detailed in his 
biographical sketch, Dr. Uribe (in partnership with Gómez Restrepo) has conducted extensive work on 
integrating depression care into primary care in Colombia and currently serves as an advisor to the 
World Bank in Washington, DC.   We have further engaged both US and Latin American-based 
research teams conducting implementation research that employs mobile health technology in several 
regions of Latin America (Dr. David Mohr at Northwestern University in Chicago and Dr. Paulo Rossi 
Menezes at the University of São Paulo in Brazil).  This partnership will leverage the investment of 
NIMH in these collective activities to enable a new level of discovery and accomplishment across 
projects and offer the opportunity to accelerate the tempo and scientific achievement from this line of 
research.  
 
The Director’s Office will also be advised by members of the investigative team at both Dartmouth and 
Pontificia Universidad Javeriana.   At Dartmouth, this includes William Torrey, MD a psychiatrist with 
extensive experience using innovative models for integrating mental health care into primary care; 
Pablo Martinez, PhD, a statistician at the Dartmouth Institute of Health Policy and Clinical Practice; 
and Leonardo Cubillos, MD, a psychiatrist formerly working at the World Bank in Washington DC in 
the areas of poverty and mental health in Latin America (Colombia is his birthplace) and now a 
psychiatrist with our group at Dartmouth. Andrea Meier serves as a trainer/monitor of several 
assessment activities. 
 
At Pontificia Universidad Javeriana, Dr. Sergio Castro (a psychiatrist), Dr. Fernando Suarez Obando (a 
PhD student under Dr. Gómez Restrepo), Dr. Ana María Medina (an anthropologist), and Dr. Magda 
Cepeda (a doctor with a PhD in epidemiology) will serve as Co-Investigators and project directors 
overseeing research staff activity in the implementation trial (under the leadership of Dr. Gómez 
Restrepo). 
 
Governmental, Non-Governmental and Multilateral Organizations. The Director’s Office will also 
be advised by a broad array of stakeholders including academic organizations (Dartmouth College in 
the US; Pontificia Universidad Javeriana in Colombia; Universidad Peruana Cayetano in Peru; 
Pontificia Universidad Catolica in Chile), governmental organizations (Ministry of Health in Colombia; 
National Institute of Mental Health in Peru), as well as non-governmental and/or multilateral 
organizations (PAHO/WHO; The World Bank; industry partners; and primary care and regional 
hospital systems in Latin America).  
 
We will also include perspectives of a non-profit patient advocacy organization (“Fundacion 
Internacional Unidos Contra la Depresión” – The International Foundation United Against Depression) 
and a public/private non-profit organization focused on quality decision-making in clinical practice and 
health policy (“Instituto de Evaluación Technológica en Salud” – The Institute of Technical Evaluation 
in Health).   We had additionally engaged several large insurance companies in Colombia, which will 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
12
be key to the sustainability of this work.    
 
Overall, we have been fortunate to assemble an outstanding and broad team of collaborators and 
stakeholders which greatly enhances our ability to achieve the goals of the proposed project. These 
partnerships will also ensure that the study design is responsive to local needs, resources and expertise 
and enables synergies beyond what can be achieved through a traditional research study.   
 
Advisors from Chile and Peru.  Although implementation activities will primarily take place in 
Colombia, experts from the countries of Chile and Peru will participate as collaborators throughout the 
entire project as part of research capacity-building activities (to inform implementation and shared 
learning in other parts of Latin America).   
 
National Institute of Mental Health (NIMH) Project Scientists.  As per the Cooperative Agreement, 
the project team will work closely in partnership with NIMH Project Scientists on the execution of the 
proposed work and the evolution/refinement of this work over time based on NIH input, learning based 
on data collected over time, and shared learning among grantees under this funding opportunity (Drs. 
Denise Pintello and Beverly Pringle).  We will also work closely with NIMH Program Official Dr. 
Holly Campbell-Rosen. 
 
Administrative Staff within the Core.  Program coordinator administrative support staff will work 
under the leadership of the Director’s Office to support the Administrative Core.  One part-time support 
staff will work at Dartmouth under the guidance of Dr. Marsch and the other will work at Pontificia 
Universidad Javeriana in Colombia under the guidance of Dr. Carlos Gómez Restrepo.   These 
individuals will coordinate logistics for all in-person and webinar meetings, arrange space needs, oversee the 
timeline, organize all project publications and presentations, IRB approval and budget on project activities.  
4.0 
INTRODUCTION  
4.1 
Background and Significance to the Field  
Mental health disorders are increasingly recognized as a major cause of the global burden of disease, 
accounting for an estimated 7.4% of the disease burden worldwide1 and significantly contributing to 
disability and death.  The overall annual economic cost for mental health disorders is estimated at over 
$300 billion and is expected to exceed $16 trillion over the next 20 years.2, 3  Expanding access to 
mental health care in a manner that can rapidly scale and have a substantial population impact is a 
significant challenge globally.4  
 
This challenge is particularly evident in low and middle-income countries (LMICs) where more than 
half of the world’s population live.5-7   Between 76-85% of persons with severe mental disorders in 
LMICs receive no treatment at all for these problems.  The healthcare workforce in LMICs is also 
grossly insufficient – with an average of one psychiatrist to serve every 200,000 people and even fewer 
mental health providers trained in the delivery of psychosocial interventions.    Additionally, only 36% 
of persons in LMICs are covered by mental health legislation, in stark contrast to 92% of persons in 
high-income countries. 
 
Latin America is one region of the world composed of LMICs where the burden of mental health 
problems is high and services for mental health are low (and account for <2% of the health budget in 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
13
the region).8   Over 37%, 59% and 71% of persons in Latin America with severe mental illness, major 
depression, and alcohol use disorders, respectively, are in need of treatment but do not receive care.9  In 
the Latin American country of Colombia, non-communicable diseases account for over 61% of total 
disability-adjusted life years (DALYs), with mental health disorders accounting for almost 13% of 
those DALYs.    A recent National Mental Health Survey in Colombia, conducted in 2003,10 showed 
that 8 out of 20 Colombians has a lifetime prevalence of one or more mental health disorders.  
Depression and substance use, often coupled with other mental health problems resulting from endemic 
violence in Colombia, are particularly striking concerns in the region.  Only 11%10 of persons with a 
mental health disorder in Colombia receive mental health care.   
 
In 2008, the World Health Organization (WHO) launched the Mental Health Gap Action Program 
(mhGAP) including strategies and activities to scale up care for mental, neurological, and substance 
abuse disorders – with a priority focus on LMICs.11  Recognizing the high burden of disease from 
mental disorders in the Latin American region and the close relationship between mental health and 
physical health, the Directing Council of the Pan American Health Organization (PAHO) within the 
WHO adopted a Plan of Action on Mental Health in its Strategic Plan 2014-2019.12  The Strategy and 
Plan of Action calls for Member States to position mental health as a priority and lays the route to 
improve mental health programs in the region over the next 10 years.   
 
Colombia has embraced this plan and is currently preparing to launch (under the leadership of the 
Colombian Ministry of Health) a new model of mental health care (called “Modelo Integral en 
Atención en Salud” – Integrated Model of Health Care)13 that will reinforce primary care-based mental 
health care – a long-standing weakness in the Colombian health system.    The articulated goals of the 
model are to ensure access to care, continuity of care, integration of care, capacity to resolve health 
problems and quality of care.   Ongoing measurement of patient outcomes is another priority within the 
model.   
 
In order to achieve the goals of the model, a key element is to leverage empirically-supported mobile 
behavioral health technology to create an entirely new model of mental health service delivery within 
primary care.   Indeed PAHO noted the need to “study and maximize use of technology (mobile 
phones…and Internet) to guarantee access to mental health services in remote and neglected 
communities.”12  Specifically, advances in digital technologies have created unprecedented 
opportunities to assess and modify behavioral health at a population level and thus facilitate the rapid 
and widespread “scaling-up” of evidence-based mental health care.  Over 90% of persons worldwide 
have access to mobile phone services, with Colombians having an average of 108.3 phones for every 
100 inhabitants (an average of more than one phone per person).  Growing evidence suggests that 
increased access to these technologies is also evident in many traditionally underserved populations 
where disparities are prevalent.   
 
Another key element of Colombia’s mental health care model is human resource development focused 
on the creation of mental health skills within primary care teams.  Given the strikingly limited 
availability of mental health trained workforce in Colombia and elsewhere in Latin America (as 
reviewed above), raising awareness of evidence-based screening and interventions for mental health 
within the primary care workforce is critical to integrated care.   
 
Similar models of mental health care are in various stages of planning in other Latin American 
countries, including Chile and Peru.14-16 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
14
 
The proposed project was developed in response to the striking mental health treatment service delivery 
needs in Latin America broadly (and Colombia in particular) and the strong commitment to making 
substantive changes in the region. 
 
As detailed in this protocol, we will (1) harness mobile behavioral health technology for mental health 
(with a primary focus on depression and a secondary focus on problematic alcohol use) (2) launch new 
workforce training and service delivery models (including the integration of technology into service 
delivery), (3) launch and evolve an integrated data management system for systematic data tracking and 
outcomes assessment, and (4) launch and grow a learning collaborative of organizations integrating 
mental health into primary care.   We will launch this project at multiple primary care sites in various 
parts of Colombia, with a plan to inform potential subsequent adoption in several other Latin American 
countries, including Chile and Peru. 
5.0 
OBJECTIVES  
5.1 
Primary Objectives 
Administrative Core  
Support a core organizational structure and management approach to maximally benefit from a broad 
array of stakeholders and ensure efficient and successful coordination and integration of the activities 
across project Cores  
 
Aim1a: To provide scientific and programmatic leadership to ensure efficient and successful 
coordination and integration of the activities across the Hub’s projects.  
  
Aim 1b:  To provide a novel infrastructure to enhance synergy among an interdisciplinary expert team 
by facilitating productive communication, centralization of knowledge and resources, and integration of 
methods and results across research activities 
 
Scale-Up Core  
Aim 2. Conduct systematic, multi-site mental health implementation research in both rural and urban 
primary care settings with a broad group of stakeholders in the US and Latin America 
 
Capacity Building Core  
Use science-based methods and information to build sustainable capacity for conducting mental health 
implementation research and informing mental health policies and programs in Latin America   
 
Aim 3a:  Establish resources and an infrastructure to aid Colombia and its regional partners of Chile 
and Peru to build capacity for mental health implementation research   
 
Aim 3b:   Build sustainable capacity to use science-based methods and information for developing 
mental health policies and programs 
5.2 
Secondary Objective(s)  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
15
A secondary Aim of this project is to engage in ongoing dialogue and shared learning activities with other 
“hubs” of grantees funded under this same NIMH funding mechanism, focused on scaling up evidence-
based mental health care in low and middle-income countries.  
 
This project protocol focuses primarily on the planned human subjects activity within the Scale-
Up Core (multi-site mental health implementation research project). 
6.0 
FORMATIVE RESEARCH  
In preparation for the Scale-Up Core’s multi-site implementation research project, our team has 
engaged in an array of formative research activities.   These activities are briefly described in this 
section.  These activities launched after a leadership planning team held at Dartmouth College in June 
2016, followed by our all-team project kick-off meeting and initial industry advisory board meeting in 
Bogotá, Colombia in September 2016.  All formative research activities were reviewed and 
appropriately approved by the Institutional Review Boards at Dartmouth College as well as Pontificia 
Universidad Javeriana (and study sites, as needed).  These preparatory activities were deemed exempt 
from review by the NIMH Data and Safety Monitoring Board (DSMB). 
 
These formative research activities included:  
 
Technology Usage/Access Survey in Colombia 
 
Focus Groups and In-Depth Interviews with Multiple Stakeholders 
 
Behavioral Health Integration in Medical Care (BHIMC) assessment: Adapation for 
Colombia and baseline assessment at primary care study sites 
 
We will discuss each of these activities, in turn. 
 
6.1. Technology Usage/Access Survey in Colombia 
 
We conducted an anonymous survey in Colombia to examine the extent to which patients in primary care 
systems use Internet, mobile devices, such as smartphones (e.g., iPhone, Android, Blackberry) and tablet 
computers (e.g., iPad), to answer clinical questions and find medical and health information.  
 
The specific objectives of this project were to: 
 
• Evaluate socioeconomic characteristics of patients and their possible relation to patterns of technology use 
 
• Establish health care service conditions of patients in the primary care network  
 
• Assess patient's mobile device use for finding medical information related to general health and mental health 
 
Methods  
Survey sites  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
16
We recruited participants to complete the survey from 6 sites.  Specifically, we recruited participants in Bogotá 
D.C. at 1) Fundación Javeriana de Servicios Médicos ondontológicos inter-universitarios “Carlos Márquez 
Villegas” Javesalud (The Javeriana Foundation of Inter-University Odontologic Medical Services).  We also 
recruited participants in two health care systems in the town of Duitama (Boyacá): 2) Empresa Social del Estado 
Hospital Regional de Duitama (The State Social Services Regional Hospital Of Duitama) and 3) Santa Rosa de 
Viterbo Regional Health Center.  And, we recruited participants in three healthcare systems in Tolima: 4) 
Hospital Armero-Guayabal, 5) Hospital de Chaparral, and 6) Hospital Granja de Lérida, Primer Nivel.  These 
regions span both urban and rural (e.g., farming) communities and thus enable broader generalizability of survey 
results. 
 
Our research team at Javieriana University has academic partnerships between the Javeriana University and 
Javesalud, Hospital Regional de Duitama and Santa Rosa.  Via this project, the research team at Javeriana has 
established partners with Lerida, Chaparral, and Guayabal.  Javesalud is in Bogatá, 30 minutes from Javeriana.  
Duitama and Santa Rosa are both about 3 hours from Bogotá, and Javeriana has interns from the School of 
Medicine in each of these locations and the Director of the interns in Duitama is part of our research project as a 
Psychiatrist in Lerida. 
 
Survey design and implementation  
The survey explored facilitators of mobile device use in medical information seeking, barriers to access, internet 
connection conditions, familiarity with medical resources, and most frequently used resources. The questions in 
the survey were designed by the research team members, based on prior similar survey studies (such as 
technology assessments conducted in local hospitals and/or low and middle income countries, as well as the 
Colombian mental health survey).    
 
The survey was implemented via a computerized survey engine (LimeSurvey) delivered on a tablet to patients 
recruited in waiting rooms of the participating primary care survey sites.  At each site, a research assistant asked 
patients if they would be willing to complete the survey during their time in the waiting room before their 
medical appointment.  The research assistant explained the nature of the survey and was available to supervise 
the process and answer any questions from patients, as needed.  The survey was self-administered and 
anonymous and generally took about 10-15 minutes to complete.   
 
As of the time of submission of this protocol for review by the NIMH Data and Safety Monitoring Board 
(DSMB), we have completed participant recruitment for this survey and have a total of 1,580 completed surveys 
from patients across our survey sites. These data have been helpful in planning our implementation project across 
sites. For example, these data have been helpful in understanding the likelihood that patients can access mobile 
therapeutic tools for mental health on smartphones (as opposed to other sources, such as computers set-up onsite 
at primary care sites) as well as patients’ interest in using digital therapeutic tools for mental health. Based on 
data collected to date, we expect that more patients in Javesalud will have access to smartphones compared to 
patients in more rural settings (where access to the mobile intervention may include computer access at primary 
care sites).   
 
6.2. Focus Groups and In-Depth Interviews with Multiple Stakeholders 
 
Objective 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
17
We conducted qualitative research to explore perceptions, abilities, attitudes, practices and experiences 
of health professionals, administrators, patients and community organizations on access, continuity, 
integration, quality and resolution of problems in mental health in primary care.  This research was 
intended to understand existing models of providing screening and mental health care to patients in 
Colombia.  As part of this research, participants were asked to discuss the possible use of mobile health 
technologies for assessment and follow-up support in the treatment of depression and alcohol use 
disorders.   
 
Methods 
We primarily used a focus group methodology with four different groups of participants (as detailed 
below). In addition, we performed semi-structured in-depth interviews to deepen our understanding of 
important aspects identified in the focus groups. 
 
Survey sites  
We recruited participants for this qualitative research from the same 6 sites where we conducted the technology 
survey described above (to enable broad sampling and increase generalizability):  1) Fundación Javeriana de 
Servicios Médicos ondontológicos inter-universitarios “Carlos Márquez Villegas” Javesalud (The Javeriana 
Foundation of Inter-University Odontologic Medical Services). 2) Empresa Social del Estado Hospital Regional 
de Duitama (The State Social Services Regional Hospital Of Duitama) and 3) Santa Rosa de Viterbo Hospital in 
Duitama (Boyacá).  And, we recruited participants in three healthcare systems in Tolima: 4) Hospital Armero-
Guayabal, 5) Hospital de Chaparral, and 6) Hospital Granja de Lérida, Primer Nivel.   
 
Participants 
We identified the following groups of participants for the focus groups and in-depth interviews in order 
to obtain diverse stakeholder input: 
 
● Health professionals (nurses, physicians, psychologists and social workers) were asked about 
the importance of identifying and treating mental health problems (particularly depression and 
alcohol), knowledge they have about the clinical guidelines in Colombia for treatment of 
depression and alcohol use disorders, the possibility and acceptability of working with patients 
with depression or alcohol use disorders, their interest in encouraging patients' active involvement 
in their own care, among other fundamental aspects to improve the quality and integration in 
health care and expected benefits or burden for their practice (time, etc.) if depression and alcohol 
is addressed in a systematic way. 
 
● Administrative workers (e.g., intake staff, practice managers, administrative coordinators, 
technology directors, billing administrators) were asked about technical and administrative 
possibilities of introducing new elements into the care model, such as, mobile applications and 
other information technology solutions in healthcare, forms of financing, coordination with 
national structures such as the Ministry of information and communications technologies to link 
patient data to health insurance providers’ information platforms. Similarly, we inquired about 
what changes may be needed in workflows to improve mental health care. 
 
● Patients were asked about the facilitators and barriers to accessing health services and 
particularly mental health services in primary care, quality and timeliness of care, facilities for 
obtaining specialized mental health care, and proactivity of patients in treatment. 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
18
● Community organizations (e.g., patient associations) represented an important participant group 
because they may become support networks for patients with depression and alcohol use 
disorders. They were asked about their facilitators and barriers for behavioral health promotion, 
and referral to health services and particularly to mental health services in primary care, quality 
and timeliness of care. 
 
Focus Groups Procedure 
We held focus groups with each group of participants in all study sites in Colombia (as described above) 
to achieve a total of 14 focus groups (with 104 participants).  Each focus group convened between 6 to 
10 people.  In Bogotá, we held two additional focus groups, one with patients diagnosed with major 
depression and the other with patients diagnosed with an alcohol use disorder due to the availability of 
patients’ groups.  
 
Focus groups were led by an experienced facilitator (anthropologist) who received training in focus group 
methodology and the specific topics to discuss with the group. The groups also had two additional 
members of the research team present to observe, and take notes during the focus group. Each focus 
group was audio recorded for analysis. Each focus group lasted about 1.5 to 2 hours and each group’s 
meeting was held in private meeting rooms. 
 
In-Depth Interviews  
We conducted 9 semi-structured interviews to enable a more in-depth understanding of the elements 
identified in the focus groups, including 3 patients with depression and/or alcohol use disorder, 3 health 
professionals, and 3 administrative workers. Note that community members were not included in in-depth 
interviews because they were not part of the health care system (as were the other three stakeholder 
groups).  Participants were recruited from focus groups by convenience, availability and willingness to 
participate. Each interview lasted approximately 60 minutes.  Interviews were conducted by 2 
interviewers with training in qualitative methods and experience in conducting research in health topics 
(the same staff who conducted focus groups). Participants provided permission to digitally record the 
interview and, in addition, notes were taken by the interviewer. 
 
These qualitative data from focus groups and in-depth interviews informed our planned model of 
mental health screening and care that we plan to implement in our upcoming implementation research 
project. These data are still being collected but we expect they will inform our model of 
implementation into the workflow at each study site and increase our understanding of other mental 
health resources (if any) available in the communities surrounding our study sites. 
 
6.3. Behavioral Health Integration in Medical Care (BHIMC) assessment: Adaptation for 
Latin America and baseline assessment at primary care study sites 
 
Objective 
One of the aims of this research project is to assess the organizational capacity of a sample of Colombian 
primary health service centers to address behavioral health conditions.   To accomplish this, we 
systematically modified the Behavioral Health Integration in Medical Care (BHIMC) measure (which 
has been validated in the U.S.) for use in the Colombian health care system.  We then completed this 
modified BHIMC measure as a baseline assessment to characterize the extent to which the primary care 
sites that will participate in the implementation trial have embraced screening/behavioral health in their 
service delivery model.   As described in Section 12.0 below, we also plan to administer this tool at these 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
19
study sites over time during the implementation research project to capture change over time (pre/post 
implementation of the novel mental health care model). 
 
The Measure 
The Behavioral Health Integration in Medical Care (BHIMC) index is an organizational measure of the 
level of behavioral health integration in medical practice settings. It evaluates policy, clinical practice 
and workforce dimensions of integration using mixed methods, i.e. combination of document review and 
observation. The structure of the half day site visit is as follows: (1) Meet with leadership (Chief 
Executive Officer, Chief Medical Officer, Chief of Operations, Directors of Behavioral Health and 
Nursing, etc.), (2) Tour the organization, (3) Interview with providers (3-5 in small group), (4) Interview 
with support staff (3-5 in small group), (5) Interview with patients (individually), (6) Review documents 
(4-5 patient records, manuals, brochures, informational material, screening and intake forms, etc.), and 
(7) Debrief.  
 
Data from these sources are used to make ratings on a 36-item 5-point rating scale: 1 = Minimally 
Integrated (MI); 2=  Minimally Integrated/Partially Integrated (MI/PI); 3 = Partially Integrated (PI);  4= 
Partially Integrated/Fully Integrated (PI/FI); 5 = Fully Integrated (FI).  As reflected in this scale, scores 
of 2 or 4 are reflective of intermediary levels between the standards established by scores of 1 (MI), 3 
(PI) and 5 (FI).  The BHIMC has been used in other contexts (by the Dartmouth team) to assess behavioral 
health integration in primary care, and community health clinics.17   It was originally developed with the 
support of the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA). 
 
Adaptation of BHIMC for use in Colombia 
To adapt the BHIMC, we divided our work into four concurrent tasks: 
 
a. Understand the evidence of value of integration of mental health in primary health 
services in low and middle income countries 
 
[Note that we are separately conducting a systematic review on this topic and we 
registered this protocol in the Prospero Database on March 1, 2017 
https://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017057340]. 
 
b. Understand the BHIMC (the original instrument developed and used in the USA) 
 
c. Based on the two above, slightly adapt the existing tools and processes to the 
Colombian context.  This largely included translation to Spanish and changing reference 
to payer systems in the Colombian (and not the U.S.) context. 
 
d. Work jointly with the industry actors (health insurance payers via our industry 
advisory board) to ensure that the adaptation is grounded in the existing reality 
 
Briefly, in understanding the BHIMC, the core team read and analyzed the associated manual about this 
instrument, met with the original Dartmouth-based developer of the instrument, and held multiple 
rounds of discussion to adapt several questions and the gradients of the scale to the Colombian health 
care system. The core team also redesigned the process based on the local norms and values. In so 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
20
doing, the team periodically consulted with various members of our industry advisory board in 
Colombia to ensure the tool best reflects the reality of the Colombian context. 
 
Note that we conducted a systematic search for any similar instruments that had been used in Colombia 
but did not find any (and thus decided to adapt the BHIMC to the Colombian health care context). 
 
The core team then participated in a 3-day, face to face training (held at Pontificia Universidad 
Javeriana in Bogotá) on the use of the BHIMC, led by a team member from Dartmouth who has long-
standing expertise in the use of the BHIMC (Andrea Meier at the Dartmouth Center for Technology 
and Behavioral Health, directed by Dr. Marsch).   Given the interest in the use of this tool in Colombia, 
various members of the Colombian Ministry of Health also participated in this training.  On day 1 of 
the training, the Core team met to review the logistics of the training and then Ms. Meier led a 
presentation on the background of the BHIMC measure, scoring sheets and associated reports.  She also 
provided a practicum using sample documents and case studies.  On day 2 of the training, the team 
conducted a field visit and assessment with the BHIMC at a health care site that agreed to serve as a 
training site (to help ensure the real-world training experience for trainees).  During this field visit, the 
core team piloted the adapted tool and process. On day 3 of the training, the trainees completed 
BHIMC scoring (including associated graphs and summaries) based on the prior day’s field site visit. 
 
We have completed baseline BHIMC data collection at the primary care sites that will collaborate on 
our planned implementation research project. And as noted elsewhere, we plan to administer this tool at 
these study sites over time (every year) during the implementation research project to capture change 
over time (pre/post implementation of the novel mental health care model). 
7.0 
STUDY DESIGN  
7.1 
Summary Overview of Implementation Study Design  
In the proposed implementation research project, we plan to launch and evaluate a new mental health 
service delivery model in Latin America.  Specifically, in this new mental health service delivery 
model, we propose to: (1) harness mobile behavioral health technology for mental health (with a 
primary focus on depression and a secondary focus on problematic alcohol use), (2) launch new 
workforce training and service delivery models (including the integration of technology into service 
delivery), (3) launch and evolve an integrated data management system for systematic data tracking and 
outcomes assessment, and (4) launch and grow a learning collaborative of organizations integrating 
mental health into primary care.  We will launch this project at multiple primary care sites in various 
parts of Colombia, with a plan to inform subsequent adoption in several other Latin American 
countries, including Chile and Peru. 
 
As detailed below, we will initially pilot test this service delivery model at a single site (starting in the 
late summer of 2017, which is year 2 of the project) and then refine the model based on pilot data. We 
will then expand implementation across multiple Colombia-based healthcare sites in urban and rural 
communities on a staggered basis (Years Two-Five).  Consistent with a modified stepped wedge design 
(multiple baseline design),18, 19 we will implement across sites on a staggered basis and expand the 
number of sites in which we implement over time.  By conducting this multi-site implementation 
research project, we can assess the extent to which the implementation model and associated outcomes 
are replicable across sites and/or the extent to which the model needs to be modified for differing 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
21
contexts. 
 
We will evaluate (Years Two-Five) the ability of our proposed approach to accelerate the translation of 
evidence-based mental health services into practice and expand research capacity at multiple levels.   
Specifically, we will measure implementation outcomes, including: (1) the ability of this model to 
accelerate the adoption of science-based mental health service delivery (e.g., increase provision of 
evidence-based resources to more individuals), (2) the acceptability of the model for healthcare service 
delivery (the ability of the technology-based service-delivery model to increase patient activation and 
engagement in their own self-management), and (3) the cost-evaluation of services in the model.  We 
will also collect patient level data outcomes (depression, quality of life, alcohol use) as a secondary 
focus to assess (4) the model’s impact on public/population health (e.g., its effects on accelerating 
improved behavioral health and health outcomes, and improving patient quality of life, functioning, 
etc.).  Patient level data will be collected via computerized assessment and will be deidentified (not 
linked to patient name, date of birth or other identifying information). 
 
We will also measure implementation context factors to evaluate barriers and facilitators to 
implementation in each of the following domains: intervention characteristics, organization 
characteristics (e.g., climate, readiness), individual characteristics (i.e., provider/staff attitudes, 
experiences; patient attitudes and experiences), and external influences (e.g., socio-political 
characteristics local technology infrastructure i.e., wireless in the community, state policy/regulations, 
and reimbursement models).  Strategies related to planning, facilitating provider and patient 
engagement in use of the novel mental health service delivery model, and potential sustainability issues 
will also be explored.  We will additionally conduct exploratory moderator/mediator analyses to 
examine how implementation context variables relate to implementation outcomes.   
 
We will first describe in more detail the various components of the mental health care model to be 
implemented followed by a detailed description of the proposed implementation research project 
protocol and all planned assessments. 
8.0 
COMPONENTS OF MENTAL HEALTH CARE MODEL TO BE IMPLEMENTED 
In this section, we first describe each aspect of the mental health service delivery model, followed by 
our implementation research plan. 
 
(1) Mobile behavioral health technologies for mental health.  As reviewed above -- given the ubiquity 
of access to technology worldwide -- patient monitoring and behavior change tools delivered on mobile 
platforms may enable widespread reach and scalability of evidence-based mental health interventions.  
A growing body of scientific research has shown that mobile behavioral health tools can have a large 
impact on health behavior and health outcomes, and increase quality, reach, and personalization of care 
in a manner that is cost-effective.  Mobile behavioral health tools can produce health outcomes that are 
comparable to, or better than, outcomes produced by care delivered by highly trained clinicians.  Also, 
by having on-demand access to “just in time” therapeutic support via electronic devices, individuals 
can prevent costly escalation of problems related to behavioral health and unnecessary healthcare 
utilization.20-28 Technology provides new opportunities for self-care in response to the needs of each 
individual.   
 
Technology-based interventions can be available 24/7 and thus allow for “on-demand,” ubiquitous 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
22
access to therapeutic support, thereby creating unprecedented models of intervention delivery and 
reducing barriers to accessing care.  This provides the possibility of addressing the “5,000 hour 
problem” -- namely that patients typically only spend a few hours a year in front of a doctor and in the 
5,000+ other waking hours, their doctor neither knows what they are doing nor has effective tools to 
support them in the management of their health care.29 Leveraging technology in this way offers great 
promise for leading to new models for science-based approaches for promoting behavioral health and 
health outcomes.30-33 
 
Mobile behavioral health technology represents a key part of our planned mental health service 
delivery model for scaling-up access to evidence-based mental health care in Latin America.  A 
key component of this system is a novel, mobile-based platform (Laddr® from Square2 Systems) that 
offers science-based self-regulation monitoring and health behavior change tools via an integrated 
platform to a wide array of populations.  Laddr embraces the commonalities in the principles of 
effective behavior change and transcends the currently siloed nature of mobile health tools.  This 
includes tools for activating behavior change, solving problems and overcoming obstacles to effective 
behavior change, teaching skills and providing guidance on the execution of behavior change, and 
maintaining the end user’s motivation to change.  Laddr enables frequent, longitudinal assessment of 
patient-reported outcomes in naturalistic contexts, offers science-based self-regulation behavior change 
tools of relevance to an array of populations, and enables ongoing monitoring of health behavior. The 
uniqueness of Laddr lies in the fact that it integrates tools that have been developed via an iterative 
patient-centered approach and shown (in over a dozen NIH-supported studies conducted by Multiple 
Principal Investigator Dr. Marsch and colleagues) to be highly effective for a wide array of behavioral 
phenomena ranging from, for example, substance misuse, alcohol misuse, mental health, risk-taking, 
chronic pain management, medication adherence, diet, exercise, diabetes management, and smoking.20-
28   Laddr is available on multiple platforms (including desktop, Android, iPad, and tablets).  To our 
knowledge, this is the only mobile ecosystem that employs the fundamental principles behind the 
science of behavior change to flexibly apply to a broad array of populations based on their goals, needs, 
and preferences and independent of their “disease” or “disorder”.   
 
Laddr will be offered to focus primarily on depression management – the most significant mental 
health problem experienced in Latin America.  Specifically, the core functionality of Laddr (e.g., 
problem-solving therapy) will be structured to focus on an end user’s management of depression and its 
impact on their functioning and quality of life.   The program will secondarily focus on problematic 
alcohol use and its relationship to depression management.    
 
Consistent with the science of behavior change described above, these key strategies targeting depression 
management (and secondarily alcohol abuse and its relation to depression) focus on increasing an individual’s 
personal and social resources that support and reinforce healthy, goal-directed behavior and reduce self-defeating 
behavior, including tools and resources to:  
 
1. Help activate/motivate behavior change based on an individual’s values (a scientific process known as 
“Behavioral Activation” [BA]).  In this process, individuals are provided with tools and strategies to help 
identify their values, in areas such as health, parenting, family relations, social relations, work/career, leisure, 
and/or personal growth.  BA helps individuals take steps to create an environment that supports healthy and goal-
directed behavior consistent with their values.   It also includes systematic tracking of behavior and 
consequences of behavior to help identify and disrupt (with learned skills) self-defeating behavioral patterns.  
For example, this may include disrupting negative thought-patterns by identifying and understanding triggers of 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
23
negative thought-patterns and self-defeating responses to such triggers and then replacing self-defeating 
responses to triggers with alternative, coping strategies.  
 
2. Effectively solve problems needed for successful behavior change (a scientific process known as 
“Problem Solving Therapy” [PST]).34-36  PST is a practical and easily learned intervention for many self-
defeating behaviors, including depression.  The goal of PST is to teach individuals skills in solving problems as a 
means of enabling them to self-manage and to control negative states and behavior.  Its treatment process focuses 
on participants’ appraisal of specific problems, their identification of the best possible solutions, and the practical 
implementation of those solutions, as well as increasing exposure to pleasant37, 38 events.  
 
3. Teach skills and provide guidance in the execution of behavior change (scientific processes known as 
“Cognitive Behavioral Therapy” [CBT]39, 40 and the “Community Reinforcement Approach” [CRA]41 to 
behavior change).  CBT teaches a broad array of skills and behaviors to manage problematic emotions, 
behaviors, and cognitive processes with the goal of helping individuals reduce self-defeating behaviors (such as 
problematic substance use or negative thinking patterns) and increase and maintain health behaviors.   Examples 
include managing negative thinking, identifying and altering cognitive distortions, communication skills, 
decision-making skills, stress management, and time management.  CRA is an extension of CBT, designed to 
help individuals establish and maintain new healthy patterns of behavior and leverage social, recreational, family 
and vocational reinforcers to help them in making positive behavior change.    
 
4. Offer support (or “reinforcement”) of successes in behavior change from an online support network of 
the user’s choosing (e.g., friends, family) (another key component of CRA).42, 43  
 
Based on these scientific approaches, Laddr initially is structured as a “process loop” that includes a 
“discovery” process to help end users identify values/goals and motivate progress toward goals, such 
as diet, activity level, smoking and/or recreational goals (BA), an “action” process to track (in real 
time) progress toward goals, functioning, chains of health behavior/medical regimen adherence (or 
lapses in health behavior), a “remediation” process to access any of over 70 “guides” (delivering 
interactive CBT/CRA) based on end user needs/preferences, and “reinforcement” functionality to 
reinforce progress toward goals and health behavior.44, 45  
 
Evidence-Base. The efficacy of this approach to treat depression has been supported in numerous 
randomized controlled trials, including studies conducted in primary care and medically ill populations.  
Meta-analyses have shown consistently strong effect sizes of this approach in increasing depression-
free days by as high as 50%.  This approach also markedly reduces depression symptoms and improves 
functioning.  This work has also demonstrated that this approach to depression care creates a level of 
therapeutic affiliation (an essential component of psychotherapy) equivalent to live therapy. This 
approach also has been shown to be cost-effective by decreasing medical care expenditures and 
increasing work productivity.37, 46, 47 
 
The functionality of Laddr has been shown to be as effective as science-based, behavioral therapy 
delivered by highly trained therapists in promoting objectively measured abstinence from substances of 
abuse.48  It has also been shown to double abstinence rates compared to standard substance abuse 
treatment delivered in the U.S. (which may not always be science-based).49, 50  This program has been 
shown to be more effective than traditional treatment models for even the most challenging of cases– 
including persons who have a history of chronic relapse to substance use, persons with high 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
24
ambivalence about changing their problematic substance use, and persons with high anxiety.51, 52  The 
effectiveness of this system has been shown with a wide array of persons with problematic substance 
use, including use of alcohol, opioids, cocaine/other stimulants, marijuana, and poly-substance use.50   
 
Note that the applicant team is in the process of creating a version of Laddr in Spanish to function as 
described above for Latin American communities.  Although we will conduct pilot testing with a wide 
array of stakeholders in the proposed project (described below) in which we will evaluate and refine 
this system before and after it is deployed, we did not seek funds in the budget of this project – which is 
focused on implementation science and capacity building-- to complete our Latin American version of 
this mobile ecosystem. 
 
(2) Workforce Training in new Service Delivery Models.  Primary care is the main source of 
healthcare for most persons in Colombia and elsewhere in Latin America, including for the poorest 
citizens.  Turnover rates of primary care physicians is high, with many physicians remaining in their 
position at a given program for less than 2 years.  However, nurses and social workers tend to remain in 
their positions much longer.  Although mental health care may be available via specialty referral, wait 
lists for mental health specialty programs often exceed 2 months.53 
 
Our team (under the leadership of Multiple PI Gómez and Consultant Uribe) has extensive experience 
conducting mental health training among healthcare providers in Latin America.54-57  Our workforce 
training plan will be based on these prior successes as well as similar models that have been shown to 
be successful in training the primary care workforce in Latin America.53, 58, 59 
 
Specifically, the training will be available to all levels of providers in our partnering primary care sites, 
including one training for nurses, nurse assistants, social workers and health promoters and a second 
training for general physicians and chief nurses. Trainings for the first group will entail a 16 hour, in-
person mental health training, followed by access to virtual training support.  Trainings for the second 
group of physicians and chief nurses will be shorter, given their time constraints and given the plan for 
the first group of healthcare workers (who remain in their roles in primary care longer) to largely 
assume responsibility for screening/embedding depression care into primary care.  The training will 
focus on mental health disorders in primary care and screening strategies for detecting mental health 
problems – with a primary focus on integrating depression screening and care into primary care.   A 
secondary focus will be on alcohol and substance use and its relation to depression.  This work will be 
based on clinical practice guidelines developed by Dr. Gómez and Dr. Uribe and their team at the 
Pontificia Universidad Javeriana and refined in collaboration with the present applicant team as part of 
the formative work the team has conducted over the past several years.  Typical clinical case histories 
will be used as examples for training and learning evaluation.  Consistent with models Dr. Gómez and 
his team have previously used, the virtual training component will consist of a set of computerized 
education modules with built-in quizzes to assess learning (offered on a platform at Pontificia 
Universidad Javeriana). Dr. Gómez will take the lead on creating the content for these online 
educational modules, along with his course development team at Pontificia Universidad Javeriana, as 
he has done in the many prior courses created by his group. 
 
Trainings for the first group include 7 main modules on topics of: (1) Introduction and conceptual bases 
to mental health; (2) How mental health care works within primary care and within the broader 
healthcare system; (3) Community processes in primary care; (4) Network identification and 
characterization for community action; (5) Support tools and methods for screening and defining care 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
25
pathways; (6) Support interventions -- Emotional “first aid” including brief and group interventions; 
and (7) Community based mental health rehabilitation.  The abbreviated training for the second group 
includes 5 main modules on topics of: (1) Introduction and conceptual bases to mental health; (2) 
Community Processes in Primary Care; (3) Interviewing patients with mental health problems; (4) 
Conducting a clinical history for mental health; and (5) Early detection, diagnosis and management of 
mental health disorders. 
 
Note that we will also train this clinical workforce at each partnering primary care site on the Laddr 
mobile platform.   While Laddr has been developed to be intuitive and self-driven, we will ensure that 
at least one person at each site can provide first-line technical assistance as needed (also with the help 
of the technical assistance staff person at Pontificia Universidad Javeriana who will provide support to 
all partnering primary care sites and serve as a valuable resource for consultation (e.g., if challenges 
with Internet connectivity arise). 
 
Trainings will be conducted prior to implementation of the mental health service delivery model at each 
of our partnering primary care sites (see chart of Timeline/Milestones below).  We anticipate training 
approximately 5-25 nurses, nurse assistants, social workers and health promoters at each site and 
training about 3-10 general physicians and chief nurses at each site. [Note the wide range in the number 
of trainees per site reflects the varying sizes of study sites across urban and rural settings].  Dr. Gómez 
(an expert in depression care) and a psychiatrist on this project at Pontificia Universidad Javeriana will 
lead the trainings. Each individual who completes the training will receive a certificate (diploma) for 
this course from Pontificia Universidad Javeriana.  Dr. Gómez has similarly issued certificates for 
many prior workforce training programs he has led. 
 
(3) Integrated Data Management System.  We will create an integrated data management system for 
this project for systematic data tracking and outcomes assessment.  Participants’ completion of study 
assessments will be completed via an encrypted Internet connection via 128-bit Secure Sockets Layer 
(SLL) -  the standard technology for securing eCommerce and eBanking transactions on the Internet.  
We plan for all these project data to be coordinated through REDCap (Research Electronic Data 
Capture),60 which we intend to set-up at our partnering university in Colombia, Pontificia Universidad 
Javeriana.   The REDCap Consortium is composed of 1,520 active institutional partners in 92 countries 
who utilize and support REDCap in various ways.   Both Dartmouth and Javeriana will have access to 
REDCap and the data collected in this project (via web-based access).  We will set-up data 
monitoring/sharing access privilege to ensure the integrity of data collection/management (under the 
leadership of Senior Data Manager Mary Ann Greene at Dartmouth).   
 
We will seek Institutional Review Board (IRB) approval at Dartmouth as well as from Pontificia 
Universidad Javeriana and any partnering health care systems, as required.  Any reportable events will 
be reported to both Dartmouth’s and Javeriana’s IRBs.  Our data management expert at Dartmouth 
(Mary Ann Greene) will serve as an expert advisor on data management systems broadly and REDCap 
specifically. 
 
(4) Learning Collaborative. As primary care programs join the implementation project and launch the 
proposed novel mental health service delivery model as part of their model of care, they will be able to 
join a learning collaborative of all sites implementing this model.  This learning community will meet 
quarterly by webinar and share lessons learned – including successes and challenges in implementation.  
These meetings will be in addition to the once per year in-person meeting in which these individuals 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
26
will participate.  Data collected at each site will be aggregated by site (by the Dartmouth team) for 
review and discussion at this quarterly meeting.  Quarterly meetings of this learning community will be 
co-led by Multiple PI Gómez and Consultant Uribe.  Learning collaboratives have been shown to be 
highly effective in promoting shared learning among a broad community.61  
9.0 
STUDY POPULATION FOR IMPLEMENTATION STUDY 
9.1 
Provider and administrative staff Inclusion Criteria:  
• 
Aged > 18 years  
• 
Have worked for the study site for at least 3 months. 
9.2 
Participant Inclusion Criteria: 
• 
Aged > 18 years 
• 
Patients at one of our collaborating primary care sites  
• 
Screen positive for minor (score of 5-9), moderate (score of 10-14), moderately severe (score of 
15-19) or severe (score of 20-27) depression on Patient Health Questionnaire (PHQ-9)62 and/or 
screen positive for problematic alcohol use on Alcohol Use Disorder Identification Test 
(AUDIT) developed by the World Health Organization (a score of 8 or more on 10-item 
AUDIT)63 and have a confirmed diagnosis of depression and/or alcohol use disorder based on 
clinical consultation at the primary care site 
• 
Willing to provide informed consent to use mobile intervention and complete study assessments 
9.3 
Participant Exclusion Criteria  
• 
Diagnosis with co-occurring severe mental illness (e.g., schizophrenia, bipolar disorder, 
depression with psychotic features) 
• 
Alcohol withdrawal symptoms that require higher level of care (e.g., emergency medical or 
inpatient treatment) 
• 
Express suicidal intention.  This assessment will be based on a combination of a patient’s response 
on the PHQ-9 assessment followed by further assessment by a primary care clinician.  The PHQ-
9 measure will not be used solely to determine eligibility on this criterion.  Persons who express 
suicidal intention will immediately be provided immediate crisis management in accordance with 
the crisis management protocol at the collaborating primary care site.  
• 
Intoxicated or otherwise impaired at the time of assessment (rendering them incapable of informed 
consent) 
 
 
No Sex/Gender or Racial/Ethnic groups will be excluded.   Based on patient data at our partner sites, 
we expect that approximately 65% of the participants will be women.   And, we expect that 
approximately 60% will be white, 10% will be black, and 30% will be a mixture of other racial groups.   
 
Note that individuals will not be excluded if they do not have access to smartphones. Although the 
mobile therapeutic tool that will be offered to participants as part of this project is not accessible on 
feature phones (non-smartphones), it is web-based and can be accessed on tablets or other types of 
computers.   Although patients are likely to most benefit from mobile access to the intervention (for 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
27
anytime/anywhere access), each collaborating study site will set-up computers at their site to allow 
patients to access the mobile intervention onsite if they so choose. 
9.4 
Participant Recruitment, Screening and Informed Consent 
Administrative staff at the partnering primary care site who “check-in” patients upon arrival will ask 
each adult to complete the screeners (for depression and problematic alcohol use) at the time of check-
in.   Screening for depression and problematic alcohol use will become standard practice at all 
collaborating sites.  So, all patients at the site will be screened regardless of their participation in the 
study.  Screeners will be administered via paper or tablet (according to patient preference and site 
capability) in the waiting room.  Note that patients who are already actively participating in the study 
will not be asked to complete this screener at check-in. 
 
The screener for depression will start with the two questions on the Whooley assessment 64: (1) “During 
the last 30 days, have you been bothered by feeling down, depressed or hopeless (YES/NO)?”; (2) 
“During the last 30 days, have you been bothered by little interest or pleasure in doing things 
(YES/NO)?”  A positive response on either or both questions will then lead to a Patient Health 
Questionnaire (PHQ-9)62 screener for depression.   
 
The brief screener for alcohol, the Alcohol Use Disorder Identification Test Alcohol Consumption Questions 
(AUDIT-C)65, consists of 3 questions: (1) “How often do you have a drink containing alcohol?” (Never; Monthly 
or less; 2-4 times a month; 2-3 times a week; 4 or more times a week); (2) “How many standard drinks 
containing alcohol do you have on a typical day?” (1 or 2; 3 or 4; 5 or 6; 7 to 9; 10 or more); and (3) “How often 
do you have six or more drinks on one occasion?” (Never; Less than monthly; Monthly; Weekly; Daily or almost 
daily). The AUDIT-C is scored on a scale of 0-12. In men, a score of 4 or more is considered positive, and in 
women a score of 3 or more is considered positive, which can assist with identifying problematic alcohol use. 
Persons who answer positively to the AUDIT-C questionnaire will then be asked to complete the full Alcohol 
Use Disorder Identification Test (AUDIT)63 to screen for alcohol use problems. 
 
Screening results will be provided to the primary care clinician (by the administrative staff or clinical 
intake worker) either prior to, or at the time when, the patient enters an exam room to see the primary 
care clinician.  Given the state of the electronic health records at participating sites (e.g., some do not 
have them), we do not envision that tablet results will be directly entered into the electronic health 
record (although this may evolve over the life of the project). The administrative staff worker or clinical 
intake worker will score the results on each screener before the patient sees the primary care clinician 
(unless scoring is automatically done on a computerized screener).   Both the responses to individual 
screener questions as well as the scored results will be provided to the clinician (including designations 
of mild, moderate, moderately severe and severe).   A suicide risk assessment will be conducted by the 
clinical staff if item 9 of the PHQ-9 is positively answered or if deemed otherwise clinically important 
by the clinician during examination.  Each site has an internal protocol for evaluating and managing 
suicide risk as identified on the PHQ-9 or in discussions between a patient and clinician. Specifically, 
in this process, each site follows Colombian guidelines in which a clinician asks the following 
questions about suicidal ideation, plans, gestures, and attempts, or behaviors of self-harm (to identify 
any patient needing immediate psychiatric or emergency referral): 
1.       Do you feel worth living? 
2.       Do you want to be dead? 
3.       Have you ever thought about ending your life? 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
28
4.       If so, have you thought about how you would do it? What method would you use? 
5.       Do you have access to a way to carry out your plan?* 
6.       What prevents you from getting hurt? Do you feel worth living? 
In the event of a positive screen for depression and/or alcohol use problems, the clinician at the primary 
care site will complete a more in-depth diagnostic interview to better understand the data provided by 
patients on the screeners.   Participants who screen positively for depression (score of > 5) and/or 
alcohol use problems (score of > 8) and meet all other inclusionary criteria will be informed about the 
treatment options available to them under this project in which the site is participating.  As part of this 
process, the primary care clinician (or his/her designee at the primary care site) will inform patients 
about the goals of the project, the types of services they would be offered as part of the project and 
what patients would be asked to do if they elect to participate (e.g., use digital therapeutic tool for 
supporting mental health; complete period assessments over time). A research staff member or a 
clinician at the study site will provide participants with an informed consent form to participate in the 
study.  We intend to ask participants to complete consent forms electronically within REDCap.  
Participants will also receive a blank paper copy of the consent form for their reference (and for 
research contact information on the form). This process is consistent with requirements of Dartmouth’s 
IRB.  Patients who provide informed consent will then be asked to complete baseline participant 
assessments (described in section 12.0 of this protocol).   Details of the Informed Consent process are 
included in Section 15.4.  The research team will only track outcomes from patients who provide a 
signed study consent form. 
 
Note that patients who elect to join the study will be advised in the consent form that the research team 
would like to be able to access their screening data collected in their medical record before study 
consent (for depression and problematic alcohol use) for inclusion in the research data.  We will also 
obtain data on patients’ attendance records at the primary care study site. 
 
All patients who join the study will be assigned a unique study identification number (which will be 
linked to all patient data collected from a given participant).  The percent of patients who are eligible to 
participate but who decline participation will be tracked, along with patients’ reported reasons for non-
participation. These data will be collected at each collaborating primary care site but will be tracked 
centrally across all sites. These data will help the project team better understand the acceptability of the 
model of care being offered in this project and help the team consider modifications to the 
implementation model to improve the model of care, as needed. 
 
At the end of their study period, participants will be given in person (or emailed if the participant has 
already completed their follow-up visit and we are unable to give them their letter in person) a close-out 
letter that thanks them for their participation in the study, encourages them to continue seeking 
treatment for their medical condition, and provides them with a link to the DIADA project where they 
eventually will be able to view aggregate study results. These letters will be stored on a password-
protected study computer and in the study’s regulatory binder, which is stored in a locked filing 
cabinet.  
 
To mitigate any potential increased risk of loss of confidentiality, we will 1) only email letters if we are 
unable to give participants their letter in person 2) email instead of mail these letters to reduce the risk 
that anyone other than the participant sees the letter 3) ensure that the email address to which we send 
these letters is personal to the participant, rather than a shared email.  
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
29
Patients who decline to participate but who may have severe depression or suicide risk will be treated 
in accordance with each clinical site’s suicide risk management policies.  The patient’s primary care 
physician will be informed about the results of the screening and the patient will be managed according 
to the primary care site’s usual protocols and guidelines. 
 
Consistent with the goal of this project, patients will be offered a multi-component model of science-
based mental health care.  All participants who consent to participate will be given access to the mobile 
therapeutic tool. A trained staff member at the primary care site will show them how to use the tool.   
As noted above, participants will be offered the option to access the tool on their smartphones.   
Additionally, or as an alternative to smartphone access, patients will be offered the opportunity to 
access this web-based resource at one of several computers that will be set-up at the primary care site 
(likely 2-4 computers per site).  Note we have included WiFi/hotspot/router set-up costs in the study 
budget for select sites (e.g., rural), as needed. 
 
All participants will be encouraged to access community-based resources for depression and/or 
problematic alcohol use, when available.  For example, participants may be informed of local 
Alcoholics Anonymous meetings or other substance use/mental health support groups available in their 
communities.  Participants may also be referred to speciality mental health care (e.g., psychologist or 
other mental health worker); however, we expect few participants will engage in such care due to its 
very limited availability in most settings in Colombia. 
 
Participants diagnosed with depression (particularly moderate to severe depression) may also be offered 
access to anti-depressant medications, as determined in consultation with the primary care physician. 
Medication prescribing for depression will be conducted in accordance with the Colombian clinical 
guidelines of best practices.    Medications for treatment of alcohol use disorders are not currently 
available in Colombia.  If they become available during the course of this project, we will modify the 
protocol, as appropriate, to embrace evidence-based medications for the treatment of alcohol use 
disorders. 
 
If a patient meets criteria for severe depression, or has significant suicide risk, treatment may be either 
done by, or coordinated with, a psychiatrist (in accordance with the Colombian clinical guidelines of 
best practices). 
 
We plan to develop a printout of a decision aid tool for clinicians to use, designed as a matrix to help 
patients understand the various options available to them as part of their care (including the advantages 
and disadvantage of the various options).  We may also develop a video or other educational resources 
to help patients understand how the various care options available to them work to support a process of 
shared decision-making. 
 
 
The flow of participant activity is summarized in the accompanying figure. 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
30
 
 
Note that as detailed in the section 15.4 on Informed Consent, both provider and administrative staff at 
partnering primary care sites will all be offered the opportunity to complete the implementation context 
and outcome measure in this research project (including nurses, nursing assistants, social workers, 
health promoters, physicians and charge nurses, program administrators).  All participating staff will 
also be asked to provide consent to participate. Provider and administrative staff all need to be aged > 
18 years and have worked for the study site for at least 9 months. Research staff will provide informed 
consent to each staff member.  We expect approximately 4-8 provider staff per site will complete the 
provider measure of implementation context and outcomes and about 2-5 administrator staff per site 
will complete the organizational level measure of implementation context and outcomes at each 
timepoint. 
10.0 
SITE SELECTION 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
31
10.1 
Number of Sites  
We will collaborate with 6 primary care networks spanning diverse rural and urban locations across three 
states in Colombia: (3) Cundinamarca (Javesalud, Soacha, Guasca), (2) Boyacá (Santa Rosa, Tundama) 
and (1) Tolima (Armero-Guayabal). [See Colombia map below].  None of these primary care systems 
routinely screen for/treat depression or alcohol use disorders within their primary care programs.  The 
Bogotá site (Javesalud) will serve as the pilot study site and will remain a study site thereafter. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10.2 
Site Characteristics  
Fundación Javeriana de Servicios Médicos ondontológicos inter-universitarios (The Javeriana 
Foundation of Inter-University Odontologic Medical Services) “Carlos Marquez Villegas” Javesalud 
Javesalud is a Colombian ambulatory health center that focuses on patient-centered primary care within 
a family medicine framework (working with children, adolescents, adults and pregnant women). 
Javesalud has 7 outpatient primary care program servicing over 95,000 individuals in Bogotá, 
Colombia. 
 
Empresa Social del Estado Hospital Regional de Duitama - Santa Rosa de Viterbo Branch  
Santa Rosa de Viterbo Regional Branch offers primary care ambulatory services (outpatient, emergency 
care, ambulance transportation) covering a population of 14,000 persons, 47% of whom live in rural 
areas.  It also provides general medical and psychological services. 
 
Salud de Tundama 
Bogotá 
Tundama 
Santa Rosa 
Guasca 
Armero-Guayabal 
 
Soacha 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
32
Salud de Tundama is a health organization that offers primary care outpatient services, self-care 
promotion and risk management for adult and child populations in Duitama and in the surrounding 
area. It provides low complexity general medical services such as pre-conception education, promotion 
and prevention programs, basic laboratory tests, and vaccinations.  
 
Empresa Social del Estado San Antonio Guativa 
The Guasca hospital is a public healthcare center that is affiliated with Hospital San Antonio in 
Guatavita. The hospital has about 4 general practitioners and serves around 110 patients daily. Guasca 
is a town located in Cundinamarca, about 2 hours from Bogotá DC, with about 15,000 inhabitants; 60% 
of them live in semi-urban areas. 
 
Hospital Armero-Guayabal (Empresa Social del Estado Hospital Nelson Restrepo Martinez)  
Armero-Guayabal Hospital provides primary care and ambulatory, emergency and short stay 
hospitalization services covering the population for the municipality of Armero-Guayabal and its rural 
counties.  
 
Empresa Social del Estado Hospital San Juan Bautista 
The Soacha hospital (the Hospital Mario Gaitán Yanguas) is a public healthcare center that is affiliated 
with the Health Secretary from the Department of Bogotá. The Soacha hospital has 10 general 
practitioners and serves about 300 patients daily. Soacha is a town located in Cundinamarca, about one 
hour from Bogotá DC. Soacha is part of the metropolitan area of Bogotá DC, with about 556,268 
inhabitants. 99% of the population lives in urban areas.  
10.3 
Rationale for Site Selection 
 
These sites were selected to add diversity of primary care sites and participants across multiple states and 
regions of Colombia.  These sites span urban and rural contexts (including farming communities). They 
also provide diversity in access to/usage of mobile technology.  All of these parts of Colombia are 
burdened by limited access to mental health screening and resource delivery and thus all of these 
communities may benefit from the project.  Including these diverse sites enhances the generalizability of 
study findings. 
 
Additionally, as noted above, Javeriana has an academic partnership with Javesalud (which is only about 
30 minutes from Javeriana). Javeriana also has an academic partnership with Empresa Social del Estado 
Hospital Regional de Duitama - Santa Rosa de Viterbo Branch.   Javeriana does not have pre-existing 
academic partnerships with the other sites but has already established relationships and confirmed 
commitments of collaboration from the leadership of those sites. Javeriana also has multiple interns that 
complete some of their training at these sites. 
 
10.4 
Wi-Fi Expansion to Rural Sites  
The government of Colombia has recently launched a plan to increase the availability and use of 
information technologies across the country in a plan called “Vive Digital 2014-2018.”66  In this plan, 
Colombia aspires to become a leader in the use of social apps for the poorest, and its goal is to become 
the first country in Latin America with wide-reaching high speed Internet. In 2014, 1,078 municipalities 
in the country were connected through the high speed fiber optic Internet.  The plan also has a goal of 
achieving 4G WiFi in 1000 zones across the country and increasing by 4-fold the number of persons with 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
33
mobile Internet by 2018.  Relatedly, in the health sector, the implementation of a unified unique electronic 
health record is also a national priority.  However, we will help enable Internet connectivity at partner 
sites as needed (e.g., rural locations) and have budgeted resources in our NIMH study to support this.   
11.0 
STUDY PROCEDURES  
11.1 
Pilot Implementation Study 
We will launch a pilot implementation project at a single site (Javesalud primary care site).  We aim to 
launch this pilot project in the summer of 2017. 
 
In this process, we will train staff at the site in accordance with the workforce training plan described 
above, set-up the data management system described above, and include up to 50 patients at the site 
who meet diagnostic criteria for depression who have access to this new mental health service delivery 
model (including Laddr access).   
 
As part of the workforce training (described above), clinicians will be trained on the standardized 
screeners of depression and alcohol use.   As reviewed above, study participants who are identified as 
meeting criteria for depression and/or problematic alcohol use will meet with a mental health-trained 
clinician at the primary care site and offered access to the mental health resources described above. 
 
The outcomes to be measured in this pilot include the original measures and assessment timepoints 
detailed in version 8.0 of the protocol (e.g. they include the Alcohol TLFB instead of the QDS and 
include the ED-5D).  As the central focus of this pilot is to identify any challenges in implementation, 
we will use information that we learn from the pilot to modify our assessments and assessment timeline 
for the full implementation trial. This pilot project will allow us to refine the mental health service 
delivery model as well as the content of the Laddr platform as needed before expanding implementation 
within Javesalud as well as at other sites. However, in order to maintain continuity of data within the 
pilot, we will contain to measure follow-up assessments for the pilot using the measurements and 
assessment timeline detailed in version 8.0 of the protocol.  This will allow us to compare data 
collected from participants in the pilot study across multiple timepoints. 
11.2 
 Implementation Research 
At the completion of the pilot phase, we will expand our implementation across multiple Colombia-
based primary care sites in urban and rural communities (using the same procedures described for the 
pilot above).    
 
Consistent with a modified stepped wedge design (multiple baseline design),18, 19 we will implement 
across sites on a staggered basis and expand the number of sites in which we implement over time.  
Specifically, all sites will complete core organizational measures (described below) every 6 months to a 
year depending on the measure.  And then we will launch at our second primary care site in the fall of 
2018; Site 3 will launch in the spring of 2019; Site 4 will launch in the fall of 2019 and the final 2 sites 
will launch in the spring of 2020.   We plan for about a six month time window between the launch of 
implementation activities across sites. 
 
We have engaged key leadership at all participating sites from the initial project kick-off meeting we 
held in Bogotá, Colombia in September 2016.  And, we continue to engage with them regularly as part 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
34
of our formative research activities.  This ongoing dialogue about project activities will help expedite 
the launch of implementation research at each of our collaborating sites. 
 
Using the primary care site as the unit of analysis,67 the single-subject approach offered via the stepped-
wedge design will enable the repeated measurement of outcomes at the organizational level, including 
aggregated effects on clinicians, patients, and practices.  In this single-subject, stepped-wedge design, 
each site acts as its own pretest control (from baseline data collection).  During the data collection 
period, core organizational indicators (as described in Section 12.0) will be assessed every 6 months.  
 
Note that we considered conducting an experimental randomized controlled trial, but based on the goals 
of this implementation science and capacity building research project, we elected to use this design to 
balance rigor in the design with creating real-world sustainable models to scale-up mental health 
service delivery in Latin America.    
 
This stepped-wedge design offers advantages compared to randomized controlled trials.68 This design 
costs less than randomizing a large sample of organizations required to run the inferential statistics 
associated with randomized controlled trials (or cluster randomized trials).  It makes it easier to 
understand the dynamics of change processes (such as implementation) than randomized, controlled 
trials do.  It avoids the ethical problem in randomized, controlled trials of withholding interventions 
from a proportion of participants who might benefit.   Further, given the project’s goal of scaling-up 
and sustaining evidence-based mental health care in Latin America, providing the novel intervention to 
as many sites as possible is ideal (which would not have been possible in a randomized, controlled 
trial). Additionally, a stepped wedge design allows us to train study site staff in mental health screening 
and care delivery, which enhances the potential for sustainability of this care model. 
 
The single-subject approach emphasizes recurrently tracking behaviors and practices in a “repeated 
time series”69, 70 that permits the study of change patterns. The stepped-wedge designs that often 
accompany single-subject studies provide the intervention to all participants, but isolate the 
independent variable by introducing the intervention at specified points.71 
 
As described in the Capacity Building core of this application, in Year 5, we will conduct trainings to 
launch implementation efforts in the countries of Chile and Peru (at least 2 sites per country).  These 
additional countries will have participated in the learning collaborative from years 1-5.  We will 
additionally evaluate the ongoing sustainability and/or expansion of the novel service delivery model 
both within Colombia and elsewhere in Latin America.  Given that numerous stakeholders in the region 
will be participating in the evolution of this work from its inception and throughout the entire process 
(including payers who may embrace this approach within their business model), and given how this 
work may allow Latin America to help realize their goals and stated policies on scaling up mental 
health care within primary care sites, we may observe growth of this model over time beyond what is 
explicitly planned in this project. 
 
The timeline below shows the planned staggered implementation across collaborating primary care 
sites. 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
35
 
11.3 
Participant Reimbursement  
Participants in the implementation trial will not be compensated for their participation with the exception 
of the small group of participants who agree to participate in qualitative interview data collection (who 
will receive a gift card equivalent to $10 USD after each interview). 
12.0 
STUDY ASSESSMENTS  
In accordance with our planned stepped-wedge (multiple baseline) design, we expect to complete the 
implementation measures of the Behavioral Health Integration in Medical Care Index (BHIMC) 
measure and the Time-driven activity-based Costing Approach measure (TDABC) approximately every 
year prior to the time of implementation launch at a given site (starting with baseline in the spring of 
2017); at the time of implementation launch; and approximately every year thereafter at each site.   
We plan to administer the implementation context and outcomes measure (with providers and 
organizations) shortly after they have been trained in the model of care (approximately 3 months before 
site launch at a given site) and approximately every 6 months thereafter at each site from the start of the 
site’s launch   
 
Most participant-level assessments will be completed at the time a patient joins the study and every 3 
months thereafter. The Integrated Measure of Implementation Context and Outcomes in Low and 
Middle Income Countries will not be administered to patients at baseline and will be administered at six 
Javesalud
Santa Rosa de Viterbo 
Salud de Tundama
Guasca
Armero-Guayabal
Soacha 
MONTH
Year 2
Aug-17 BHIMC
BHIMC
BHIMC
0
Nov-17 IMICO, PSAT, TDABC
3
Feb-18 Site Launch
6
Mar-18
TDABC
TDABC
7
May-18
Provider training, IMICO, PSAT
9
Year 3
Jul-18 Qualitative  Interviews
11
Aug-18
Site Launch, BHIMC
12
Sep-18
13
Nov-18  IMICO, PSAT
Qualitative  Interviews
15
Jan-19
Provider training
17
Feb-19 Qualitative Interviews BHIMC
Qualitative  Interviews, IMICO 
PSAT
Site Launch, BHIMC, TDABC, 
IMICO, PSAT
BHIMC, TDABC
TDABC, BHIMC
TDABC, BHIMC
18
May-19
Qualitative  Interviews
Provider training, IMICO, PSAT
21
Year 4
Aug-19 IMICO, PSAT
IMICO, PSAT, BHIMC, TDABC
Qualitative  Interviews, 
IMICO, PSAT
Site Launch, BHIMC, TDABC
24
Nov-19
Qualitative  Interviews
Provider training, IMICO, PSAT Provider training, IMICO, PSAT
27
Feb-20 IMICO, PSAT, BHIMC, TDABC IMICO, PSAT
IMICO, PSAT, BHIMC, TDABC
Qualitative  Interviews, 
IMICO, PSAT
Site Launch, BHIMC, TDABC
Site Launch, BHIMC, TDABC
30
May-20
Qualitative  Interviews
Qualitative  Interviews
33
Year 5
Aug-20 IMICO, PSAT
IMICO, PSAT, BHIMC, TDABC
IMICO, PSAT
IMICO, PSAT, BHIMC, TDABC
Qualitative  Interviews, 
IMICO, PSAT, BHIMC, TDABC
Qualitative  Interviews, 
IMICO, PSAT, BHIMC, TDABC
36
*Patient measures for the 
full implementation study 
are collected at baseline 
and every three months 
thereafter for a period of 
12 months (5 time points). 
Measures include: PHQ-8, 
QDS, WHODAS, GAD-7, 
NSMOS/NMED. The HPQ is 
administered every 6 
months. The ED-5D and 
TLFB are only collected for 
pilot participants 
**Qualitative Interviews 
include: Administrative 
interviews, provider 
interviews, and patient 
interviews

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
36
month intervals instead of three month intervals, given the lack of sensitivity of the measure to change 
over short periods of time. The Health and Performance Questionnaire (HPQ) will be also be 
administered at 6 month instead of three month timepoints, given the length and patient burden of the 
measure.   
12.1 
Quantitative Outcome Measures  
 
12.1.1 Summary of Quantitative Outcome Measures 
We will measure the ability of our proposed approach to accelerate the translation of evidence-based 
mental health services into practice and expand research capacity at multiple levels, including the 
following implementation outcomes: 
 
(1) Ability to accelerate the adoption of science-based mental health service delivery (including its ability to 
increase provision of evidence-based mental health resources to more individuals) 
 
(2) Acceptability as a model of healthcare service delivery (the ability of the technology-based service-
delivery model to increase patient activation and engagement in their own self-management)  
 
(3) Cost-evaluation of services in the model 
 
We will also collect patient level data outcomes to assess: 
 
 (4) Impact on public/population health (including its effects on accelerating improved behavioral 
health and health outcomes, and improving patient quality of life and functioning). 
 
Implementation and Patient level outcomes are detailed below. 
 
To promote an understanding of contextual facilitators and barriers to successful implementation of 
technology-based treatments, a common framework for measurement of implementation factors and 
implementation outcomes is needed so data can be meaningfully synthesized and interpreted across 
studies. Such harmonization of measurement has been strongly heralded by behavioral treatment 
development and implementation science researchers as critical to moving the fields forward.72-75  
 
Overall Framework for selection of implementation measures. We will use the Consolidated 
Framework for Implementation Research (CFIR;76 Damschroder et al., 2009) as an organizing 
framework to guide compilation of measures to evaluate barriers and facilitators to implementation in 
each of the following domains: intervention characteristics, organization characteristics (e.g., climate, 
readiness), individual characteristics (i.e., provider/staff attitudes, experiences; patient attitudes and 
experiences), and external influences (e.g., socio-political characteristics, local technology 
infrastructure i.e., wireless in the community, state policy/regulations, and reimbursement models).  
Strategies related to planning, facilitating provider and patient engagement in use of the novel mental 
health service delivery model, and potential sustainability issues will also be explored.  
 
We will focus generally on three categories of implementation outcomes for inclusion in the assessment 
battery: penetration (i.e., the extent of integration of the innovation within a service setting and its 
subsystems), acceptability (i.e., perception among stakeholders that intervention is acceptable) and 
costs (i.e., resources and unit costs).  
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
37
 
12.1.2 Quantitative Implementation Measures 
 
Integrated Measure of Implementation Context and Outcomes in Low and Middle Income 
Countries.  We have worked with our diverse group of stakeholders to review existing measures for 
the implementation context and outcome constructs. We have compiled existing measures based on 
available meta-analyses and systematic reviews of the literature,77 as well as existing, and growing, 
repositories of measures related to health services implementation science.78, 79  We have reviewed 
potential measures based on psychometric properties, use across diverse populations, predictive validity 
for implementation outcomes, and extent to which they are practical and easy to use within diverse 
settings.80  
 
During the course of this process, we identified a measure that captures our broad array of planned 
implementation metrics which has been used extensively in multiple low and middle income countries.  
This metric was developed by the lead investigators of the NIMH-funded hub working in Myanmar 
(led by Drs. Murray and Bass at Johns Hopkins University, with whom we have consulted extensively 
on the use of this metric in our project in Latin America). This measure was specifically designed to 
measure implementation factors of relevance in low and middle income countries which are not 
reflected in U.S.-based implementation metrics.81  This measure, composed of a Consumer (patient) 
instrument, a Provider level instrument, an Organizational Staff level instrument and a 
Sustainability instrument will be our primary quantitative assessment tool in this project. 
 
The Consumer (patient) instrument consists of scales to measure Acceptability (17 items), Adoption 
(12 items), Appropriateness (13 items), Feasibility (14 items), and Penetration (8 items). The Provider 
level instrument contains 16 items to measure Acceptability, 9 items for Adoption, 16 items for 
Appropriateness, 20 items for Feasibility, 8 items for Penetration and additional scales to measure 
Organizational Climate (13 items), and Organizational Leadership (10 items). At the Organizational 
Staff level there are 10 Acceptability items, 13 Adoption items, 12 Appropriateness items, 14 
Feasibility items, 8 Penetration items, 15 Organizational Climate items and 10 Organizational 
Leadership items. Each scale is scored on a 4-point ordinal scale ranging from 0 “not at all” to 3 “a lot,” 
with an additional category for “don’t know/not applicable.” 
 
In addition, the Sustainability instrument was adapted from the Program Sustainability Assessment 
Tool (PSAT).  The original PSAT is a 40-item instrument used to assess a program’s current capacity 
for sustainability across a range of organizational and contextual factors.82 The goal of the PSAT is to 
identify barriers and facilitators to program sustainability, which can be used to guide development of a 
sustainability action plan.83 The PSAT was developed based on a literature review of 85 studies focused 
on sustainability of public health programs. Data from these studies identified eight core domains that 
affect a program’s capacity for sustainability. These domains include: environmental support, funding 
stability, partnerships, organizational capacity, program evaluation, program adaptation, 
communications, and strategic planning.84 It has demonstrated internal consistency reliability, structural 
validity and usability.82, 85  
 
Background to Development of Consumer, Provider, Organizational Staff Level and 
Sustainability Level Implementation Measure for Low and Middle Income Countries (LMICs). 
Colleagues leading an NIMH research hub in Myanmar extended their long-standing work in adapting 
and testing mental health measures across cultures and contexts86-88 to adapt and develop measurement 
instruments to study implementation of mental health programs in low and middle income countries. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
38
Based on results from an initial study using implementation measures developed in the US to study the 
implementation of evidenced-based mental health programs in Zambia, Iraq and Thailand, they 
determined that there was a need for new implementation science tools. The implementation measures 
developed in the US, even though adapted, showed inadequate psychometric properties, were reported 
to be impractical for regular use, had multiple items that did not work well in the study contexts (e.g. 
questions about insurance reimbursements, automated billing, etc.), made assumptions about 
underlying health care systems (access to regular training and educational material, accreditation 
agencies, etc.), did not tap into relevant implementation outcomes (i.e. Proctor et al. 2011),73 and were 
not applicable across multiple stakeholder levels.  
 
In response, the team developed a measure (The Integrated Measure of Implementation Context and 
Outcomes in LMICs) based on the implementation science outcomes of Acceptability, Adoption, 
Appropriateness, Feasibility, Sustainability and Penetration.73  Instruments were populated through 
several steps: 1) careful delineation and operationalization of outcomes in the context of low and 
middle income countries; 2) review of three leading implementation science frameworks: Consolidated 
Framework for Implementation Research (CFIR)76; the Reach, Effectiveness, Adoption, 
Implementation, and Maintenance framework (RE-AIM)89; and the Exploration, Preparation, 
Implementation and Sustainment (EPIS) framework90; 3) consultation with experts in the field of 
international health, health systems, global mental health, and implementation science; and 4) use of 
logistical frameworks to draft each instrument and link them to overall program goals.  
 
Their new instruments were tested using mixed methods as part of a project focused on scale-up of 
evidence-based mental health programs in Iraqi, Kurdistan and Myanmar. Questions related to adoption 
and sustainability were explored qualitatively only. The reliability and construct validity of the 
instruments were tested among consumers (N=155 in Kurdistan; N=157 in Myanmar), mental health 
providers (N=26 in Kurdistan and Myanmar), organizational level staff (N=52 in Kurdistan; N=4 in 
Myanmar), and policy level personnel (N=12 in Kurdistan; N=4 in Myanmar). Results indicated that 
questions were easily translated and understood. Reliability results showed adequate internal 
consistency reliabilities for all provider level scales across settings (Chronbach’s Alphas ranged from 
0.71 for the feasibility scale in Myanmar to 0.95 for the appropriateness scale in Myanmar). In 
Kurdistan, organizational level scales all showed adequate internal consistency reliability. At the 
consumer level, the acceptability and penetration scales showed adequate internal consistency 
reliabilities (α>0.70) in both sites. Psychometric properties of the organizational level scales in 
Myanmar and policy-level scales in both settings were not evaluated due to prohibitively small sample 
sizes.  
 
Evaluation of construct validity showed that all scales at the provider level except the penetration scales 
were strongly positively correlated with the other implementation domains in both sites. Evidence to 
support the construct validity of scales at the consumer and organizational level was weaker.  
 
Based on the instrument testing results in Kurdistan and Myanmar, the team has spent the past six 
months revising the measurement instruments and identifying new instruments that would add to the 
existing measures. Revision has been done based on the qualitative and quantitative results from the 
previous studies and in collaboration with experts in the field of implementation science. The resulting 
instruments are intended for use across different populations and settings and will be tested in 
Myanmar, the Ukraine and Zambia. The instruments have been developed based on the work with 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
39
mental health programs, but are intended to be adaptable for use in general health contexts in low and 
middle income countries.  
 
We will translate this measure into Spanish and make any changes (as needed) for use in the 
Colombian context (although we expect no such changes may be needed).  We will share lessons 
learned in using this measure in Latin America with the Myanmar team and other NIMH-funded hubs 
working in low- and middle-income countries. 
 
Behavioral Health Integration in Medical Care Index (modified for Colombia). As described 
above, the Behavioral Health Integration in Medical Care Index (BHIMC) is an organizational measure 
of the level of behavioral health integration in medical practice settings. It evaluates policy, clinical 
practice and workforce dimensions of integration using mixed methods, i.e. combination of document 
review and observation.   The BHIMC will be administered by a trained research staff member.  The 
BHIMC has been adapted for use in Colombia, has been translated to Spanish, and has been described 
extensively in Section 6.3 above. 
 
12.1.3 Cost of Implementing the Model of Care 
 
The cost of providing mental health care within primary care settings in Latin America is not well 
characterized. It is critical to develop a better understanding of the costs of delivering mental health 
care in order to enable efficient and effective allocation of resources, including staff and medications, 
and to facilitate the integration of mental health care services into other primary care settings 
throughout Colombia, Latin America, and elsewhere. Measuring the costs of delivering the integrated 
model of mental health care in the current project will also be important for supporting the long-term 
sustainability of this model across diverse settings by allowing provider organizations to make 
informed decisions about staffing and budgeting.  
 
It is important to note that measurement of the costs and resources used for delivering the proposed 
model of care are distinct from the price or fees for providing services or the reimbursement rates for 
different services. The costs of implementing the model of care are also distinct from the potential cost-
benefit of providing mental health care integrated within primary care settings from the individual, 
payer, or societal level perspective.  
 
The Time-driven Activity-based Costing (TDABC) Approach. The costs of implementing the 
proposed model of care for depression and alcohol use disorders will be measured using the time-
driven activity based costing (TDABC) approach.91, 92 The TDABC method of cost measurement 
involves creating a detailed process map to illustrate every administrative and clinical process activated 
during the treatment of depression and alcohol use disorders over a complete care cycle. The care cycle 
refers to the standard treatment course for patients seen at each clinic and typically begins with the 
moment that patients enter the clinic and continues through to the moment when they leave, and 
includes any necessary follow-up visits. The TDABC method will be used to determine the costs of the 
specific human, equipment, and facility resources used for delivering mental health care to patients as 
part of the model of care.  
 
The TDABC approach will involve the following four steps: 
 
STEP 1 (‘Activities’): First, it is necessary to identify all of the activities, and the personnel who 
perform these activities, involved in providing treatment for depression and alcohol use disorders at 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
40
each of the clinic sites. This step will require shadowing patients and meeting clinical and 
administrative staff involved in the care process to determine how much time they spend on each of 
these activities. Process maps will be created to illustrate the flow of patients, to identify all activities 
involved in providing care, and to facilitate the remaining steps in the TDABC approach. Separate 
process maps will be generated for each of the participating clinical sites. A combination of shadowing, 
surveys of clinic staff, and interviews with staff and administrators will be used to determine the time in 
minutes required for each activity outlined in the care process. 
 
STEP 2 (‘People’): This step involves calculating the cost per unit of time for each of the personnel 
and equipment used throughout the various activities identified in Step 1. By determining the costs per 
unit of time, this step generates capacity cost rates. For example, to estimate the cost per minute for all 
the clinical and administrative staff involved in the care process, an individual’s annual compensation 
will be divided by the total number of minutes per year that the person is available to work with 
patients. A combination of surveys and interviews will be used to gather these details. 
 
STEP 3 (‘Materials’): This step involves measuring the amount of materials, supplies, and 
medications that are consumed during the care processes identified in Step 1 and assigning costs to 
each of these items. Consumables can refer to any type of items that are used during the delivery of 
care, including latex gloves, test strips, or medications.  
 
STEP 4 (‘Facilities’): This final step involves determining the costs of things that are not directly 
consumed during the care process. For example, indirect costs such as the facility costs, electricity, 
building maintenance and cleaning, administrative costs, and health IT costs. 
 
The TDABC approach enables providers to determine the costs of specific care pathways at the 
individual patient-level. This makes it possible to assess how costs potentially change based on the 
increasing complexity of patients’ conditions (e.g., increasing age, multiple comorbidities, co-occurring 
substance use disorder), or as revisions are made to the delivery of care.  
 
The TDABC approach can also make it possible to identify inefficiencies in the provision of care, and 
can inform efforts to move staff to higher value activities in order to improve outcomes for patients. For 
example, the TDABC approach can illustrate activities in the care process that could be performed by 
lower-paid staff instead of highly trained and expensive nurses or physicians, thereby enabling the 
more costly personnel to perform more complex clinical duties. This is especially important in settings 
where highly trained personnel such as nurses and physicians are in limited supply. Though efforts are 
necessary to ensure that any tasks that are shifted to different personnel will be performed at the same 
or higher level of quality. 
 
The TDABC approach has been used in several clinical settings in the United States and in Europe, and 
has contributed to significant improvement in the efficiency and effectiveness of care delivery for 
patients with diverse medical conditions. Through improved resource allocation, the TDABC approach 
can identify opportunities in the delivery of care to improve value for patients by providing more 
effective care for each unit of cost. For example, in the current study, the TDABC approach may 
demonstrate that the proposed model of integrated care for depression and alcohol use disorders 
contributes to increases in costs. However, by linking cost measurement to patients’ clinical outcomes, 
it may be possible to demonstrate that the added costs contribute to better outcomes and improved 
value for patients.  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
41
 
Timeline for TDABC Measurement. The TDABC measure will be collected before implementation 
of the integrated model of care in order to obtain an initial assessment of the costs of providing primary 
care at each of the study sites. This initial measurement using the TDABC approach will be essential in 
order to fully document use of staff, equipment, and facility resources within each clinical setting. The 
TDABC measure will then be collected following implementation of the care model (every year) to 
assess any changes in resource utilization. The TDABC approach will make it possible to capture 
changes in activities and associated costs following the implementation of the new model of care. All 
follow-up measurement using the TDABC approach will occur together with collection of the BHIMC 
tool. 
 
TDABC Training. Training for using the TDABC approach will be provided to members of the 
research team and collaborators during a 2-day meeting to be held in Bogota in June 2017. This training 
will involve an overview of the procedures for accurately collecting data on resource utilization during 
patient visits using surveys and interviews. The training will also involve practicing this method at a 
demonstration site in Bogotá. The training will be conducted in a similar manner as the training for use 
of the BHIMC tool, and will contribute to ongoing project capacity building activities. 
 
Other Cost Measures 
 
Non-Study Medical and Other Services (NSMOS). We will include a measure associated with cost 
impact: Non-Study Medical and Other Services (NSOMS).  The NSOMS assesses patients’ medical 
resource use that is not part of the intervention, including non-treatment therapy visits, physician visits, 
residential and/or hospital detoxification, hospital and emergency department visits, and medication use 
through patient self-report. Through this effort, we will have a data collection instrument that can be 
applied in multiple settings.   We will additionally assess cost per beneficiary data, as possible, using 
data from our partnering primary care sites. We have slightly modified this measure by removing 
sections that are not applicable to the primary care context in which we are working (ex: 
hospitalizations). 
 
Non-Medical Expenses for Depression. This measure assesses the non-medical costs of depression.93 
This measure is in Spanish and has been used in Latin America and used by our team at Javeriana. 
 
Additional Measures 
 
Health and Work Performance Questionnaire (HPQ).  This measure, developed by the World 
Health Organization, assess the impact of depression on work performance (including sickness absence, 
presenteeism, and critical incidents).94 This measure is in Spanish and has been used in Latin America 
and used by our team at Javeriana. 
 
Patient Outcomes. In addition to the patient measure for use in low and middle income countries 
described above, the primary patient outcome measure we will use to assess our hypothesized reduction 
in depression is the standardized Patient Health Questionnaire (PHQ-8).95  We will assess problematic 
alcohol use (among the sample with problematic alcohol use) via the Quick Drinking Screen (QDS),96 
which is a 3-item questionnaire which asks about average drinking habits over the last ninety days.    
To assess functioning, we will use the 12-item World Health Organization Disability Assessment 
Schedule 2.0 (WHODAS)97, 98 (to measure functional status and health-related quality of life).   

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
42
Specifically, the WHODAS 2.0 assesses health-related difficulties across six different domains of 
functioning that are linked conceptually and operationally to the International Classification of 
Functioning, Disability and Health (ICF). The six major life domains are related to an individual’s level 
of functioning and include: cognition (understanding and communication); mobility (ability to move 
and get around); self-care (ability to attend to personal hygiene, dressing and eating, and to live alone); 
getting along (ability to interact with other people); life activities (ability to carry out responsibilities at 
home, work and school); and participation in society (ability to engage in community, civil and 
recreational activities).  The WHODAS 2.0 has shown good validity, internal consistency, test-retest 
reliability, and agreement with other measures of disability across many different settings and 
countries. The WHODAS 2.0 is also available in Spanish, and has been found to perform well across 
different cultures, among different subgroups, among people with physical disorders, and among 
people with mental health problems or addictions. Importantly, the WHODAS 2.0 has been used in 
several large clinical trials conducted in low-income and middle-income countries evaluating the 
integration of depression treatment or mental health care into primary care or other community settings, 
or treatment of alcohol use disorders in primary care settings.99-102  
We will also assess anxiety among patients enrolled in this study. Anxiety is one of the most prevalent 
mental health conditions in Colombia.107-109 According to the National Mental Health Survey (2015) 
the estimated lifetime prevalence of any anxiety disorder in adults between 18-44 years of age, and 
adults older than 45 years of age is 4.5% and 3.1% respectively.109 Symptoms of anxiety frequently co-
occur with depression and alcohol use disorders.110, 111 Symptoms of anxiety are known to interfere 
with treatment for depression and alcohol use disorders, and negatively impact long-term treatment 
outcomes and illness course.112 Given the high prevalence of anxiety disorders in Colombia, these 
symptoms likely interfere with treatment for other mental health conditions. Therefore, in the current 
study, it is critical to assess whether patients receiving treatment for depression or alcohol use disorders 
as part of the proposed model of care also experience symptoms of anxiety. It is important to ascertain 
whether the presence of anxiety may have an impact on treatment outcomes over time. We will use the 
General Anxiety Disorder screener (GAD-7), which is a 7-item self-reported screening questionnaire 
that has been validated to assess for generalized anxiety disorder (GAD) in outpatient and primary care 
settings.113 The GAD-7 assesses seven items using a 0 to 3 scale. Screening and severity rating is based 
on the total score. The GAD-7 has been used to screen for anxiety disorders in primary care settings in 
low-income and middle-income countries99, and among individuals exposed to violence and armed 
conflict in different regions of Colombia.114 
Most patient measures will be assessed at baseline and every 3 months for a period of 12 months (for a 
total of 5 assessment timepoints per patient). The Integrated Measure of Implementation Context and 
Outcomes in Low and Middle Income Countries will not be administered to patients at baseline and 
will be administered at six month intervals instead of three month intervals, given the lack of sensitivity 
of the measure to change over short periods of time. The Health and Performance Questionnaire (HPQ) 
will be also be administered at 6 month instead of three month timepoints, given the length and patient 
burden of the measure. Although we will seek data from patients at these time intervals, data can be 
collected within 2 weeks before or after the targeted assessment date. Patients will be prompted by 
research and/or clinical staff when they are due to complete patient assessments.  All measures will be 
offered to patients in Spanish. 
 
COVID-19 Impact Outcomes. We will also assess the impact of COVID-19 among patients, 
providers, and administrators enrolled in this study currently or who have completed the study. Given 
the unknown and widespread implications of the COVID-19 pandemic in Colombia, it is critical to 
understand the context of COVID-19 for interpreting the study’s findings. We will use the COVID-19 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
43
Impact Survey, which is a self-reported questionnaire developed by the NIMH U19 Scale-Up Hubs. 
The COVID-19 Impact Survey has been slightly modified from the version prepared by the NIMH U19 
Scale-Up Hubs by removing subsections that do not assess the potential impact of COVID-19 (i.e., we 
removed the question “are you having thoughts of killing yourself right now” because, as worded, this 
question will not allow us to understand if responses are related to COVID-19), adding site-specific 
language (i.e., “since the lockdown in March 2020”), and adding follow-up items to clarify responses. 
The patient instrument consists of scales to measure Local Response to COVID-19 (1 item), COVID-
19 Exposure (5 items), Impact of COVID-19 (19 items) and Access to Mental Health Services (10 
items). The provider instrument consists of scales to measure Impact on Mental Health Services (8 
items), Impact on Provider Burden and/or Burnout (3 items), and Impact on Experiences of Stigma (4 
items). The administrator instrument consists of scales to measure Local Policy Response to COVID-19 
(1 item) and Impact on Mental Health Services (6 items). The COVID-19 Impact Survey will be 
administered to patients already enrolled in the study at their follow-up visit(s) that occur every 3 
months or to patients who have already completed the study at a separate scheduled visit after 
completion of the study. The COVID-19 Impact Survey will be administered to providers and 
administrators at their follow-up visit(s) that occur every 6 months or at separate scheduled visit(s). All 
measures will be offered to participants in Spanish.  
 
12.2 
Qualitative Outcome Measures 
In addition to the quantitative outcome measures described above, we will also conduct interviews with 
partner setting administrators, clinical and administrative staff, and patients to evaluate the current state 
of treatment in Colombia and to help inform the model.  These interviews will be conducted by our 
qualitative research team at Javeriana at 3- and 6-months post-implementation (+/- a 2 week window 
around the targeted data collection date).  The goal of these interviews will be to evaluate stakeholder 
acceptance of the program and success of the dissemination strategies over the implementation period.  
 
 
12.2.1. Administrative/provider Interviews 
Administrative/provider interviews will assess implementation experiences, challenges to 
implementation and strategies used to facilitate implementation with different patient populations, 
changes made to workflow with use of the intervention, and salience of monitoring and feedback 
processes to inform providers and other stakeholders about the implementation success (approx. 5 at 
each site). We will also assess the extent to which patients initiated use of the program via inquiry of 
their providers after learning of the program through patient channels.   
 
 
12.2.2. Patient Utilization Interviews/observations 
Patient Utilization. We will recruit and follow a sample of patients (approx. 5 at each site) during the 
implementation project to explore feasibility issues regarding use, barriers and facilitating strategies for 
using the program, optimal dissemination strategies for promoting use of the program, and overall 
experiences using the program over time.  These patient participants will be encouraged to document 
their experiences with the mental health service delivery model in an ongoing manner (e.g., note 
jotting, pictures, audio recording).  
 
 
13.0 
STATISTICAL DESIGN AND ANALYSES  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
44
13.1 
Statistical Analyses. 
Organizational implementation outcomes and patient-level implementation outcomes will be analyzed 
via linear mixed effects models (LMM) as is recommended for data from stepped wedge designs,102 
with the primary comparison being mean outcome before and after implementation of the novel care 
model. Each site will contribute observations both before and after implementation, and the model will 
include a fixed effect for time-period (to account for trends in outcome over time that are due to factors 
other than treatment implementation) and whether or not a time-period is before or after 
implementation. The primary hypothesis will be examined by testing if the coefficient of the pre- vs. 
post-implementation indicator term is different from zero. To account for potential correlation of 
observations within site, the model will include a random site effect; thus all statistical tests comparing 
outcomes pre- and post- implementation will take this within-site non-independence into account. 
LMMs use all available outcome data and allow for an unequal number of observations across sites and 
across participants per site. 
 
Patient-level outcomes (evaluated longitudinally in participants from sites implementing the novel care 
model) will be evaluated via linear mixed effects models. These models will include fixed effects for 
time from enrollment to evaluate whether patient-level outcomes improve over time in sites 
implementing the novel care model. The models will also include a random site effect to account for 
similarities of outcomes of individuals within the same site, and random individual-level intercept and 
slope terms, to account for non-independence of repeated assessments within individual.  
 
In addition to analyzing site-specific results (comparing sites before and after receiving the novel 
mental health service delivery model vs. the matched comparison site), we will aggregate data and 
conduct cross-site analyses to examine the extent to which patterns of results are similar/differ across 
populations and contexts.  To integrate data across studies to explore patterns and generality of 
outcomes, we will use structural equation modeling (SEM)-based meta-analysis and meta-analytic 
SEM (MASEM)115, 116.  MASEM is a type of path analyses synthesizing meta-analytic findings and it 
involves a two-stage approach: the first stage integrates data from several studies to produce pooled 
correlation or covariance matrices, and the second stage uses the pooled matrices as input data for 
inclusion in an SEM framework.   
 
We will also document participants’ mobile technology access/use in the study and examine how observed 
differences in access/use may relate to the perceived utility of the model as well as patient outcomes. 
 
13.2 
Rationale for Sample Size and Statistical Power  
Sample Size.  As we implement and evaluate our planned model of care across our collaborating 6 sites 
in accordance with our stepped wedge design, we expect a sample size of at least 1200 patient 
participants across this implementation project (sites that launch earlier will likely contribute more data 
and rural sites may contribute less data overall). These participants will contribute data to participant-
level outcome variables (at 5 timepoints; enrollment, 3M, 6M, 9M, and 12M) and participant-level 
implementation outcomes (during the site evaluations every 6 months after implementation). 
Organizational implementation measures will be collected from providers and staff during 6-12 month 
intervals (both pre- and post-implementation) for a total of up to 6 intervals, with 10 providers and 10 
staff members contributing data at each interval, and 25 patients contributing data during an interval for 
the IMICO and PSAT.   
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
45
The modified stepped wedge design (6 sites, 2-6 time-periods, 10 observations per site per time-period) 
for the organizational implementation measures will yield 80% power at the two-sided 0.05 
significance level to detect differences pre- vs. post-implementation that are at least 0.55-0.73 times a 
standard deviation.102 This is a moderate-large standardized effect size. For the patient measures with 
25 observations per site per time period, there will be 80% power at the two-sided 0.05 significance 
level to detect differences pre- vs. post- implementation that are at least 0.35-0.56 times a standard 
deviation. Because this is a modified stepped wedge design in that the number of assessment time-
periods varies per site, we have based the power calculations on a full stepped wedge design with three 
and four time-periods, the average number of time-periods available per site.   
 
The BHIMC and TDABC are collected once per site per time-period by research staff members. 
Assessment of change across time will be primarily used to evaluate qualitative changes pre- and post-
implementation, but large effect sizes (1.78-2.4 standardized effect or greater) will be observed with 
80% power at the two-sided 0.05 significance level. This sample size will allow us to detect the 
stability and replicability of cost savings among members within the target population. And, by 
recruiting about a portion of the sample from each of the partnering primary care systems, this sample 
size will allow us to demonstrate the feasibility and scalability of this model in multiple settings.   
Longitudinal analysis of patient-level outcomes post-implementation will include data from at least 
1200 participants. At a sample size of at least 1200 (with, at least, two time-period measures per 
participant which is a highly conservative estimate of data collection), we will have at least 80% power 
at the two-sided 0.05 significance level to detect small pre- vs. post-implementation effect sizes 0.13-
0.17. 
13.3 
Interim Analysis  
We propose to conduct an interim analysis at the completion of the pilot project. We also propose to 
conduct an interim analysis half-way through the implementation study.  This interim analysis will not 
include stopping rules for efficacy or futility. Instead, the results of these analyses will aid in modifying 
our implementation process, as needed, to better ensure sustainability. Formal efficacy evaluation will 
not take place until the completion of data collection.  
13.4 
Exploratory Analysis  
We will additionally conduct exploratory moderator/mediator117 analyses to examine how 
implementation context variables relate to implementation outcomes.  A powerful approach to 
inference for indirect or mediated effects is bootstrapping.  Bootstrapping118 involves taking a random 
sample from the data with replacement numerous times and using the variability in the statistic from 
sample to sample to construct an interval estimate119-121 conveying the direction, magnitude, and 
precision of an indirect effect.  Indirect or mediated effect estimation that involves multiple mediators 
or lengthier causal chains is typically done in the context of a structural equation model122 (SEM) or 
more recent extensions of SEM.123, 124 Where current, adequate measures of mechanisms of action are 
unavailable, novel measures can be developed using confirmatory factor analysis125 or item-response 
theory (IRT)126 modeling. The two-stage approach to meta-analysis of structural equation models 
described by Cheung and Chan115 can be used to synthesize indirect effects across a series of studies 
investigating similar interventions, mechanisms of action, and outcomes. 
13.5 
Missing Data and Dropouts  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
46
The number (%) of sites and patients with complete data will be reported. If scales have recommended 
methods for dealing with missing data, these will be applied. If scales do not have recommended missing 
data methods, multiple imputation will be used using 10 imputations and a fully conditional model based 
on Markov Chain Monte Carlo (MCMC). Once multiple imputations are conducted, the imputed data set 
will be analysed as a secondary analysis.  Note that all quantitative data will be collected in REDCap.  
REDCap allows participants to skip questions they do not wish to answer.  If a participant drops out 
before we collect all data, their collected data will still be used in analyses. 
13.6 
Demographic and Baseline Characteristics  
Baseline demographic and clinical variables will be summarized for participants enrolled in the study. 
Descriptive summaries of the distribution of continuous baseline variables will be presented with 
percentiles (median, 25th and 75th percentiles), and with mean and standard deviation. Categorical 
variables will be summarized in terms of frequencies and percentages. 
13.7 
Safety Analysis  
Adverse Events (AEs), including Serious Adverse Events (SAEs), will be presented as: (1) the number 
and proportion of participants experiencing at least one incidence of each event overall; and (2) the total 
number of each event overall in tabular form. Listings of SAEs will be sorted by system organ class 
(SOC), and preferred term (PT). Detail in these listings will include severity, relationship to study, and 
action taken, as available. 
13.8 
Incidental Findings 
An issue of increasing importance in human participant research is that of “incidental 
findings.”  Incidental findings refer to observations of potential clinical significance unexpectedly 
discovered in research participants and unrelated to the purpose or variables of the study.  If any 
research staff member and/or study clinician becomes aware of a potential incidental finding about a 
patient during the course of this study (e.g., patient may have undiagnosed symptoms of schizophrenia), 
this will be communicated to the clinical team at the primary care study site and the team will follow 
their standard clinical policies to appropriately assess the patient and provide/refer to care as needed. 
14.0 
INFRASTRUCTURE SUPPORT 
14.1 
Stakeholders 
This partnership was launched via the support of the World Bank and includes a broad array of 
stakeholders, including academic organizations (Dartmouth College in the US; Pontificia Universidad 
Javeriana in Colombia; Universidad Peruana Cayetano in Peru; Pontificia Universidad Catolica in 
Chile), governmental organizations (Ministry of Health in Colombia; National Institute of Mental 
Health in Peru), as well as non-governmental and/or multilateral organizations (PAHO/WHO; The 
World Bank; industry partners; and primary care and regional hospital systems in Latin America).  
 
We have expanded this broad group of stakeholders in Latin America in the proposed project to also 
include perspectives of non-profit patient advocacy organizations (Fundacion Internacional Unidos 
Contra la depression) and a public/private non-profit organization focused on quality decision-making 
in clinical practice and health policy (Instituto de Evaluación Technológica en Salud).   We have 
additionally engaged large insurance companies in Colombia, which will be key to the sustainability of 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
47
this work.   We have further engaged both US and Latin American-based research teams conducting 
implementation research that employs mobile health technology in several regions of Latin America 
(including Brazil, Guatemala, and Ecuador) (Dr. David Mohr at Northwestern University in Chicago 
and Dr. Paulo Rossi Menezes at the University of São Paulo in Brazil).  This partnership will leverage 
the investment of NIMH in these collective activities to enable a new level of discovery and 
accomplishment across projects and offer the opportunity to accelerate the tempo and scientific 
achievement from this line of research.   This partnership will also greatly enhance research capacity 
across Latin America.   Overall, we have been fortunate to assemble an outstanding and broad team of 
collaborators and stakeholders which greatly enhances our ability to achieve the goals of the proposed 
project. These partnerships will also ensure that the study design is responsive to local needs, resources 
and expertise and enables synergies beyond what can be achieved through a traditional research study.   
14.2 
Project Launch and Annual Team Meetings 
In the first quarter of Year 1, the project leadership held a planning meeting at Dartmouth (June 2016) 
and then we met in person with our entire team of stakeholders in Bogotá, Colombia for the initial 
project planning and project launch (September 2016). This face to face meeting with all stakeholders 
will then be repeated annually. 
14.3 
Administrative Meetings 
The Administrative Team holds weekly meetings with both the Director’s office and research team for 
ongoing communication and study management. 
14.4 
Learning Collaborative Meetings 
Learning Collaborative sessions will be held quarterly by webinar and share lessons learned – including 
successes and challenges in implementation.  These meetings will be in addition to the once per year in-
person meeting in which these individuals will participate.  Data collected at each site will be 
aggregated by site for review and discussion at this quarterly meeting. Quarterly meetings of this 
learning community will be co-led by Multiple PI Gómez and Consultant Uribe. This will allow for a 
rich sharing of information -- both from the discussions of collaborative participants as well as the rich 
systematically-collected dataset – which are key to refining and expanding science-based 
implementation efforts over time. 
14.5 
Research Capacity-building Meetings 
This collaboration will entail quarterly “Research Capacity Building meetings” of the full Research 
Team, Consultants, Governmental, Non-Governmental and Multilateral Organizations, and NIH Project 
Scientists led quarterly by the Director’s Office with all project personnel to discuss progress within 
each site, opportunities for increased cross-site collaboration/learning and areas where resources could 
be more efficiently shared, needs/progress of investigative teams, emerging trends in the field, priorities 
of the work, and training opportunities to address educational gaps/interests.  Additionally a key part of 
this discussion will focus on shared learning and research capacity building across other parts of Latin 
America (Brazil, Ecuador, Guatemala) where Drs. Mohr and Menezes are conducting NIMH-funded 
research on scaling-up evidence-based mental health care. 
14.6 
Annual NIH Meetings 
Core members of the project team will participate in annual NIH meetings to share lessons learned 
across NIMH-funded hubs. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
48
14.7 
Partnership and Advisory Board Meetings 
We have engaged a governmental/multilateral organization advisory board in Latin America [including 
members of PAHO/WHO, the Ministry of Health in Colombia, the Ministry of Communication and 
Information Technology in Colombia] intended to ensure the intervention model is grounded in mental 
health policy and that the resulting data and lessons learned from the project inform the evolution of 
mental health policy over time across Latin America.  
 
Organizations within this advisory board will be represented at in-person meetings of the entire project 
team (including research, governmental, payer, and non-governmental and multilateral organization 
partners) to be held in Bogotá, Colombia. They will be asked to review and discuss implementation 
research experiences and outcomes during the project and advise of their relevance to mental health 
policy. They will also be asked to communicate to this broad group of stakeholders the nature and 
evolution of mental health policy in Latin America over the life of the project.  
15.0 
REGULATORY COMPLIANCE AND SAFETY  
15.1 
Regulatory Compliance  
This study will be conducted in accordance with the current version of the protocol, in accordance with 
the ethical principles outlined in the Declaration of Helsinki, International Conference on Harmonization 
Good Clinical Practice (GCP) Guidelines, and all other applicable regulatory requirements.  An 
Operations Manual will be provided as a reference guide and study quality assurance tool. 
 
15.2 
Statement of Compliance  
This study will be conducted in compliance with the appropriate protocol, current Good Clinical Practice 
(GCP), the principles of the Declaration of Helsinki, and all other applicable regulatory requirements.  
Participating sites must obtain written approval of the study protocol, consent form, other supporting 
documents, and any advertising for participant recruitment from their local Institutional Review Board 
(IRB) in order to participate in the study.  Prior to study initiation, the protocol and the informed consent 
documents will be reviewed and approved by an appropriate Ethics Review Committee (ERC) or IRB(s) 
and the NIMH DSMB.  Any amendments to the protocol or consent materials must be approved before 
they are implemented.  Annual progress reports and local Serious Adverse Event (SAE) reports will be 
submitted to each IRB, and the NIMH DSMB according to their usual procedures. 
15.3 
Institutional Review Board Approval  
Prior to initiating the study, investigators will obtain written IRB approval to conduct the study from the 
Dartmouth College Committee for the Protection of Human Subjects (CPHS). Should changes to the 
study protocol become necessary, protocol amendments will be submitted in writing by the investigators 
for IRB approval prior to implementation. In addition, CPHS in collaboration with the Research Ethics 
Committee at Pontificia Universidad Javeriana will approve the study protocol, all consent forms, 
recruitment materials, and any materials given to the participant. Annual reports and progress reports will 
be submitted to the IRBs annually or at a frequency requested by each IRB so that continuous study 
approval is maintained without lapse. The lead investigator is responsible for maintaining in his research 
files copies of current IRB/IEC approval notice(s), IRB-approved consent document(s), including 
approval for all protocol modifications. These materials must be available at any time for audit. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
49
15.4 
Informed Consent  
As detailed in Section 9.3., patients at the partnering study sites who meet all eligibility detailed in 
Section 9.1 will be provided with an informed consent form to participate in the study [See Consent 
sheet in Appendix].  We intend to ask participants to complete consent forms electronically within 
REDcap (with appropriate electronic backups).   
 
Research assistants will routinely move between study sites to aid in the oversight of participant 
recruitment, study implementation, and participant assessments.  However, because we do not expect 
that research assistants will necessarily be at a given site every time a patient may be identified as 
eligible to join the study, and because the ultimate goal of this project is to create sustainable models of 
implementation, our partnering clinical staff at each study site will be trained on providing the 
electronic informed consent form to eligible patients and discussing with the patient both the 
interventions being offered to patients as well as the study expectations.  Each site will identify one or 
more staff to function in this capacity.  Clinician training on these procedures will be included in the 
planned staff training activities on this model of mental health care described in Section 8.0.  As part of 
this training, clinicians will be required to demonstrate mastery of the consent process via role-playing.  
And they will be observed periodically by research staff to ensure procedures are being implemented 
with fidelity.  Specifically, clinicians will screen patients, identify eligible patients, discuss treatment 
options for the patient and offer the consent form to patients who express interest in accessing the 
study’s therapeutic resources (e.g., the web/mobile intervention) and willingness to complete study 
assessments in accordance with the assessment timeline (every 3 months, as described in Section 12.0).   
Clinicians will offer to read aloud the content of the consent form to the patient.  The study Principal 
Investigators will regularly review participant inclusionary information to ensure eligibility criteria are 
being met. 
 
Consent Procedure.  As part of the consent process, the clinician describing the consent process will 
discuss potential risks and benefits of participation.  Risks to patients are minimal but include talking 
about/answering questions about sensitive topics (such as depression and alcohol use) that may be 
difficult or emotionally upsetting to patients.  The clinician providing consent will also discuss how the 
staff are trained to handle/manage these risks.  S/he will also provide information about confidentiality 
and the voluntariness of participation.  The clinical site will document if patients sign consent to join 
this study (in their medical record or a case note). 
 
Information on confidentiality will emphasize that no personal information will be associated with the 
data; all data will be assigned with a study identification number (ID), and only this ID will be used for 
the study assessments. The list of study IDs (along with all research data) will be kept in a password 
protected, encrypted database, which is only accessible to approved research staff.  Individual identifying 
information will only be maintained in a separate encrypted database with passwords known only to the 
PIs and specific members of the research team.  No identifying personal characteristics will be used in 
any publication or presentation. Information on voluntariness will emphasize that participation in the 
study is completely voluntary. It will be highlighted that patients retain the right to refuse answers to any 
questions that they do not feel comfortable with. Patients retain the right to withdraw from the study any 
point in time, and refusal to participate will have no negative effect on the health care the patient receives 
through the study sites or at any other health care institution.  Patients will have the opportunity to 
carefully review the written consent form and ask questions regarding the study prior to signing the 
informed consent form. The consent forms will need to be electronically signed by the patient and the 
individual who will oversee the informed consent process.  Patients will be provided with a hard copy of 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
50
the consent form and staff will document in the research record that participants received this copy.  
Depending on the patient’s interest, results of the study will be shared once the study is completed. 
 
Wherever necessary, the participant’s family will be advised of the need for confidentiality so that the 
family understands the conditions under which information will be shared. In addition to this, the 
individual providing consent at a study site will reiterate confidentiality to family members if any family 
member would be present at screening or at any other outcome assessment.  Note that adult patients in 
Colombia typically visit their primary care provider alone.  However, in cases where patients may wish 
to have a family member present, it is typical to discuss medical matters in the presence of family 
members. In these circumstances where patients may wish to have a family member present during the 
visit, their primary care provider will make an effort to confirm with the patient in private that their wish 
is to have their family member present. This approach is intended to ensure that the patient’s wishes are 
respected, and is standard practice within primary care settings in Colombia. 
 
While obtaining the consent, the patient will be explicitly informed that any incidence of violence (e.g., 
expressed intent to physically harm another person; see consent) required by law to be reported to 
authorities may be reported to the authorities without the consent of the patient. Furthermore, any 
clinical worsening (e.g. imminent risk of suicidal ideation) may result in referral for additional 
treatment and the research staff or clinicians could potentially disclose this information to treatment 
providers of the study or to family members.  If a research staff member thinks a patient is at risk of 
harm to herself or to someone else, the research staff will report this to a physician/clinician at the 
study site without the consent of the patient. This information is also reflected in the informed consent 
form. 
 
In addition to the above noted consent procedures, if an illiterate patient is eligible and wants to enter 
the study, the research assistant will read the informed consent form verbatim to the eligible participant. 
One impartial witness will be present during the reading and signing of the informed consent form to 
attest to the apparent understanding of the participant and their willingness to participate. If after 
listening to and fully understanding the informed consent form the patient wants to enter the study, the 
participant will use their thumbprint as a signature on the signature line of the informed consent form. 
Afterwards, the impartial witness will complete the information that accompanies the signature of the 
participant (PRINTED NAME and date) and the research staff will complete a note in the comments 
section of the informed consent, explaining the consent process for the participant. All of the 
procedures listed above that apply to literate participants will also apply to illiterate participants (i.e. 
they will be given opportunities to ask questions about any study procedures, will be provided with a 
hard copy of the informed consent for, etc.).  
 
Patients who provide informed consent will then be asked to complete baseline participant assessments 
(described in section 12.0 of this protocol). Assessments will be delivered via a computerized 
assessment engine (REDcap) although paper versions of assessments and the informed consent form 
will be available if necessary (e.g., patient is uncomfortable completing computerized assessment or 
there are internet connection/technical difficulties). Any data collected onto paper assessments will be 
double-data entered into the REDCap database. 
 
Additionally, provider and administrative staff at partnering primary care sites will all be offered the 
opportunity to complete the implementation context and outcome measure in this research project 
(including nurses, nursing assistants, social workers, health promoters, physicians and charge nurses, 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
51
program administrators).  All participating staff will also be asked to provide consent to participate. 
Provider and administrative staff all need to be aged > 18 years and have worked for the study site for 
at least 3 months. Research staff will provide informed consent to each staff member, who will be 
asked to sign an electronic consent in REDcap (along with the electronic signature of the research staff 
member who completed the informed consent process with staff). During a pause in face-to-face study 
activities due to safety concerns related to COVID-19, the research staff may mail informed consent 
documents for providers and administrative staff to sign and return by postal mail. Alternatively, study 
staff may provide participants the opportunity to provide informed consent signatures electronically 
through REDcap. In all cases, the study team will provide the phone number of the coordinator of each 
prospective participant’s study site to allow each potential participant to voice questions or concerns 
prior to the provision of consent. We expect approximately 4-8 provider staff per site will complete the 
provider measure of implementation context and outcomes and about 2-5 administrator staff per site 
will complete the organizational level measure of implementation context and outcomes at each 
timepoint [See Consent sheet in Appendix]. 
 
A separate informed consent will also be sought for the sub-sample of patients as well as the subset of 
staff who are invited to participate in the qualitative study, as described in Section 12.2 [See Consent 
sheets in Appendix].  A qualitative research staff member will explain the purpose of the qualitative study 
to prospective participants and invite them to participate and ask for consent for the interview being 
recorded using a digital audio recorder. During a pause in face-to-face study activities due to safety 
concerns related to COVID-19, study staff may allow all participants in the qualitative study to provide 
informed consent by postal mail or electronically, through REDcap. The study team will provide the 
phone number of the coordinator of each prospective participant’s study site to allow each potential 
participant to voice questions or concerns prior to provision of consent. 
 
We will transcribe the audio recordings of the qualitative interviews to allow us to analyze this data. 
These transcriptions will not include information that could identify participants, and will be stored, 
along with the analog notes and audio recordings, in a locked facility in Javeriana University. We may 
use a translation service, such as google translate or a professional translation service, to translate the 
transcribed interviews into English. These translations will include no information that could identify 
participants and will also be stored in a locked facility in Javeriana University. 
 
A separate informed consent will be sought for the currently enrolled patients, patients who have 
completed the study and are being re-contacted, providers, and administrators to participate in the 
COVID-19 Impact measure (which is being added as an additional study assessment), as described in 
Section 12.1. The IRB at Dartmouth and the IRB at Javeriana approved to waive a written consent from 
participants to complete the COVID-19 Impact measure as an additional study assessment due to 
COVID-19 related safety concerns. Instead, participants will be asked to provide verbal consent if they 
are willing to complete this additional assessment. In situations when it is not possible for in-person 
activities, the study team will obtain verbal consent through phone calls. During the conversation, a 
research staff member will provide information about the measure to participants by reading aloud a 
script verbatim, and invite them to participate, and ask them to give verbal consent. Whether the 
participant chooses to give consent or not will be documented for all invited participants. After the 
conversation, the study team will send a copy of the consent script via email, physical mail, or SMS to 
participants. For patients who already completed the study intervention, the study team will send a copy 
of the consent script via email or SMS and will serve as the final point of contact with our study. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
52
We will recruit ethnically diverse adult patient participants from various states in Colombia in both 
rural and urban locations. All recruitment efforts will receive appropriate IRB approval.   
15.5 
Confidentiality  
Confidentiality will be maintained in accordance with all applicable US federal regulations and/or 
state/Commonwealth law and regulations. Data will be maintained in confidence and such information 
will be divulged to the IRB, Ethical Review Committee, or similar expert committee; affiliated 
institution; and employees only under an appropriate understanding of confidentiality with such board or 
committee, affiliated institution and employees. Participant records will be held confidential by the use 
of study codes, secure storage of any documents that have participant identifiers, and secure computing 
procedures for entering and transferring electronic data.  
 
Study sites may be required by their institutions to obtain authorization from participants for use of 
protected health information. Sites will be responsible for communicating with their IRBs or Privacy 
Boards and obtaining the appropriate approvals or waivers to be in regulatory compliance. Releases of 
participant identifying information that are permitted by the HIPAA regulations (as applicable), but 
which are prohibited by other applicable federal regulations and/or state/Commonwealth law and 
regulation, are prohibited.   
15.6 
Investigator Assurances 
Each study site must file (or have previously filed) a Federal Wide Assurance (FWA) with the DHHS 
Office for Human Research Protection setting forth the commitment of the organization to establish 
appropriate policies and procedures for the protection of human research subjects, with documentation 
sent to NIMH or its designee. Research covered by these regulations cannot proceed in any manner prior 
to NIMH receipt of certification that the research has been reviewed and approved by the IRB provided 
for in the assurance (45 CFR 46.103(b) and (f)). Prior to initiating the study, the principal investigator at 
each study site will sign a protocol signature page, providing assurances that the study will be performed 
according to the standards stipulated therein. 
15.7 
Financial Disclosure  
All investigators will comply with the requirements of 42 CFR Part 50, Subpart F to ensure that the 
design, conduct, and reporting of the research will not be biased by any conflicting financial interest. 
Everyone with decision-making responsibilities regarding the protocol will confirm to the sponsor 
annually that they have met their institutional financial disclosure requirements.   
 
A financial disclosure form and conflict of interest management plan has already been submitted to NIH 
from Dartmouth College for Dr. Marsch.  Specifically, Principal Investigator Lisa Marsch, PhD is 
affiliated with Square2 Systems Inc., the business that developed the Square2® to be employed in the 
planned study.  Dr. Marsch has worked extensively with her academic institution (Dartmouth College) 
to monitor the relationship between Square2 and her academic institution and ensure any potential 
conflict between her roles in each organization is appropriately managed.  This Conflict of Interest 
management plan has also been submitted by Dartmouth College to the NIH.  Additionally, all research 
staff who will work on this project will be informed of this relationship and provided with contact 
information of a third party whom they can contact if they should have any questions or concerns about 
this relationship.  Further, the statisticians who will conduct all planned data analyses have no affiliation 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
53
with Square2 Systems, Inc.  Dr. Marsch has successfully employed similar procedures in prior NIH-
funded projects.   
15.8 
Clinical and Quality Monitoring 
Procedures in Place to Ensure the Validity and Integrity of the Data.  All research staff members involved 
in the collection of data will be well-trained on the administration of all assessments.   They will have 
initially observed other trained staff members administering the assessments, will then be observed while 
they administer assessments and may also watch training videos on administering certain assessments 
when available.   Additionally, completed assessments will be reviewed by our project Data Manager, and 
any errors detected in the administration of an assessment will be immediately brought to the attention of 
a study investigator and corrected.   
Procedures to guarantee the accuracy and completeness of the data, during data collection, entry, 
transmission and analysis.  Additionally, electronic tracking systems will be put into place to track when 
assessments are due for each participant and when they are completed.  Any overdue assessments will be 
identified by a research staff member within the same week they are due (or sooner depending on the 
assessment), and the research staff responsible for administration of the assessment will be notified to 
immediately complete the assessment with the appropriate participant.  Also, exhaustive contact 
information will be obtained for all participants, and updated regularly to help assure that we can 
successfully reach participants throughout the study and during follow-up.  The Data Manager and 
statistician(s) will further review all data to ensure their accuracy (e.g., no data out of range, no 
alphanumeric data, etc.).  We have successfully used similar procedures on other research projects. 
Investigators will host periodic visits by the Dartmouth Project Manager to ensure study procedures are 
conducted appropriately and that study data are generated, documented and reported in compliance with 
the protocol, GCP, and applicable regulations. These monitors will audit, at mutually agreed upon times, 
regulatory documents, informed consent forms and corresponding source documents for each participant. 
Monitors will have the opportunity and ability to review any study-associated document or file. 
Monitors will assess whether submitted data are accurate and in agreement with source documentation 
and will also review regulatory/essential documents such as correspondence with the IRB. Areas of 
particular concern will be participant informed consent forms, protocol adherence, reported safety events 
and corresponding assessments, and principal investigator oversight and involvement in the trial.  
Qualified personnel at Javeriana (Quality Assurance or QA monitors) will provide site management for 
each site during the trial. QA staff will audit source documentation, including informed consent forms. 
This will take place as specified by the local protocol team, PI or lead team and will occur as often as 
needed to help prevent, detect, and correct problems at the study sites. QA personnel will verify that study 
procedures are properly followed and that site personnel are trained and able to conduct the protocol 
appropriately. If the QA personnel’s review of study documentation indicates that additional training of 
site study personnel is needed, QA personnel will undertake or arrange for that training. 
For all data, range and consistency checks will be performed at regular intervals separately for each data 
source and, where relevant, consistency checks will be carried out. Any queries identified will be resolved 
promptly by the data management team, and the database will be updated, maintaining the audit trial. 
15.9 
Inclusion of Women and Minorities  
No Sex/Gender or Racial/Ethnic groups will be excluded.   Based on patient data at our partner sites, 
we expect that approximately 65% of the participants will be women.   And, we expect that 
approximately 60% will be white, 10% will be black, and 30% will be a mixture of other racial groups.  
We expect approximately 70% will be Hispanic. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
54
15.10 
Regulatory Files  
The regulatory files should contain all required regulatory documents, study-specific documents, and all 
important communications. Regulatory files will be checked at each participating site for regulatory 
document compliance prior to study initiation, throughout the study, as well as at study closure. 
15.11 
Records Retention and Requirements  
The regulatory files should contain all required regulatory documents, study-specific documents, and all 
important communications. Regulatory files will be checked at each participating site for regulatory 
document compliance prior to study initiation, throughout the study, as well as at study closure. 
15.12 
Reporting to Sponsor 
The site principal investigator agrees to submit accurate, complete, legible and timely reports to the 
Sponsor (NIMH), as required. These include, but are not limited to, reports of any changes that 
significantly affect the conduct or outcome of the trial or increase risk to study participants. Adverse 
Event reporting and Serious Adverse Event reporting will occur as described below (section 15.16). At 
the completion of the trial, the Lead Investigator will provide a final report to the NIMH, as required. 
15.13 
Audits  
The Sponsor (NIMH) has an obligation to ensure that this trial is conducted according to good research 
practice guidelines and may perform quality assurance audits for protocol compliance. The Lead 
Investigator and authorized research staff from Dartmouth, Javeriana, NIMH and other agencies such as 
the Department of Health and Human Services, the Office for Human Research Protection (OHRP) and 
the sites’ Institutional Review Board may inspect research records for verification of data, compliance 
with federal guidelines on human participant research, and to assess participant safety. 
15.14 
Study Documentation  
Study documentation includes all case report forms, workbooks, source documents, monitoring logs and 
appointment schedules, sponsor-investigator correspondence, and signed protocol and amendments, 
Ethics Review Committee or Institutional Review Committee correspondence and approved consent 
form and signed participant consent forms. 
 
Source documents include all recordings of observations or notations of clinical activities and all reports 
and records necessary for the evaluation and reconstruction of the clinical research study. Whenever 
possible, the original recording of an observation should be retained as the source document; however, a 
photocopy is acceptable provided that it is a clear, legible, and exact duplication of the original document. 
15.15 
Protocol Deviations  
Any departure from procedures and requirements outlined in the protocol will be classified as either a 
major or minor protocol deviation.  The difference between a major and minor protocol deviation has to 
do with the seriousness of the event and the corrective action required.  A minor protocol deviation is 
considered an action (or inaction) that by itself is not likely to affect the scientific soundness of the 
investigation or seriously affect the safety, rights, or welfare of a study participant.  Major protocol 
deviations are departures that may compromise the participant safety, participant rights, 
inclusion/exclusion criteria or the integrity of study data and could be cause for corrective actions if not 
rectified or prevented from re-occurrence.  Sites will be responsible for developing corrective action 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
55
plans for both major and minor deviations as appropriate.  Those corrective action plans may be 
reviewed/approved by the lead investigators with overall approval by the site’s IRB.  All protocol 
deviations will be monitored at each site for (1) significance, (2) frequency, and (3) impact on the study 
objectives, to ensure that site performance does not compromise the integrity of the trial.   
 
Additionally, each site is responsible for reviewing their local IRB’s definition of a protocol deviation or 
violation and understanding which events need to be reported. Sites must recognize that the IRB 
definition of a reportable event may differ and act accordingly in following all reporting requirements 
for both entities. 
15.16 
Safety Monitoring  
Principal Investigators’ Roles and Responsibilities for Data and Safety Monitoring.  The Principal 
Investigators will assume primary responsibility for overseeing all data and safety monitoring functions 
to ensure the safety of participants in the research proposed within this application and to ensure the 
validity and integrity of the data obtained in these studies.  The Principal Investigators will also regularly 
meet with the Co-Investigators and Consultants to track study progress and review these monitoring 
procedures.  The investigative team will regularly oversee all aspects of the study, including participant 
recruitment, Informed Consent, data collection, data management and data analysis procedures, as well 
as regularly assess the risk/benefit ratio associated with participation in the study.    
 
The investigative team will meet regularly (scheduled meetings once weekly) for the entire duration of 
the project. 
 
Data and Safety Monitoring Board (DSMB)  
An independent NIMH Data and Safety Monitoring Board (DSMB) will examine accumulating data to 
assure protection of participants’ safety and data integrity throughout the duration of the study. The 
DSMB conducts established biannual reviews of accumulating safety and efficacy data (an additional 
reviews as indicated). It will determine whether there is support for continuation of the trial, or evidence 
that study procedures should be changed, or if the trial should be halted, for reasons relating to the safety 
of the study participants, the efficacy of the treatment under study, or inadequate trial performance (e.g., 
poor recruitment) or concerns about data integrity. 
 
The study data will be reviewed by the NIMH Data and Safety Monitoring Board (DSMB) biannually. 
The DSMB will receive a data report from the study team on a schedule determined by the DSMB. The 
study team’s submission to the DSMB is expected to follow the established reporting format developed 
in consultation with NIMH collaborator(s). The report will include the major variables necessary for 
monitoring safety and quality of data collection and integrity of the study, including subject enrollment 
and retention. The DSMB will also review the study protocol and consents before the onset of the study, 
and will review amendments to these documents. Based on this review, the DSMB has the authority to 
prevent the study to start or to stop the study after it has started.  
 
The Principal Investigators will ensure all required DSMB reports are prepared, as required (as well as 
all required reporting to IRBs). 
 
 
Adverse Events (AEs)  

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
56
The Principal Investigators may appoint a Study Clinician (MD, PhD, or PI) for this study, who will 
review or provide consultation for each Adverse Event (AE) as needed. These reviews will include an 
assessment of whether the event was serious (Serious Adverse Event (SAE), whether the event was 
expected or unexpected, and the possible relatedness of the event to the study intervention or other study 
procedures. The Study Clinician will also provide advice for decisions to exclude, refer, or withdraw 
participants as required. This will include events that are serious, related and unexpected. The study staff 
will be trained to monitor for and report adverse events and Serious Adverse Events. 
Definition of Adverse Events and Serious Adverse Events 
 
An adverse event (AE) is any untoward or unfavorable medical occurrence in a human subject, whether 
or not considered study intervention related, which occurs during the conduct of a trial.  An adverse event 
can therefore include high risk of suicide, threat of harm toward self or others, or clinical deterioration 
as defined by worsened physical or mental health.   
 
Suspected adverse reaction is any adverse event for which there is a reasonable possibility that the study 
intervention caused the adverse event. A reasonable possibility implies that there is evidence that the 
study intervention caused the event. 
 
Adverse reaction is any adverse event caused by the study intervention. 
 
An adverse event, suspected adverse reaction, or adverse reaction is considered “serious” (i.e., a 
serious adverse event, serious suspected adverse reaction or serious adverse reaction) if, in the view of 
either the study medical clinician or sponsor, it: 
 
1) Results in death: A death occurring during the study or which comes to the attention of the 
study staff during the protocol-defined follow-up period, whether or not considered caused by 
the study intervention, must be reported. 
2) Is life-threatening: Life-threatening means that the study participant was, in the opinion of the 
medical clinician or sponsor, at immediate risk of death from the reaction as it occurred and 
required immediate intervention. Suicide attempts will be automatically classified as an SAE, 
given that they are life-threating.    
3) Requires inpatient hospitalization or prolongation of existing hospitalization.  
4) Results in persistent or significant incapacity or substantial disruption of the ability to conduct 
normal life functions.  
5) Is a congenital abnormality or birth defect. 
6) Important medical event that may not result in one of the above outcomes, but may jeopardize 
the health of the study participant or require medical or surgical intervention to prevent one of 
the outcomes listed in the above definition of serious event.  This may include threat of harm 
toward self or others. 
 
Definition of Expectedness 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
57
Any adverse event is considered “unexpected” if it is not consistent with the natural course of the mental 
health problems under study in this project. Therefore, an “unexpected event” is any event non-consistent 
with the natural course of depression or alcohol use disorder (for example, a stroke that is unrelated to 
either of these conditions). 
 
An “expected event” could include depressive symptoms, including suicidal ideation or attempt, heavy 
drinking, including hospitalizations due to heaving drinking, and other associated medical conditions). 
 
Site’s Role in Assessing Severity and Causality of Adverse Events 
Appropriately qualified and trained study personnel will conduct an initial assessment of seriousness, 
severity, and causality when eliciting participant reporting of adverse events. A study medical clinician 
will review reportable AEs for seriousness, severity, and causality on at least a weekly basis.  
 
Each of the sites has established practices for managing medical and psychiatric emergencies, and the 
study staff will continue to utilize these procedures. Treatment providers at each site will be responsible 
for monitoring participants for possible clinical deterioration or other problems, and for implementing 
appropriate courses of action. 
The Principal Investigators will train all project staff to recognize and report any adverse event 
immediately to them.   An adverse event involving human subjects can include high risk of suicide, threat 
of harm toward self or others, or clinical deterioration as defined by worsened physical or mental health.  
Other adverse events may also include the inadvertent disclosure by research staff of confidential research 
information to other persons and/or to staff of criminal justice or government agencies. 
 
Guidelines for Assessing Severity 
The severity of an adverse event refers to the intensity of the event: 
 
Grade 1 
Mild 
Transient or mild discomfort (typically < 48 hours), no or minimal 
medical intervention/therapy required, hospitalization not necessary 
(non-prescription or single-use prescription therapy may be 
employed to relieve symptoms)  
Grade 2 
Moderate 
Mild to moderate limitation in activity some assistance may be 
needed; no or minimal intervention/therapy required, hospitalization 
possible. 
Grade 3 
Severe 
Marked limitation in activity, some assistance usually required; 
medical intervention/ therapy required hospitalization possible. 
 
 
 
 
Guidelines for Determining Causality 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
58
The study medical clinician will use the following question when assessing causality of an adverse event 
to study intervention where an affirmative answer designates the event as a suspected adverse reaction: 
 
Is there a reasonable possibility that the study intervention caused the event?  
  
An example of an adverse event caused by the study intervention could be: a participant had a panic 
attack caused by their use of the application.    
 
 
* We are defining study participation as starting when a patient becomes a study participant, so the 
individual (patient or provider) in question would need to have consented and enrolled in the study. This 
means that screening is not included as “study participation.”  
 
In the event such adverse events are reported to the Principal Investigators, they will immediately communicate 
the event to the appropriate institutional official at the institutional review board (IRB) followed by a written report 
in accordance with the timeline outlined in the table below.  The institutional official with the IRB will then make 
a decision whether the reported event is a Serious Adverse Event (SAE) that must be reported to the appropriate 
Federal Agency.  A SAE may include: death, a life-threatening event, hospitalization, an incapacitating event, all 
congenital abnormalities or birth defects among research subjects, or an medical event that may not result in one 
of the above outcomes, but may jeopardize the health of the study participant or require medical or surgical 
intervention to prevent one of the outcomes listed in the above definition of serious event.  If necessary, the event 
will first be reported to the Federal Agency by telephone followed by a written report within three days. 
  
 
 
 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
59
A summary of reportable events, including their associated reporting requirements, is provided in the table below: 
 
Reportable Event 
When is Event Reported to:  
Reported By 
IRB/DSMB 
Suspensions 
or 
Terminations 
Research activities will be stopped 
immediately. 
Any 
suspension 
or 
termination of approval will include a 
statement of the reason(s) for the 
action and will be reported to 
NIMH DSMB/NIMH PO: within 3 
business days of receipt. 
IRBs (Dartmouth and corresponding 
in-country): within 3 business days of 
receipt 
Principal Investigator 
Death of a participant  
 
NIMH DSMB/NIMH PO: within 5 
business 
days 
of 
the 
Principal 
Investigator first learning of the death 
IRBs: within 1 business day of the 
Principal Investigator first learning of 
the death. 
Principal Investigator 
Unexpected Serious Adverse 
Events related to the study 
participation (except death for 
which see above) 
NIMH DSMB/NIMH PO: within 5 
business days of the study team 
becoming aware of the SAE 
IRBs: within 5 business days of the 
study team becoming aware of the SAE  
Principal Investigator 
Unanticipated Problems 
Involving Risks to Subjects or 
Others  
NIMH DSMB/NIMH PO: within 10 
business 
days of 
the 
investigator 
learning of the event 
IRBs: within 10 business days of the 
investigator learning of the event 
Principal Investigator 
Serious or Continuing 
Noncompliance with study 
protocol and human subjects 
regulations 
NIMH DSMB/NIMH PO: within 10 
business days  
IRBs: within 10 business days  
Principal investigator/Institution 
AEs that are deemed expected 
and/or unrelated to the study 
NIMH DSMB/NIMH PO: in the 
annual data report  
IRBs: at the time of continuing review 
Principal Investigator 
SAEs that are deemed expected 
and/or unrelated to the study  
NIMH DSMB/NIMH PO : in the bi-
annual data report  
Principal Investigator 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
60
IRBs: at the time of continuing review 
 
Protocol deviations: 
- Major (also known as 
protocol violations) 
 
 
-Minor 
NIMH DSMB/NIMH PO: within 10 
business days  
IRBs: within 10 business days  
 
NIMH DSMB/NIMH PO: in the 
annual data report 
IRBs: within continuing review 
Principal Investigator 
 
Trial Stopping Rules.  There are no a priori stopping rules in this study.  As previously described, the Principal 
Investigators will oversee all data and safety monitoring functions to ensure the safety of participants in this 
research project and to ensure the validity and integrity of the data obtained in these studies.  The Principal 
Investigators will also regularly meet with Co-investigators and Consultants to track study progress and review 
these monitoring procedures.  If at any point the investigative team determines that the risk of a given participant 
continuing in the study is greater than the benefit of participating, this individual’s participation will be 
discontinued and referred elsewhere as appropriate.  This participants’ study data will still be included in statistical 
analyses.  If there is evidence that the risk to participants in general is greater than the benefits to them, this team 
will halt the trial and appropriate action will be taken (e.g., protocol revised, participants referred elsewhere as 
appropriate).    
16.0 
POTENTIAL RISKS AND PROTECTIONS AGAINST RISK  
Potential Risks: The potential risks associated with the data collected are low for both patients 
and staff. 
 
Risks from assessments and Laddr intervention: The types of data we plan to collect from 
both patients and staff (assessment, self-report questionnaires) will not harm the subjects’ 
financial standing, employment, employability, or reputation, or expose the subject to civil or 
criminal liability. The types of risk associated with the data collected include possible fatigue, 
frustration, or the discussion of sensitive or personal information. If participants do not wish to 
answer any questions, they may elect not to do so, and may continue in the remainder of the 
study without penalty.   
 
Patient participants may also feel uncomfortable reporting on their mental health.     
Additionally, participants may be concerned about confidentiality risk when using the Internet 
to access Laddr®, even on secured, encrypted connections.   They may also be worried about 
prompts they receive on a mobile device designed to remind them to engage with the mobile 
intervention. 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
61
Participants may also find that some of the topics that Laddr addresses, including depression 
and/or alcohol use, may be sensitive topics. And they may be uncomfortable with some of this 
content.  It is also possible they may find that this content does not help them.   
 
 
Risks from electronic databases: There may be risks to subjects by virtue of their 
representation in electronic databases, principally involving the risk that privacy or 
confidentiality might be compromised if there were lapses in security of the information 
contained in these databases.  
 
Protection against risks from assessments and Laddr intervention: To protect against the 
possible risks associated with participant assessments, which are fatigue and frustration, it will 
be made clear to participants, both during the informed consent process that they are free to 
discontinue their participation at any point without penalty. Any participant that experiences 
significant discomfort during assessment will be able to discontinue participation. 
 
To protect against concerns that others may see when a participant receives a prompt from the 
Laddr system, the content of the prompts sent will be intentionally vague.   While they will be 
designed to be meaningful to individual participants, they will not include specific references to 
study participation. Any participant that experiences significant discomfort during use of Laddr 
will be able to discontinue participation. If a participant has any concerns at any time during this 
study, or if they experience any worsening of symptoms or an adverse event due to use of the 
Laddr mobile intervention, this participant can discontinue participation and they can contact 
their doctor and the research directors for this study, Dr. Sergio Castro, at 320-8320 ext. 2812 or 
by email at sergiomariocastro@gmail.com and/or Dr. Magda Cepeda at 57-301-362-1356 or by 
email at mccepedag@gmail.com.  
 
Protection against risks from electronic databases: To protect the privacy or confidentiality 
of subjects’ data stored in electronic databases, every effort will be made to safeguard the 
confidentiality of research records, using data files free of information enabling individual 
identification of subjects, lock-and-key access to paper records, and computer data files 
maintained with encryption, password protection, and behind firewalls. We will remove 
individual identifying information from data representations so that security failures would not 
put individual privacy and confidentiality at risk. Individual identifying information will only be 
maintained in a separate encrypted database with passwords known only to the PIs and specific 
members of the research team.  
 
We plan for the assessment batteries/tasks and resulting data to be available and coordinated 
through REDCap (Research Electronic Data Capture).  Participants’ completion of assessments 
will be completed via an encrypted Internet connection via 128-bit Secure Sockets Layer (SLL) 
-  the standard technology for securing eCommerce and eBanking transactions on the Internet.  
 
We will maintain Dartmouth IRB approval as well as IRB approval from Pontificia Universidad 
Javeriana and our study sites, as needed.  Our data management expert at Dartmouth (Mary Ann 
Greene) can serve as an expert advisor on data management systems broadly and REDCap 
specifically. 
 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
62
All usage data provided by participants when using the web-based Laddr® intervention will be 
stored on a secure server behind 2 firewalls and will not be accessible to anyone not affiliated 
with the research project.  Note that this mobile program will not collect data from participants 
about their suidical ideation or intent to self-harm.  This mobile system will track how often 
participants use the mobile system which will aid the research team in understanding 
participants’ usage of/engagement with the mobile intervention.  Also, all data stored on these 
servers will be coded by participant ID number and encrypted for security purposes using SLL.   
All of these data will also be transferred via an encrypted Internet connection to the Dartmouth 
REDCap site. 
 
While none of the procedures are associated with significant medical risk, we have plans for 
medical emergencies should they occur. Emergency telephone numbers and procedures are 
posted next to telephones in the primary care patient rooms.  In addition, primary care partners 
will be required to establish (if they don’t already have) a protocol for handling patients who 
communicate suicidal intent at any point during their participation. 
17.0 
ANTICIPATED BENEFIT 
17.1 
Potential Benefits of the Proposed Research to Human Subjects and Others 
Participants will be receiving a web-based mental health intervention, they may benefit from a 
psychosocial intevention that may help them reduce health risk behavior.  Healthcare systems 
will be receiving training and resources to embed science-based mental health care into primary 
care. 
17.2 
Importance of the Knowledge to be Gained 
Overall, this project will create new knowledge to inform unprecedented, science-based 
approaches to scaling-up mental health implementation research and building sustainable 
research capacity and science-based policies and programs in Latin America. This project brings 
together an outstanding expert team to test and refine an entirely new model for delivering 
widespread, science-based, mental health care in Latin America.  If successful, this approach 
can be expanded over time to embrace other areas of mental health (e.g., severe mental illness), 
chronic disease management, as well as health promotion prevention interventions based on 
community needs and priorities in Latin America.  This project may also serve as an important 
demonstration project to LMICs globally as they tackle the significant burden of mental health 
disorders and scale-up access to evidence-based models of mental health service delivery. 
18.0 
REFERENCES  
1. 
Institute for Health Metrics and Evaluation (IHME). GBD Compare. Seattle, WA: IHME, University of 
Washington;  [July 8, 2015]. Available from: http://vizhub.healthdata.org/gbd-compare/. 
2. 
Thornicroft G, Tansella M. What are the arguments for community-based mental health care?2003. 
Available from: http://www.euro.who.int/__data/assets/pdf_file/0019/74710/E82976.pdf. 
3. 
Bloom DE, Cafiero ET, Jané-Llopis E, Abrahams-Gessel S, Bloom LR, Fathima S, Feigl AB, Gaziano 
T, Mowafi M, Pandya A, Prettner K, Rosenberg L, Seligman B, Stein AZ, Weinstein C. The global economic 
burden of noncommunicable diseases Geneva: World Economic Forum; 2011. Available from: 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
63
http://www3.weforum.org/docs/WEF_Harvard_HE_GlobalEconomicBurdenNonCommunicableDiseases_2011.p
df. 
4. 
Collins PY, Patel V, Joestl SS, March D, Insel TR, Daar AS, Anderson W, Dhansay MA, Phillips A, 
Shurin S, Walport M, Ewart W, Savill SJ, Bordin IA, Costello EJ, Durkin M, Fairburn C, Glass RI, Hall W, 
Huang Y, Hyman SE, Jamison K, Kaaya S, Kapur S, Kleinman A, Ogunniyi A, Otero-Ojeda A, Poo MM, 
Ravindranath V, Sahakian BJ, Saxena S, Singer PA, Stein DJ. Grand challenges in global mental health. Nature. 
2011;475(7354):27-30. PMCID: PMC3173804. 
5. 
Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden 
implications: a systematic review and meta-analysis. JAMA Psychiatry. 2015;72(4):334-41. PMCID: 
PMC4461039. 
6. 
World Health Organization. Mental health systems in selected low- and middle-income countries: a 
WHO-AIMS cross-national analysis Geneva: WHO; 2009 [July 8, 2015]. Available from: 
http://whqlibdoc.who.int/publications/2009/9789241547741_eng.pdf. 
7. 
Bruckner TA, Scheffler RM, Shen G, Yoon J, Chisholm D, Morris J, Fulton BD, Dal Poz MR, Saxena S. 
The mental health workforce gap in low- and middle-income countries: a needs-based approach. Bull World 
Health Organ. 2011;89(3):184-94. PMCID: PMC3044251. 
8. 
Alarcon RD. Mental health and mental health care in Latin America. World Psychiatry. 2003;2(1):54-6. 
PMCID: PMC1525063. 
9. 
Okpaku SO.  Essentials of global mental health. New York: Cambridge University Press; 2014. p. 29. 
10. 
Ministerio de la Protección Social y Fundación FES Social. Estudio Nacional de Salud Mental Colombia 
Colombia2005 [July 8, 2015]. Available from: 
http://onsm.ces.edu.co/uploads/files/1243030_EstudioNacionalSM2003.pdf. 
11. 
World Health Organization. mhGAP: Mental Health Gap Action Programme : scaling up care for 
mental, neurological and substance use disorders Geneva: WHO; 2008 [July 8, 2015]. Available from: 
http://whqlibdoc.who.int/publications/2008/9789241596206_eng.pdf?ua=1. 
12. 
Pan American Health Organization. Strategy and plan of action on mental health. Washington, DC: Pan 
American Health Organization; 2009 [July 8, 2015]. Available from: 
http://www2.paho.org/hq/dmdocuments/2009/MENTAL_HEALTH.pdf. 
13. 
Colombia Ministry of Health. MinSalud presenta modelo de atención en salud para el Guainía. Bogata 
[July 10, 2015]. Available from: http://www.minsalud.gov.co/Paginas/MinSalud-presenta-modelo-de-
atenci%C3%B3n-en-salud-para-el-Guain%C3%ADa.aspx. 
14. 
Razzouk D, Gregorio G, Antunes R, Mari JD. Lessons learned in developing community mental health 
care in Latin American and Caribbean countries. World Psychiatry. 2012;11(3):191-5. PMCID: PMC3449352. 
15. 
Pan American Health Organization. Panama Consensus Washington, DC: Pan American Health 
Organization; 2010 [July 8, 2015]. Available from: 
http://www.paho.org/hq/index.php?option=com_content&view=category&id=1167&layout=blog&Itemid=353&
lang=en. 
16. 
Pan American Health Organization. Brasilia Consensus Washington, DC: Pan American Health 
Organization; 2013 [July 8, 2015]. Available from: 
http://www.paho.org/hq/index.php?option=com_content&view=category&id=1167&layout=blog&Itemid=353&
lang=en. 
17. 
McGovern MP, Urada D, Lambert-Harris C, Sullivan ST, Mazade NA. Development and initial 
feasibility of an organizational measure of behavioral health integration in medical care settings. J Subst Abuse 
Treat. 2012;43(4):402-9. 
18. 
Cook TD, Campbell DT. Quasi-experimentation: design & analysis issues for field settings. Boston: 
Houghton Mifflin; 1979. 
19. 
Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 
2006;6:54. PMCID: PMC1636652. 
20. 
Dallery J, Kurti A, Martner S. Technological approaches to assess and treat cigarette smoking. In: 
Marsch L, Lord S, Dallery J, editors. Behavioral health care and technology: using science-based innovations to 
transform practice. New York: Oxford University Press; 2014. p. 95-112. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
64
21. 
Marsch LA, Gustafson DH. The role of technology in health care innovation: a commentary. J Dual 
Diagn. 2013;9(1):101-3. PMCID: 3626450. 
22. 
Marsch LA, Dallery J. Advances in the psychosocial treatment of addiction: the role of technology in the 
delivery of evidence-based psychosocial treatment. Psychiatr Clin North Am. 2012;35(2):481-93. PMCID: 
3362794. 
23. 
Marsch LA. Leveraging technology to enhance addiction treatment and recovery. J Addict Dis. 
2012;31(3):313-8. PMCID: 3650640. 
24. 
Campbell ANC, Muench F, Nunes EV. Technology-based behavioral interventions for alcohol and drug 
use problems. In: Marsch LA, Lord SE, Dallery J, editors. Behavioral health care and technology: using science-
based innovations to transform practice. New York: Oxford University Press; 2014. p. 40-57. 
25. 
Zullig LH, Shaw RJ, Bosworth HB. Applying technology to medication management and adherence. In: 
Marsch LA, Lord SE, Dallery J, editors. Behavioral health care and technology: using science-based innovations 
to transform practice. New York: Oxford University Press; 2014. p. 81-94. 
26. 
Ben-Zeev D, Drake RE, Brian RM. Technologies for people with serious mental illness. In: Marsch LA, 
Lord SE, Dallery J, editors. Behavioral health care and technology: using science-based innovations to transform 
practice. New York: Oxford University Press; 2014. p. 70-80. 
27. 
Schueller SM, Chokshi M, Mohr DC. Using behavioral intervention technologies to reduce the burden of 
mood and anxiety disorders. In: Marsch LA, Lord SE, Dallery J, editors. Behavioral health care and technology: 
using science-based innovations to transform practice. New York: Oxford University Press; 2014. p. 58-69. 
28. 
Valle CG, Tate DF. Technology-based interventions to promote diet, exercise and weight control. In: 
Marsch LA, Lord SE, Dallery J, editors. Behavioral health care and technology: using science-based innovations 
to transform practice. New York: Oxford University Press; 2014. p. 113-38. 
29. 
Asch DA, Muller RW, Volpp KG. Automated hovering in health care--watching over the 5000 hours. N 
Engl J Med. 2012;367(1):1-3. 
30. 
Substance Use and Mental Health Services Administration. Results from the 2012 National Survey on 
Drug Use and Health: summary of national findings. NSDUH Series H-46, HHS Publication No. (SMA) 13-
4795 Rockville, MD.: Substance Abuse and Mental Health Services Administration; 2013. Available from: 
http://www.samhsa.gov/data/NSDUH/2012SummNatFindDetTables/NationalFindings/NSDUHresults2012.htm. 
31. 
Des Jarlais DC, Paone D, Milliken J, Turner CF, Miller H, Gribble J, Shi Q, Hagan H, Friedman SR. 
Audio-computer interviewing to measure risk behaviour for HIV among injecting drug users: a quasi-randomised 
trial. Lancet. 1999;353(9165):1657-61. 
32. 
Office of National Drug Control Policy. National drug control strategy Washington, D.C.2014 [June 21, 
2015]. Available from: http://www.whitehouse.gov/sites/default/files/ondcp/policy-and-research/ndcs_2014.pdf. 
33. 
Jones S, Fox S. Generations online in 2009: Pew Research Internet Project; 2009 [June 21, 2015]. 
Available from: http://www.pewinternet.org/2009/01/28/generations-online-in-2009/. 
34. 
Hopko DR, Robertson SMC, Lejuez CW. Behavioral activation for anxiety disorders. Behav Analyst 
Today. 2006;7(2):212-32. 
35. 
Lundervold DA, Talley C, Buermann M. Effect of Behavioral Activation Treatment on fibromyalgia-
related pain anxiety cognition. Int J Behav Consult Ther. 2006;2(1):73-84. 
36. 
Spates CR, Pagoto SL, Kalata A. A qualitative and quantitative review of behavioral activation treatment 
of major depressive disorder. Behav Analyst Today. 2006;7(4):508-21. 
37. 
Bell AC, D'Zurilla TJ. Problem-solving therapy for depression: a meta-analysis. Clin Psychol Rev. 
2009;29(4):348-53. 
38. 
Okorodudu DE, Bosworth HB, Corsino L. Innovative interventions to promote behavioral change in 
overweight or obese individuals: A review of the literature. Ann Med. 2014:1-7. 
39. 
Simon SS, Cordas TA, Bottino CM. Cognitive Behavioral Therapies in older adults with depression and 
cognitive deficits: a systematic review. Int J Geriatr Psychiatry. 2015;30(3):223-33. 
40. 
Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face 
cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World 
Psychiatry. 2014;13(3):288-95. PMCID: PMC4219070. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
65
41. 
Smith JE, Meyers RJ, Miller WR. The community reinforcement approach to the treatment of substance 
use disorders. Am J Addict. 2001;10 Suppl:51-9. 
42. 
Meredith SE, Jarvis BP, Raiff BR, Rojewski AM, Kurti A, Cassidy RN, Erb P, Sy JR, Dallery J. The 
ABCs of incentive-based treatment in health care: a behavior analytic framework to inform research and 
practice. Psychol Res Behav Manag. 2014;7:103-14. PMCID: PMC3964160. 
43. 
Meredith SE, Dallery J. Investigating group contingencies to promote brief abstinence from cigarette 
smoking. Exp Clin Psychopharmacol. 2013;21(2):144-54. PMCID: PMC3657835. 
44. 
Laranjo L, Arguel A, Neves AL, Gallagher AM, Kaplan R, Mortimer N, Mendes GA, Lau AY. The 
influence of social networking sites on health behavior change: a systematic review and meta-analysis. J Am 
Med Inform Assoc. 2015;22(1):243-56. 
45. 
Maher CA, Lewis LK, Ferrar K, Marshall S, De Bourdeaudhuij I, Vandelanotte C. Are health behavior 
change interventions that use online social networks effective? A systematic review. J Med Internet Res. 
2014;16(2):e40. PMCID: PMC3936265. 
46. 
Berman MI, et al. Therapeutic alliance and depression outcomes with interactive multimedia electronic 
problem solving therapy. Behav Ther. Forthcoming. 
47. 
Arean P, Hegel M, Vannoy S, Fan MY, Unuzter J. Effectiveness of problem-solving therapy for older, 
primary care patients with depression: results from the IMPACT project. Gerontologist. 2008;48(3):311-23. 
48. 
Bickel WK, Marsch LA, Buchhalter AR, Badger GJ. Computerized behavior therapy for opioid-
dependent outpatients: a randomized controlled trial. Exp Clin Psychopharmacol. 2008;16(2):132-43. PMCID: 
2746734. 
49. 
Marsch LA, Guarino H, Acosta M, Aponte-Melendez Y, Cleland C, Grabinski M, Brady R, Edwards J. 
Web-based behavioral treatment for substance use disorders as a partial replacement of standard methadone 
maintenance treatment. J Subst Abuse Treat. 2014;46(1):43-51. PMCID: 3839618. 
50. 
Campbell AN, Nunes EV, Matthews AG, Stitzer M, Miele GM, Polsky D, Turrigiano E, Walters S, 
McClure EA, Kyle TL, Wahle A, Van Veldhuisen P, Goldman B, Babcock D, Stabile PQ, Winhusen T, Ghitza 
UE. Internet-delivered treatment for substance abuse: a multisite randomized controlled trial. Am J Psychiatry. 
2014;171(6):683-90. PMCID: PMC4079279. 
51. 
Kim S, Marsch LA, Guarino H, Acosta M. The effectiveness of web-based treatment for persons with 
extensive histories of substance abuse treatment involvement. Addiction. (under review). 
52. 
Kim S, Marsch LA, Guarino H, Acosta M. Predictors of outcome from web-based treatment for 
substance use disorders:  Results from a randomized, clinical trial. Drug Alcohol Depend. (Under review). 
53. 
Araya R, Flynn T, Rojas G, Fritsch R, Simon G. Cost-effectiveness of a primary care treatment program 
for depression in low-income women in Santiago, Chile. Am J Psychiatry. 2006;163(8):1379-87. 
54. 
Gómez-Restrepo C, Peñaranda AB, Benavides MO, Laverde JFG, Tamayo N, Sánchez DR. [Health 
professionals' accuracy in depression diagnosis after an educational intervention in primary  level. Colombian 
results of the International Depression Project]. Revista Colombiana de Psiquiatría. 2007;36(3):439-50. 
55. 
Riveros Vargas M, Bohórquez Peñaranda A, Gómez-Restrepo C, Okuda Benavides M. [Patient’s 
knowledge about depression among patients treated in primary care settings in Bogota, Colombia.   Colombian 
results of the International Depression Project.]. Revista Colombiana de Psiquiatría. 2006;35:9-22. 
56. 
Gómez-Restrepo C, Bohórquez Peñaranda A, Gil Laverde JF, Pérez Muñoz V. [Knowledge of 
depressive disorders and evaluation of an educational intervention applied to primary care physicians in Bogotá, 
Colombia. Colombian results of the IDP.]. Revista Colombiana de Psiquiatría. 2005;34:220-41. 
57. 
Gomez-Restrepo C, Tamayo-Martinez N, Ayala-Corredor C, Carrasco-Gil A, Padilla-Munoz A, Garzon 
DF, Borda JP, Verhelst P, Ramirez MP, Gonzalez LM, Gil JF. [The knowledge acquired during and adherence to 
a postgraduate course concerned with mental health aimed at school counselors]. Rev Salud Publica (Bogota). 
2014;16(6):821-33. 
58. 
Rojas G, Fritsch R, Solis J, Jadresic E, Castillo C, Gonzalez M, Guajardo V, Lewis G, Peters TJ, Araya 
R. Treatment of postnatal depression in low-income mothers in primary-care clinics in Santiago, Chile: a 
randomised controlled trial. Lancet. 2007;370(9599):1629-37. 
59. 
Araya R, Rojas G, Fritsch R, Gaete J, Rojas M, Simon G, Peters TJ. Treating depression in primary care 
in low-income women in Santiago, Chile: a randomised controlled trial. Lancet. 2003;361(9362):995-1000. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
66
60. 
REDCap: Research Electronic Data Capture Nashville: Vanderbilt University; 2015 [July 8, 2015]. 
61. 
Ayers LR, Beyea SC, Godfrey MM, Harper DC, Nelson EC, Batalden PB. Quality improvement 
learning collaboratives. Qual Manag Health Care. 2005;14(4):234-47. 
62. 
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen 
Intern Med. 2001;16(9):606-13. PMCID: PMC1495268. 
63. 
Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use 
Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful 
alcohol consumption--II. Addiction. 1993;88(6):791-804. 
64. 
Whooley MA, Avins AL, Miranda J, Browner WS. Case-finding instruments for depression. Two 
questions are as good as many. J Gen Intern Med. 1997;12(7):439-45. PMCID: PMC1497134. 
65. 
Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption 
questions (AUDIT-C): an effective brief screening test for problem drinking. Archives of Internal Medicine. 
1998;158(16):1789-95. 
66. 
[President Santos introduced Vive Digital Plan 2014-2018 in 2014 ANDICOM] Bogata: MINTIC; 2014 
[July 8, 2015]. Available from: http://www.mintic.gov.co/portal/604/w3-article-7080.html. 
67. 
Kazdin AE. Single-case research designs : methods for clinical and applied settings. New York: Oxford 
University Press; 2011. 
68. 
Biglan A, Ary D, Wagenaar AC. The value of interrupted time-series experiments for community 
intervention research. Prev Sci. 2000;1(1):31-49. PMCID: PMC4553062. 
69. 
Morgan DL, Morgan RK. Single-participant research design. Bringing science to managed care. Am 
Psychol. 2001;56(2):119-27. 
70. 
Kearns KP. Flexibility of single-subject experimental designs. Part II: Design selection and arrangement 
of experimental phases. J Speech Hear Disord. 1986;51(3):204-14. 
71. 
Barlow DH, Hersen M. Single case experimental designs strategies for studying behavior change. New 
York; Oxford; Toronto; Sydney; Paris; Frankfurt [Main i.e.] Kronberg-Taunus: Pergamon Press; 1984. 
72. 
Damschroder LJ, Hagedorn HJ. A guiding framework and approach for implementation research in 
substance use disorders treatment. Psychol Addict Behav. 2011;25(2):194-205. 
73. 
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes 
for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy 
Ment Health. 2011;38(2):65-76. PMCID: 3068522. 
74. 
Estabrooks PA, Boyle M, Emmons KM, Glasgow RE, Hesse BW, Kaplan RM, Krist AH, Moser RP, 
Taylor MV. Harmonized patient-reported data elements in the electronic health record: supporting meaningful 
use by primary care action on health behaviors and key psychosocial factors. J Am Med Inform Assoc. 
2012;19(4):575-82. PMCID: PMC3384114. 
75. 
Tai B, Sparenborg S, Ghitza UE, Liu D. Expanding the National Drug Abuse Treatment Clinical Trials 
Network to address the management of substance use disorders in general medical settings. Subst Abuse Rehabil. 
2014;5:75-80. PMCID: PMC4114899. 
76. 
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of 
health services research findings into practice: a consolidated framework for advancing implementation science. 
Implement Sci. 2009;4:50. PMCID: 2736161. 
77. 
Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a 
systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 
2013;8:22. PMCID: PMC3598720. 
78. 
Rabin BA, Purcell P, Naveed S, Moser RP, Henton MD, Proctor EK, Brownson RC, Glasgow RE. 
Advancing the application, quality and harmonization of implementation science measures. Implement Sci. 
2012;7:119. PMCID: PMC3541131. 
79. 
Society for Implementation Research Collaboration. Welcome to SIRC  [July 8, 2015]. Available from: 
https://www.societyforimplementationresearchcollaboration.org/. 
80. 
Glasgow RE, Phillips SM, Sanchez MA. Implementation science approaches for integrating eHealth 
research into practice and policy. Int J Med Inform. 2014;83(7):e1-11. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
67
81. 
Murray LK, Tol W, Jordans M, Sabir G, Amin AM, Bolton P, Bass J, Bonilla-Escobar FJ, Thornicroft 
G. Dissemination and implementation of evidence based, mental health interventions in post conflict, low 
resource settings. Intervention. 2014;12:94-112. 
82. 
Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability 
Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184. PMCID: 
PMC3900326. 
83. 
Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Using the Program 
Sustainability Assessment Tool to assess and plan for sustainability. Prev Chronic Dis. 2014;11:130185. 
PMCID: PMC3900327. 
84. 
Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, Bunger AC. Public health 
program capacity for sustainability: a new framework. Implement Sci. 2013;8:15. PMCID: PMC3599102. 
85. 
Lewis CC, Stanick CF, Martinez RG, Weiner BJ, Kim M, Barwick M, Comtois KA. The Society for 
Implementation Research Collaboration Instrument Review Project: a methodology to promote rigorous 
evaluation. Implement Sci. 2015;10:2. PMCID: PMC4308900. 
86. 
Haroz EE, Bass JK, Lee C, Murray LK, Robinson C, Bolton P. Adaptation and testing of psychosocial 
assessment instruments for cross-cultural use: an example from the Thailand Burma border. BMC Psychol. 
2014;2(1):31. PMCID: PMC4317135. 
87. 
Bolton P. Cross-cultural validity and reliability testing of a standard psychiatric assessment instrument 
without a gold standard. J Nerv Ment Dis. 2001;189(4):238-42. 
88. 
Bolton P, Tang AM. An alternative approach to cross-cultural function assessment. Soc Psychiatry 
Psychiatr Epidemiol. 2002;37(11):537-43. 
89. 
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion 
interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322-7. PMCID: PMC1508772. 
90. 
Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice 
implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4-23. PMCID: PMC3025110. 
91. 
Kaplan RS, Porter ME. How to solve the cost crisis in health care. Harv Bus Rev. 2011;89(9):46-52, 4, 
6-61 passim. 
92. 
McBain RK, Jerome G, Warsh J, Browning M, Mistry B, Faure PAI, Pierre C, Fang AP, Mugunga JC, 
Rhatigan J, Leandre F, Kaplan R. Rethinking the cost of healthcare in low-resource settings: the value of time-
driven activity-based costing. BMJ Global Health. 2016;1(3). 
93. 
Masís DP, Gómez-Restrepo C, Restrepo MU, Miranda C, Pérez A, de la Espriella M, Novoa J, Chaux A, 
Arenas Á, Torres N, Suárez M, Rondón M. La carga económica de la depresión en Colombia: costos directos del 
manejo intrahospitalario. Revista Colombiana de Psiquiatría. 
94. 
Kessler RC, Ames M, Hymel PA, Loeppke R, McKenas DK, Richling DE. Using the World Health 
Organization Health and Work Performance Questionnaire to evaluate the indirect workplace costs of illness. J 
Occup Environ Med. 2004;46 (Supplement):S23-37. 
95. 
Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of 
current depression in the general population. J Affect Disord. 2009;114(1-3):163-73. 
96. 
Dum, M., Sobell, L. C., Sobell, M. B., Heinecke, N., Voluse, A., & Johnson, K. A Quick Drinking Screen 
for identifying women at risk for an alcohol-exposed pregnancy. Addicitve Behaviors. 2009:34(9): 714-716.  
97. 
12-Item Short Form Survey from the RAND Medical Outcomes Study Santa Monica, CA: Rand Health;  
[July 8, 2015]. Available from: http://www.rand.org/health/surveys_tools/mos/mos_core_12item.html. 
98. 
Ustun TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, Saxena S, von Korff M, Pull 
C. Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. 
2010;88(11):815-23. PMCID: PMC2971503. 
99. 
Chibanda D, Weiss HA, Verhey R, Simms V, Munjoma R, Rusakaniko S, Chingono A, Munetsi E, Bere 
T, Manda E, Abas M, Araya R. Effect of a primary care-based psychological intervention on symptoms of 
common mental disorders in Zimbabwe: A randomized clinical trial. JAMA. 2016;316(24):2618-26. 
100. 
Nadkarni A, Weobong B, Weiss HA, McCambridge J, Bhat B, Katti B, Murthy P, King M, McDaid D, 
Park AL, Wilson GT, Kirkwood B, Fairburn CG, Velleman R, Patel V. Counselling for Alcohol Problems 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
68
(CAP), a lay counsellor-delivered brief psychological treatment for harmful drinking in men, in primary care in 
India: A randomised controlled trial. The Lancet. 2017;389(10065):186-95. PMCID: PMC5236065. 
101. 
Rahman A, Hamdani SU, Awan NR, Bryant RA, Dawson KS, Khan MF, Azeemi MM, Akhtar P, Nazir 
H, Chiumento A, Sijbrandij M, Wang D, Farooq S, van Ommeren M. Effect of a multicomponent behavioral 
intervention in adults impaired by psychological distress in a conflict-affected area of Pakistan: A randomized 
clinical trial. JAMA. 2016;316(24):2609-17. 
102. 
Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin 
Trials. 2007;28(2):182-91. 
103. 
Rabin R, Charro Fd. EQ-SD: a measure of health status from the EuroQol Group. Annals of medicine. 
2001;33(5):337-43. 
104. 
Zarate V, Kind P, Chuang L-H. Hispanic valuation of the EQ-5D health states: a social value set for 
Latin Americans. Value in Health. 2008;11(7):1170-7. 
105. 
Augustovski FA, Irazola VE, Velazquez AP, Gibbons L, Craig BM. Argentine valuation of the EQ-5D 
health states. Value in Health. 2009;12(4):587-96. 
106. 
Husain N, Chaudhry N, Fatima B, Husain M, Amin R, Chaudhry IB, Rahman RU, Tomenson B, Jafri F, 
Naeem F. Antidepressant and group psychosocial treatment for depression: a rater blind exploratory RCT from a 
low income country. Behavioural and cognitive psychotherapy. 2014;42(6):693-705. 
107. 
Kessler RC, Angermeyer M, Anthony JC, De Graaf R, Demyttenaere K, Gasquet I, De Girolamo G, 
Gluzman S, Gureje O, Haro JM. Lifetime prevalence and age-of-onset distributions of mental disorders in the 
World Health Organization's World Mental Health Survey Initiative. World Psychiatry. 2007;6(3):168. 
108. 
Baxter A, Scott K, Vos T, Whiteford H. Global prevalence of anxiety disorders: a systematic review and 
meta-regression. Psychological medicine. 2013;43(5):897-910. 
109. 
Gómez-Restrepo C, Tamayo MN, Bohórquez A, Rondón M, Medina RM, Rengifo H, Bautisa N. 
Depression and anxiety disorders and associated factors in the adult Colombian population, 2015 National 
Mental Health Survey. Revista Colombiana de Psiquiatría. 2016;45:58-67. 
110. 
Burns L, Teesson M. Alcohol use disorders comorbid with anxiety, depression and drug use disorders: 
Findings from the Australian National Survey of Mental Health and Well Being. Drug and alcohol dependence. 
2002;68(3):299-307. 
111. 
Kessler RC, Nelson CB, McGonagle KA, Edlund MJ, Frank RG, Leaf PJ. The epidemiology of co-
occurring addictive and mental disorders: implications for prevention and service utilization. American Journal 
of Orthopsychiatry. 1996;66(1):17. 
112. 
DiMatteo MR, Lepper HS, Croghan TW. Depression is a risk factor for noncompliance with medical 
treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Archives of internal 
medicine. 2000;160(14):2101-7. 
113. 
Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety 
disorder: the GAD-7. Archives of internal medicine. 2006;166(10):1092-7. 
114. 
Londoño A, Romero P, Casas G. The association between armed conflict, violence and mental health: a 
cross sectional study comparing two populations in Cundinamarca department, Colombia. Conflict and health. 
2012;6(1):12. 
115. 
Cheung MW, Chan W. Meta-analytic structural equation modeling: a two-stage approach. Psychol 
Methods. 2005;10(1):40-64. 
116. 
Cheung MWL, Chan W. A two-stage approach to synthesizing covariance matrices in meta-analytic 
structural equation modeling. Struct Equ Modeling. 2009;16(1):28-53. 
117. 
MacKinnon DP. Introduction to statistical mediation analysis. Mahwah, NJ: Lawrence Erlbaum 
Associates; 2008. 
118. 
Davison AC, Hinkley DV. Bootstrap methods and their application. New York: Cambridge University 
Press; 1997. 
119. 
MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V. A comparison of methods to test 
mediation and other intervening variable effects. Psychol Methods. 2002;7(1):83-104. PMCID: PMC2819363. 
120. 
Mackinnon DP, Lockwood CM, Williams J. Confidence limits for the indirect effect: distribution of the 
product and resampling methods. Multivariate Behav Res. 2004;39(1):99. PMCID: PMC2821115. 

5U19MH109988-02 
   Implementation Research Protocol 
Scaling Up Science-based Mental Health Interventions in Latin America 
      Version 15.4            April 14, 2021 
 
 
69
121. 
Shrout PE, Bolger N. Mediation in experimental and nonexperimental studies: new procedures and 
recommendations. Psychol Methods. 2002;7(4):422-45. 
122. 
Bollen KA. Structural equations with latent variables. New York: Wiley; 1989. 
123. 
Muthen BO. Beyond SEM: general latent variable modeling. Behaviormetrika. 2002;29(1):81-117. 
124. 
Skrondal A, Rabe-Hesketh S. Generalized latent variable modeling: multilevel, longitudinal, and 
structural equation models. Boca Raton: Chapman & Hall/CRC; 2004. 
125. 
Hoyle RH. Confirmatory factor analysis. In: Tinsley HEA, Brown SD, editors. Handbook of applied 
multivariate statistics and mathematical modeling. San Diego: Academic Press; 2000. p. 465-97. 
126. 
Hambleton RH, Swaminathan H, Rogers HJ. Fundamentals of item response theory. Newbury Park, CA: 
Sage Publications; 1991. 
19.0 
 APPENDIX 
• 
Informed Consent to Participate in Implementation Study – Patient Form 
• 
Informed Consent to Participate in Implementation Study – Staff Form 
• 
Informed Consent to Participate in Qualitative Interviews – Patient Form 
• 
Informed Consent to Participate in Qualitative Interviews – Staff Form 
• 
Adverse Event/Serious Adverse Event (AE/SAE) Report Form