Improving Measurement-Based Care in Youth Mental Health: A Comparison of Unidimensional and Multidimensional Approaches
Phase: NA
Conditions: Mental Disorder in Adolescence
Sponsor: University of Miami

 
 
Page 1 of 30 
  
Official Title: Personalized Assessment of Client Experiences (PACE) 
Principal Investigator Name: Amanda Jensen-Doss 
Version Number: 20 
Version Date: 05/20/24 
NCT Number: NCT04369924 
 
 
 

 
 
Page 2 of 30 
  
1) Protocol Title 
 
Improving Measurement-Based Care in Youth Mental Health: A Comparison of Unidimensional 
and Multidimensional Approaches 
 
2) IRB Review History* 
 
N/A 
 
3) Objectives* 
 
This pilot study will use a mechanism-driven approach to compare two models of MBC in 
youth-serving community mental health settings: Multidimensional MBC, which includes 
measures of multiple domains that are patient-centered, treatment-driven, and symptom-focused, 
and Unidimensional MBC, which includes symptom measures only. Multidimensional MBC 
provides feedback on a broader array of process and progress outcomes that may be more 
relevant to clients and provide more actionable feedback to clinicians, and thus should have a 
stronger impact on target mechanisms and client outcomes. However, unidimensional MBC is 
lower burden for patients and providers, so might have more implementation success. The study 
will employ a Type 1 Effectiveness Implementation Hybrid design to assess the comparative 
effectiveness of these two approaches while gathering data to inform future implementation. 
Analyses will account for whether or not a client is on-track or not-on-track for improved 
outcomes, a likely moderator of MBC’s effect on outcomes. 
 
Aim 1: Conduct preliminary development activities to ensure fit between the two MBC 
approaches, the measurement of mechanisms, and the clinical context.  
Aim 1a: Conduct stakeholder individual and group interviews to gather feedback on 
the two MBC approaches, implementation support strategies, and proposed measures of 
MBC mechanisms.  
Aim 1b: Develop algorithms to determine on-track (OT) or not-on-track (NOT) status 
and program the MBC platform to administer study measures and randomize participants. 
Aim 1c: Conduct a pre-pilot of study procedures. 
 
Aim 2: Conduct a pilot randomized control trial (N = 900 youth clients, with an n = 400 
research subsample) comparing multidimensional to unidimensional MBC.  
Aim 2a: Test the predictive validity of the OT/NOT algorithms with outcome status at 
treatment end. 
Aim 2b: Conduct an enrollment and attrition analysis to investigate potential factors 
associated with condition, client, or clinician that could influence a future RCT. 
Aim 2c: Compare the effects of multidimensional and unidimensional MBC on 
proposed MBC mechanisms and youth outcomes. 
Aim 2d: Examine whether the effects of MBC condition on outcomes are mediated by 
MBC mechanisms and/or moderated by OT/NOT status, diagnosis, and/or treatment type.  
 
Aim 3: Conduct a mixed-methods analysis of implementation factors related to 
multidimensional or unidimensional MBC. 
Aim 3a: Examine whether fidelity to MBC differs by condition.  

 
 
Page 3 of 30 
  
Aim 3b: Examine whether implementation outcomes and challenges differ by MBC 
condition.  
 
4) Background* 
 
Approximately 14 to 20% of youth experience a mental disorder annually1, with an estimated 
cost of $247 billion dollars per year in the United States alone2. Youth mental health concerns 
are often chronic and related to functional impairments including decreased educational 
attainment, juvenile justice involvement, and substance abuse1,3-7. Data consistently point to a 
“quality chasm” between routine mental health services and evidence-based practices (EBPs)3,8. 
Although numerous evidence-based treatments (EBTs) have been developed to address these 
concerns, results of effectiveness trials examining their performance in “as usual” clinical 
settings have yielded mixed results, and only small benefits over usual care on average9.  
An approach to improving healthcare that holds great potential is Measurement Based 
Care (MBC). MBC is grounded in the frequent and systematic use of assessment to regularly 
track the processes and outcomes of care10 with feedback provided to clinicians11,12. MBC has 
support as an EBP, with several meta-analyses13-18 and systematic reviews19,20 indicating an 
positive effect on patient outcomes, particularly in adult samples. As systematic quality 
measurement has been identified as a key component of organizational improvement by the 
Institute of Medicine8, MBC also supports data-informed decision-making for quality 
improvement21, paving the way for data-driven implementation of evidence-based practices. For 
ease of use and real-time data availability, MBC systems typically utilize technology for online 
measure administration and generation of feedback reports.  
MBC addresses many of the limitations that have restricted the reach of many other 
youth mental health practices. First, many EBPs are designed to address single diagnoses or 
clusters similar disorders, whereas comorbidity rates are high in youth psychopathology.15 This 
single-problem focus may decrease real-world effectiveness, and make EBPs less appealing to 
clinicians, who are concerned that they are not relevant to their clients25,26. A second, related 
problem is that clinicians face a significant training burden to become competent in the many 
single-problem EBPs that would be necessary to cover their caseloads.28 Third, surveys of 
clinicians indicate they perceive many EBPs as too rigid for personalized treatment25,29, whereas 
their attitudes toward MBC and other EBPs that cut across multiple diagnostic groups are 
positive22,23. MBC is a transdiagnostic intervention that supports personalized treatment across 
an agency’s full caseload, increasing its utility and appeal. MBC can be leveraged to inform 
clinician utilization of a range of interventions, including being embedded as “clinical 
dashboards” to guide use of other EBPs24.  
A significant barrier to advancing the use of MBC is a lack of research that tests the 
mechanisms underlying effective MBC.  MBC is founded upon the premise that high-quality 
and continuous feedback will improve clinician competency, enhancing client outcomes53. In 
typical care, clinicians receive limited standardized information client outcomes51,52, often make 
inaccurate appraisals of their own competence54, and hence fail to make adjustments to their 
practice when needed. MBC alerts clinicians to clients who are not responding to treatment, 
prompting them to change therapeutic strategies55-57. Clinicians may also share feedback results 
with clients, which could increase client engagement58,59 and enhance therapeutic alliance60,61. 
One study has shown a link between problems identified in feedback reports and content 
addressed in session25, suggesting that MBC feedback can influence clinician behavior. 

 
 
Page 4 of 30 
  
However, the underlying mechanisms of action of MBC remain largely theoretical.  Research is 
needed that explicitly investigates MBC mechanisms in order to enhance MBC effectiveness and 
improve implementation support. 
Little is known about the essential components of MBC. Models of MBC vary in a 
number of ways, including the frequency and content of assessment, the format and target of 
feedback, etc. Efforts to draw conclusions about these components have been hampered by 
differences between studies, and there is a need for studies to directly compare different forms of 
MBC20. One key issue that has cascading effects across the entire MBC process is what 
assessment data to gather. Current approaches can be grouped into unidimensional approaches 
that track symptoms26-36 and multidimensional approaches that also track therapy processes such 
as therapeutic alliance37,38. Because multidimensional MBC focuses on both treatment progress 
and processes, it leads to feedback that is more actionable than unidimensional approaches, 
which only provide feedback on symptoms. Actionable feedback is key to MBC success19,39-41; 
multidimensional MBC may therefore be more effective. It also recognizes patient and caregiver 
preference for measures that assess positive aspects of treatment progress42-46.  However, 
multidimensional MBC may have higher burden for clients and clinicians than unidimensional 
MBC, which could have implications for fidelity and sustainability.  
Systematic reviews and meta-analyses of MBC have not discriminated between these 
approaches13-20,47. However, individual randomized controlled trials (RCTs) have supported both 
approaches. Multidimensional MBC has been shown to be more effective than controls, with 
small-to-medium effects in adults48-60 and small effects in youth37,38.  Unidimensional approaches 
have small effects in adult samples26,27,30-36, but the only youth study found no effect of feedback. 
Further, MBC effectiveness may be moderated by whether a client is on-track (OT) or not-on-
track (NOT) for improved outcomes, with MBC found to be most beneficial for clients flagged 
as NOT51,59,60. The only direct comparisons for multidimensional and unidimensional MBC have 
occurred within the NOT group. Three studies have examined the effects of adding a single 
administration of a multidimensional MBC battery after a unidimensional system flags an adult 
client as NOT51,59,60, finding improved outcomes for the group that received the 
multidimensional feedback (d = .3151).   
MBC implementation is complex21,61-65, involving changes at the client (measure 
completion), clinician (feedback viewing and utilization), and organizational (e.g., training, 
implementing new technology) levels. This complexity likely contributes to MBC being 
implemented with high quality in fewer than 20% of clinical settings61. Extant literature on MBC 
implementation has primarily focused on client- and clinician-reported barriers22,63,66,67, with 
implementation supports often focused on these barries22,68-70. While these are important, 
organizational-level factors, such as information flow matched to structural and workflow 
processes71, have been largely neglected. A recent review72-74, guided by the Consolidated 
Framework for Implementation Research (CFIR)75,76, suggests that implementation is influenced 
by five factors72: intervention characteristics, the extra-organizational setting, the inner-
organizational setting, individual stakeholder characteristics, and process factors (e.g., planning, 
evaluating). Given that implementation burden may be lower in unidimensional MBC, yet 
usefulness and acceptability may be higher for multidimensional MBC, a comprehensive 
implementation approach is necessary. PI Douglas found that comprehensive MBC 
implementation support guided by the CFIR was associated with successful implementation and 
improved outcomes at one agency versus an implementation failure at a second agency37. A key 
priority is therefore implementation research focused on factors that influence MBC adoption 

 
 
Page 5 of 30 
  
and sustainability over time77. 
In sum, compared to treatment as usual, MBC has small-to-medium effect sizes, with larger 
effects for clients who are NOT, and preliminary data from adult samples suggest that 
multidimensional approaches may be more effective than unidimensional MBC, particularly 
within the NOT group (e.g., d = .3151). Although the effects of MBC are somewhat modest, its 
transdiagnostic reach is broad and it can also facilitate the implementation of other EBPs with 
larger effect sizes24. However, despite its potential, research on underlying MBC mechanisms, 
essential components, and effective implementation is minimal. This R34 addresses these gaps 
by developing procedures, mechanism measures, and implementation strategies to support a 
future R01 to study the effects of unidimensional MBC and multidimensional MBC on youth 
outcomes.  
 
5) Inclusion and Exclusion Criteria* 
 
The study will involve three groups of participants: 1) Clinicians; 2) Agency Leaders; and 3) 
Youth. Clinician Inclusion Criteria: (1) Clinician will be at least a part-time employee 
providing psychosocial treatment at the study agencies. (2) Clinician may conduct sessions 
in English or Spanish, but must be able to speak, read and understand English well enough 
to participate in English-language interviews, trainings, and consultation calls. Agency 
Leader Inclusion Criteria: (1) Agency leaders will be individuals in leadership positions 
(e.g., CEO, clinical director) at the study agencies. (2) Must be able to speak, read and 
understand English well enough to participate in English-language interviews. Youth 
Inclusion Criteria: (1) Male or female youth between the ages of 11-17 years receiving 
mental health services in the outpatient, intensive outpatient, and/or intensive home-based 
services at the participating clinics; the research subsample will be restricted to ages 11-16 
to avoid having to re-consent individuals who turn 18 during the study. (2) If the family 
consents to complete additional research measures, one parent and/or primary caregiver 
must be available and willing to participate in all study assessments. (3) Adolescent and at 
least one parent/guardian are able to complete all study procedures in English or Spanish. 
For the Aim 1 pre-pilot, the adolescent and guardian must be able to complete all study 
procedures in English; this decision was made so that study materials could be finalized 
prior to translating them into Spanish.  
 
6) Number of Subjects* 
 
Up to 15 agency leaders, up to 200 clinicians, and up to 2000 youth will take part in the study.  
 
7) Study-Wide Recruitment Methods* 
 
Recruitment and Referral Sources. Participants will be drawn from up to four clinic locations in 
Tennessee of Health Connect America, Inc. (HCA). 
Youth and their caregivers (N = up to 20 dyads) will be included in the small pilot that is part of 
Aim 1. Clinicians taking part in this pilot study will be asked to try out the measurement-base 
care (MBC) system with clients on their existing caseloads. The research team will provide 

 
 
Page 6 of 30 
  
clinicians with information about how to select appropriate cases and a recruitment flyer (Pre-
Pilot Recruitment Flyer) and will encourage them to discuss client selection with their 
supervisors if needed. In the first treatment session where the youth and caregivers complete the 
MBC measures through the Mirah platform, an invitation to complete additional research 
measures will be provided, along with consent/assent statements. Families will have the option 
to: 1) enroll in the study at that time, 2) ask to be contacted for further information before 
deciding, or 3) decline participation. Families who agree to participate will randomly assigned to 
either multidimensional or unidimensional treatment through the Mirah system and de-identified 
data will be provided to the research team. Families will receive up to $20 in gift cards for 
completing additional study measures. 
Youth and their caregivers will be included in Aims 2 and 3 in three ways. First, the four study 
clinics will be using the measurement-based care system with all clients receiving in outpatient 
and intensive outpatient mental health services (up to 2000 youth falling in the target age range) 
and have agreed to incorporate the study randomization scheme as a quality improvement effort. 
PI Douglas has successfully used this approach in two previous randomized controlled trials of 
MBC46,47. Second, in the session where the youth and caregivers complete the MBC measures 
through the Mirah platform, they will be recruited via the Mirah Platform to complete additional 
measures using the same procedures described for the Study 1 pilot trial. Recruitment will take 
the form of presenting a recruitment screen, followed by the consent form if they indicate they 
are interested in learning more about the study at the end of their measures; families will also be 
notified that this will be happening via a flyer that the clinic will be using to tell them about the 
Mirah system. These flyers are tailored for each office, but we have uploaded one as an example 
(Mirah PACE flyer- Nashville). We have also uploaded a copy of the recruitment screen 
(Recruitment Information for Research Subsample). The clinics may use Mirah at the clinic 
intake session, before the decision has been made which services the youth will receive. If a 
family consents to be in the research sample and then the youth does not go on to be assigned to 
one on the services using Mirah, they will be contacted and informed that they will no longer be 
eligible to take part in the study.  Up to 400 youth and their caregivers will be enrolled in these 
additional research activities. Finally, for the final phase of the mixed-method study of 
implementation factors, a subset of up to12 families who agreed to take part in the research 
study, stratified by their level of adherence to the MBC system, will be recruited with the 
assistance of their clinicians, who will provide them with a flyer describing the study (uploaded 
as PACE Family Triad Interview Recruitment Flyer) and asked if they would be interested in 
being contacted by research staff to hear about an opportunity to participate in qualitative 
interviews to better understand their experiences using the system. Families will be paid up to 
$60 in gift cards ($30 per informant) for participating in those interviews. 
The PIs will directly recruit Agency Leaders (N = 5-15) to participate in qualitative interviews to 
throughout the study. Leaders will include one agency leader (e.g., the clinical director) at each 
clinic, and one administrator from the Health Connect central leadership. They will be paid $20 
per hour for participating in these interviews.  
Each of the clinics will have an on-site “MBC champion” who will support recruitment efforts, 
coordinate data collection activities with the research team, and generally serve as a local leader 
for the MBC efforts.  MBC champions and agency leaders will contact clinicians to recruit them 

 
 
Page 7 of 30 
  
for the study. The research team will provide recruitment flyers to facilitate recruitment and will 
hold informational conference calls to help with recruitment as needed. During Aim 1 activities, 
up to 10 clinicians will take part in individual or group interviews to inform development 
activities. In addition, 5 clinicians will take part in the small pilot trial to refine procedures before 
Study 2, including cognitive interviews to refine study measures. In order to incentivize 
participation, clinicians will receive $20 per hour for participating in study focus groups, $30 per 
hour for taking part in individual interviews, and $36 per pilot case for completing ongoing 
measures.  
For Aim 2, agency leaders will recruit up to 200 clinicians over the duration of the study to 
participate in the trial. All agency clinicians will utilize the MBC system and take part in MBC 
implementation activities as part of the program-wide implementation, but clinicians will be 
informed that their participation in any additional research activities is voluntary and will not 
affect their relationship with the clinics. In order to incentivize participation, clinicians will 
receive $30 per hour for participating in qualitative individual  interviews and $36 per case for 
completing ongoing measures for any of their clients whose caregivers consent to complete the 
extra research measures. A group interview with clinicians (N= up to 10) will be held at the 
beginning of the pilot in year 2. Agency staff members will also complete a follow-up survey at 
the end of Year 3.  
8) Study Timelines* 
 
The proposed study is a three-year pilot effectiveness trial, comprised of two Aims. Aim 1 
activities will include a variety of development activities to prepare for the Aim 2 Effectiveness 
Implementation Hybrid Study. The project timeline is detailed below. Agency Leaders and 
Clinicians will participate for up to three years and youth participants will participate for up to 18 
months.  
 
Project Timeline 
Year 1 
Year 2 
Year 3 
Quarters by Year 
1 
2 3 
4 
1 
2 3 4 
1 2 3 4 
Study Start 
Finalize preliminary study materials and 
scheduling for phase 1 activities 
  
  
  
  
  
  
  
  
  
  
  
  
Study 1: Aim 1 Development Activities 
1. Site visit for orientation (including with on-site 
study champions), leadership and clinician 
interviews  
  
  
  
  
  
  
  
  
  
  
  
  
2. Develop algorithms 
  
  
  
  
  
  
  
  
  
  
  
  
3. MBC technology additions and preparation of 
pre-pilot study materials 
  
  
  
  
  
  
  
  
  
  
  
  
4. Site visit for pre-pilot training and pre-pilot 
launch 
  
  
  
  
  
  
  
  
  
  
  
  
5. Three-month pre-pilot study of procedures 
  
  
  
  
  
  
  
  
  
  
  
  
6. Cognitive interviews on mechanism 
measures 
  
  
  
  
  
  
  
  
  
  
  
  

 
 
Page 8 of 30 
  
7. Finalize preparation of revised study 
materials 
  
  
  
  
  
  
  
  
  
  
  
  
8. Site visit for initial training and study launch 
including clinician interviews 
  
  
  
  
  
  
  
  
  
  
  
  
Study 2: Effectiveness Implementation Hybrid Trial 
1. Site visit for initial training and study launch 
including clinician interviews 
 
 
 
 
 
 
 
 
 
 
 
 
2. Ongoing coaching and implementation 
support; onboarding new clinicians 
  
  
  
  
  
  
  
  
  
  
  
  
3. Recruitment of new clients to participate in 
study procedures 
  
  
  
  
  
  
  
  
  
  
  
  
4. Research measure collection (baseline, 6, & 
12 weeks) 
  
  
  
  
  
  
  
  
  
  
  
  
5. Ongoing collection of MBC mechanism 
measures 
  
  
  
  
  
  
  
  
  
  
  
  
6. Session-by-session administration of MBC 
measures and clinician use of feedback 
  
  
  
  
  
  
  
  
  
  
  
  
7. Ongoing collection of MBC implementation 
measures and metadata 
  
  
  
  
  
  
  
  
  
  
  
  
8. Ongoing enrollment and attrition analysis by 
study arm 
  
  
  
  
  
  
  
  
  
  
  
  
9. Semi-structured interviews with up to 12 
triads (clinician, caregiver, patient); leadership 
interviews; clinician follow-up survey 
  
  
  
  
  
  
  
  
  
  
  
  
Analysis and Reporting 
  
  
  
  
  
  
  
  
  
  
  
  
1. Randomized-experiment quantitative 
analyses 
  
  
  
  
 
 
 
 
 
 
  
  
2. Qualitative analyses 
  
  
  
  
  
  
  
  
  
  
  
  
3. Integration of all analyses and report 
preparation 
  
  
  
  
  
  
  
  
  
  
  
  
4. Write and submit R01 application for full 
SMART 
 
 
 
 
 
 
 
 
 
 
 
 
Complete and Submit Yearly Progress 
Reports, Implement Dissemination Plan 
  
  
  
  
  
  
  
  
  
  
  
  
 

 
 
Page 9 of 30 
  
 
9) Study Endpoints* 
 
The primary study endpoints are the youth- and caregiver-reported Symptoms and Functioning 
Severity Scale and the youth- and caregiver-reported Ohio Problems and Functioning Scale 
(Research Subsample Only) 
 
10) Procedures Involved* 
 
Below, we present a general overview of the study methods as outlined in the grant proposal. 
Please note that this R34 development grant is designed to generate procedures iteratively over 
the course of the study, so materials and procedures will be updated and submitted as they are 
developed through the procedures outlined below.   
METHODS FOR AIM 1: PRELIMINARY DEVELOPMENT ACTIVITIES 
 
The goal of Aim 1 is to conduct development activities to ensure fit between agency workflow 
and the MBC system, implementation support tools, and research procedures. In Development 
Phase 1 (Aim 1a), qualitative data collection will refine research measures and protocols, and 
assess potential implementation barriers and facilitators. Three qualitative components will be 
conducted with semi-structured protocols. First, either at the initial site visit or via phone or 
video conferencing shortly thereafter, individual or group interviews will be held with agency 
leaders (1-3 per clinic, for up to 15 interview participants) to help develop procedures for the 
pre-pilot study and discuss organizational-level factors that may influence MBC implementation. 
Second, individual or group interviews will also be held at the initial site visit or via phone or 
video conferencing shortly thereafter with up to 10 eligible clinicians combined from all five 
clinics. The interviews will gather information about clinical decision-making, typical clinic 
workflow, and any existing strategies for tracking treatment progress. Participants in individual 
and group interviews will respond to open-ended questions and be asked to react to sample MBC 
materials, study measures, and study procedures, and will be paid $20 per hour for their time. 
Interviews will last 30 to 90 minutes, depending on participant availability and preferences. The 
interviews will be recorded and transcribed for coding, with results used to modify the project 
materials. The interview questions have been submitted under study documents.  
 
In Development Phase 2 (Aim 1b), we will develop algorithms to identify NOT and OT 
trajectories for the SFSS to generate alerts that will be embedded into the MBC feedback reports, 
support moderator analyses of intervention condition by OT/NOT effects, and set the stage for 
the randomization scheme for the R01 SMART. Analyses will utilize existing limited identifier 
datasets from 2 previous RCTs46,47, including data from 597 youth treated by 165 clinicians 
using multidimensional MBC at 30 sites. During this phase, Mirah staff will also embed study 
measures into the system, update feedback reports with the alerts, and build a system to automate 
study randomization. 
  
In Development Phase 3 (Aim 1c), five clinicians will pilot the MBC system with up to 4 
clients each; clinicians will select clients from their own caseloads based on the clinician’s 
decision that it would be appropriate to begin using MBC with the client. Before beginning to 
treat their clients in the pre-pilot, clinicians will complete a packet of some or all of the baseline 

 
 
Page 10 of 30 
  
measures planned for the full RCT (See upload of Baseline Clinician Measures); these measures 
will be collected either via REDCAP or Mirah. Procedures for this pre-pilot will be identical to 
for Aim 2 (see below) to identify challenges related to recruitment, randomization, and/or data 
collection. The only change to the Aim 2 procedures is that these pre-pilot cases will only be 
followed for 6 weeks, complete two rounds of research measures rather than three, and be paid 
up to $20 for participation (a $10 gift card for each round of assessment measures completed). 
The use of MBC is also part of consenting to the study, so the MBC measures from the pre-pilot 
have been uploaded as a study document (MBC Measures Pre-pilot). 15- to 30-minute cognitive 
interviews will be held with the five clinicians to refine measures and feedback reports, 
conducted by telephone or video conference immediately following clinical coaching calls. In 
cognitive interviewing, the participant’s thinking ‘out loud’ when reviewing materials is 
recorded to assess how the clinician is viewing and interpreting them. Clinicians will review 
study mechanism measures to determine how they interpret the items and response options, 
ensure wording is clear, and ensure adequacy and accuracy of the items; they also to review a 
feedback report aloud. Each clinician will take part in up to 5 cognitive interviews during the 
pre-pilot. The interviews will be recorded and transcribed for coding. Following the pre-pilot and 
these interviews, final revisions will be made to study procedures and feedback reports as 
needed.  
METHODS FOR AIM 2. PILOT EFFECTIVENESS IMPLEMENTATION HYBRID 
TRIAL 
 
Design. This study will utilize a Type 1 Effectiveness Implementation Hybrid design, in which 
an effectiveness trial (Aim 2) is conducted while simultaneously gathering data to inform future 
implementation (Aim 3). The effectiveness trial will be a two-group RCT, with clients 
randomized to unidimensional or multidimensional MBC. The HCA clinics have agreed to 
randomize all of their youth clients to one of these two conditions, as they view this as a quality 
improvement initiative. Consent will be obtained from a subset of 400 clients and caregivers (i.e. 
the Research Subsample) to complete additional research measures. The implementation study 
will be a mixed-methods analysis.  
 
Procedures for effectiveness trial. After Aim 1 is complete, program-wide implementation of 
MBC will begin. The initial training will consist of a series of workshop style trainings 
conducted during regular agency staff meetings and ongoing consultation will be provided to 
agency staff thereafter. The training approach is a data-driven one, where questionnaires are used 
throughout the training and consultation to tailor implementation to that specific context. These 
questionnaires will be considered part of the implementation activities and will be administered 
to all staff members participating in the trainings. Clinicians who consent to be in the research 
study will be giving permission for these implementation data to be linked to the research data 
they provide for their clients; responses from other HCA staff will be de-identified and used to 
characterize the implementation under Aim 3. Prior to extracting these data for research 
purposes, we will submit a variable list to the IRB for approval. When the initial training is 
complete, the trial will begin program-wide at the participating HCA clinics. As the clinics add 
clients ages 11-17 to the Mirah system, the system will automatically randomize them to either 
unidimensional or multidimensional MBC on a 1:1 ratio. All clients and their caregivers will 
complete MBC measures as part of routine services, and the researchers will receive limited 
identifier datasets containing MBC data and Mirah metadata for all participants, an identifiable 

 
 
Page 11 of 30 
  
dataset for individuals who consent to take part in the research subsample. We have uploaded a 
variable list of the variables we will be requesting from Mirah. After the first time a youth and 
their caregiver complete the MBC measures, the Mirah system will proceed to screens with 
recruitment and consent information for participation in the Research Subsample. Youth and 
caregivers will be asked to assent/consent to complete additional research measures. Each time 
they complete Mirah measures for routine sessions, youth and caregivers will be administered 
one additional mechanism item; this item will be included in up to 12 sessions after they enroll in 
the study. In addition, they will complete a longer set of research measures will including the 
Ohio Scales and additional mechanism measures; these will be administered at study enrollment, 
6 weeks later, and 12 weeks after study enrollment. These longer sets of research measures will 
be administered via the REDCap system, with links e-mailed directly to the family by research 
staff. Additional Research Subsample Measures have been uploaded under Study Documents. 
Families who consent to be a part of this Research Subsample will be paid up to $30 for 
completing these measures (the project will e-mail them a $10 gift card per major assessment 
completed either directly through e-mail or by using Qualtrics and Tango Card). Clinicians 
treating clients in the Research Subsample will also be asked to complete additional measures for 
those clients and will be paid $36 for completing those measures. The Mirah system does not 
have a way to only recruit clients whose clinicians have consented to be in the research study or 
to tailor the measure schedule for those clients. Therefore, if a client agrees to be in the study and 
their clinician has not, the study will contact the clinician, let them know that they will be 
receiving links to research measures, and see whether they would like to consent to be in the 
study at that point. If they do not, then they will be instructed to disregard the requests for them 
to complete measures. These measures are also included in the Research Subsample Measures 
upload under Study Documents.  
METHODS FOR AIM 3. MIXED-METHODS ANALYSIS OF IMPLEMENTATION 
FACTORS 
 
Design. The Aim 3 qualitative data collection will focus on implementation barriers, facilitators, 
and outcomes. As in Aim 1, semi-structured individual and group interviews will be used to 
derive rich descriptive understanding of internal thought processes and feelings associated with 
MBC (e.g., acceptability or decision-making) and perceived links to behavior (e.g., fidelity of 
measure completion or feedback viewing). Three major differences are: a change in focus to 
factors associated with implementation and utilization; the inclusion of client and caregiver 
perspectives; and the addition of purposeful sampling based on quantitative fidelity data. 
Qualitative data will then be analyzed together with quantitative fidelity and implementation 
indicators. 
 
Procedures for mixed-methods data collection. Aim 3 qualitative components will include: (1) 
clinician individual and group interviews before the RCT, (2) agency leader individual and group 
interviews before and after of the RCT, (3) up to 12 triad interviews, selected from Research 
Subsample youth, caregivers, and clinicians, and (4) field notes, responses to implementation 
questionnaires, and audio recordings of training activities such as consultation calls. For the 
clinician and agency leader interviews, a priori themes to guide questioning have been developed 
through a review of the MBC literature and related fields and are organized by five factors from 
the CFIR model78,79: intervention characteristics (e.g., MBC measures), extra-organizational 
(e.g., payer MBC reimbursement), inner-organizational setting (e.g., leadership support for MBC 

 
 
Page 12 of 30 
  
use), individual stakeholder characteristics (e.g., perceived value of MBC), and process factors 
(e.g., ongoing support needs). The post-RCT agency leader interview script has been uploaded as 
FINAL Leader Semi-Structured Interview Protocol).  
 
For the triad interviews, participating clinics will provide a list of clinicians and their MBC 
implementation data. We will recruit up to 12 clinicians with a range of fidelity of feedback 
viewing over the previous month, identified using Mirah metadata. Clinicians will be recruited 
with the assistance of agency leaders, using a recruitment flyer that can be distributed by e-mail 
or handed out in meetings (PACE Clinician Triad Interview Recruitment Flyer). Interested 
clinicians will be contacted by the research team to schedule an interview to be conducted via 
zoom.  At that appointment, clinicians will be verbally consented (PACE Clinician Triad 
Interview Script) and then complete an individual qualitative interview with the study staff 
(PACE Clinician Triad Interview Verbal Consent Script). Interviews will be recorded.  
 
We will ask these clinicians to help recruit one family from their case load to participate in the 
study.  Clinicians will be provided a flyer explaining the study (PACE Family Triad Interview 
Recruitment Flyer) and obtain permission for interested families to be contacted by the research 
team by phone, text, or e-mail using the clinic’s standard release form. The flyer will include 
information stating that the family’s decision to take part in the study will not affect their 
treatment and clinicians will be asked to provide similar reassurance when telling families about 
the study. The research team will then reach out to interested families to further explain the study 
and schedule the interviews to be conducted via zoom. Caregivers will be scheduled first, 
followed by youth. At each interview the informant will first be asked to verbally consent/assent 
to the study (PACE Caregiver Triad Interview Verbal Consent Script; PACE Youth Triad 
Interview Verbal Assent Script), followed by a semi-structured interview conducted by study 
staff (PACE Client and Caregiver Qualitative Interview). These interviews will be recorded. As 
the triad interviews are conducted, we will assess for saturation of differing levels of MBC 
implementation and recruitment will cease when the qualitative data are thematically saturated. 
All participants (clinicians, youth clients, and caregivers) will receive $30 for participating in the 
one-hour qualitative interview.  While the sample size is not sufficient for generalizability (or in 
qualitative parlance, transferability), the clinician interviews will further inform our 
understanding of the MBC mechanisms of action related to the CFIT theory. The separate 
interviews with youth and caregivers will allow for a better understanding of the MBC 
mechanisms of action related to the therapeutic assessment theory.  
 
The study PIs and/or project research assistants will observe and take notes during project 
implementation activities for later qualitative coding and we will retain answers to the 
questionnaires used during training and consultation. We will also ask participants to consent to 
recording consultation calls and other training activities. It is possible that individuals who have 
not consented will take part in those activities. As such, all recordings will be transcribed, any 
identifying information removed, and then deleted to avoid retaining recordings of individuals 
who did not consent to the study.  
 
Aim 3 quantitative data includes Mirah metadata on fidelity (measure completion and feedback 
viewing rates) and measure completion time; and data on implementation monitoring (training 
attendance, evaluations, case review) and implementation barriers, facilitators, and outcomes 

 
 
Page 13 of 30 
  
(answers to implementation questionnaires, youth and caregiver perceptions of MBC 
acceptability) Aim 3 quantitative data will also include a clinician follow-up survey, which 
includes measures related clinician attitudes, use of MBC, organization-level factors, and 
implementation efforts (Pace staff post survey FINAL); measures will be collected via 
REDCAP. As clinicians joined the study, we asked clinicians to consent to allow their identified 
responses to the implementation questionnaires to be linked to our research data; responses from 
clinicians who do not consent to take part in the research will be de-identified and used to 
characterize the implementation activities only. An additional consent form will be administered 
together with the follow-up survey via REDCAP to cover that research activity (PACE Provider 
Follow-Up Consent).  
 
11) Data and Specimen Banking* 
N/A, no specimens will be collected as a part of this research study 
 
12) Data Management* 
STUDY ANALYSES 
Qualitative Analyses. To code the data from the individual and group interviews (Aim 1a), 
we will use a categorizing strategy of coding and thematic analysis108,122, in which the data will 
be systematically coded into discrete categories based on a two-pronged approach that relies on: 
(a) an a priori analytic framework; and (b) emergent categories or themes using “grounded 
theory”123. Then, analysis will move to comparing similarities and differences between 
participants and clinics. Thus, the emerging theories will be grounded in real-world patterns124 
within and between sites. Analyses will include thick descriptions of the patterns and themes that 
emerge as well as the associations between the background characteristics of participants, the 
sites with which they are affiliated, and their responses to our questions and probes. Computer-
assisted data analysis software (e.g., NVivo) will be used for coding122 patterns among 
individuals and sites. 
Quantitative Analyses. Aims 1b and 2 will involve using multilevel modeling (MLM) 
approaches125, to account for nesting of repeated measures within clients and clinicians. In all 
Aim 2 models, we will treat site as a fixed effect, due to the limited number of clinics (n=3).For 
Aim 1b, on-track (OT) and not-on-track (NOT) algorithms will developed using an extant 
dataset46,47 (N = 597). Analyses will progress in two stages, following established procedures 
previously applied to the Outcomes Questionnaire system118-120, the most widely-tested MBC 
system19. The first stage of the analysis involves generating expected change trajectories for 
SFSS scores over time. MLM will be used to estimate these individual change trajectories with 
weekly scores nested within participants, who are in turn nested within therapists. Models will be 
tested using the Hierarchical Linear Modeling (HLM) software package126 and restricted 
maximum likelihood (RML) estimation. Prior to hypothesis testing, we will examine SFSS 
distributions and apply transformations (e.g., square root, logarithmic) for the weeks in treatment 
variable as necessary118.  
The second stage of the analysis involves: (1) developing a system to identify cases at risk of 
showing treatment deterioration (NOT), and (2) testing its predictive accuracy.  Identification of 
NOT cases is based on the expected change trajectories (intercepts, slopes, and random effects) 
estimated in Stage 1. Following Cannon et al118, we will split the data into two random samples, 
a reference sample and a validation sample. As a first step, the base rate of treatment 

 
 
Page 14 of 30 
  
deterioration—a significant worsening of symptoms over the course of treatment—will be 
identified in the reference sample. Consistent with previous studies118-120, identification of the 
base rate will be based on a combination of the reliable change index127 (RCI) and clinical cutoff 
scores. Established SFSS values for the RCI and the clinical cutoff score128 will be used to 
identify cases with clinical deterioration, operationalized as an SFSS increase over treatment 
exceeding the RCI and a case ending treatment in the clinically significant range of functioning.  
The deterioration base rate from the reference sample will serve as a basis for deriving prediction 
intervals around the expected mean change trajectory in the validation sample. Cases identified 
as exceeding the prediction interval at any point during treatment will comprise the NOT group. 
Cases within the interval form the OT group.  
The validation sample will then be used to check the validity of the warning system in 
predicting true treatment deterioration. Analyses will examine whether classifying a client as 
NOT accurately predicts whether they end treatment classified as deteriorated. Sensitivity and 
specificity will be calculated and the algorithms will be adjusted as needed until adequate levels 
are reached, using Cannon et al.’s118 specificity values of .55 and specificity values of .75 as 
guides. We will then test the predictive validity of the finalized OT/NOT algorithm (Aim 2a) 
using the full RCT sample (N = 900). To prepare for the SMART R01, we will also conduct 
simulations, varying estimates entering the expected change trajectories to investigate transition 
points in the OT/NOT intervals to assess optimal timing of when to conduct the second 
randomization. 
Aim 2b will investigate potential factors associated with condition, client, or clinician that 
could influence a future SMART. GLMM (MLM with a logistic linking function) will be used to 
investigate factors associated with treatment and attrition from the study. In addition, differences 
between the full RCT sample (N = 900) and the Research Subsample (N = 400) will be examined 
utilizing data from electronic health records (EHR) to determine whether enrollment in the 
Research Subsample, and completing all 12 weeks, is associated with any demographic or 
clinical factors. We will also examine enrollment and attrition rates for clients within clinicians 
to determine whether any clinician factors might be associated with client-level participation in 
the study, along with comparing enrollment rates across clinics. As above, these analyses will 
inform design of the R01 SMART, including power analyses that take into account rates of 
NOT/OT and projected attrition.  
MLM will also be used to compare the effects of study condition on SFSS change 
trajectories, Ohio Scores, and MBC mechanisms (Aim 2c). We will examine differences in MBC 
outcomes by feedback condition, nesting within treating clinician and controlling for clinic. If 
sample sizes in the research subsample prove too small for this analytic approach, analyses of 
variance will be used to test a time by treatment interaction. As a first step in comparing 
conditions on MBC mechanisms, we will aggregate the scores across sessions (within therapist) 
by averaging them or summing count variables (e.g., matches between a client being identified 
as NOT and the therapist recognizing deterioration) to create composite variables for each 
mechanism domain in Figure 2.  
Finally, we will extend the MLMs tested to accommodate potential mediators and 
moderation by NOT status, diagnosis, and treatment type (Aim 2d). Mediation will be tested 
using the products of coefficients method 129, using bootstrapping to adjust for nonnormal, 
asymmetric confidence intervals130.  Algorithms for testing mediation using these methods are 
available in the package RMediation131 available in the software platform R. Moderation by 
NOT status will be tested by entering NOT status, along with the feedback condition by NOT 

 
 
Page 15 of 30 
  
status interaction and other model covariates. Similar models will examine moderation by 
diagnosis and treatment type.  
Mixed-methods analyses. Aim 3 will be analyzed using mixed methods to examine 
implementation factors related to MBC. We will rely on the complementarity of qualitative and 
quantitative data to ensure interrelation of the data through an iterative analysis process132,133. We 
see a significant purpose of this mixed method approach as exploratory, and will use the results 
of qualitative methods to inform and modify both future quantitative methods (e.g., surveys) and 
design implementation enhancement efforts134. We will use a combination of stakeholder 
individual interviews, group interviews, surveys, and observations of training activities to 
examine cognitive and behavioral factors such as MBC acceptability, perceived sustainability, 
etc., among clients, clinicians, and agency supervisors and administrators. With the exception of 
the triangulation of quantitative data, these qualitative data will be coded using a priori and 
grounded theory themes using Nvivo software as described in C10.1 above. Quantitative data 
will explore group differences in client-level implementation factors (e.g., time burden to 
complete measures) with youth and caregiver ratings of acceptability of MBC measure 
completion time and meta data on actual measure completion time from Mirah analytics. Agency 
leader and clinician survey data on clinician-level MBC attitudes and organization-level 
implementation factors will be aggregated by agency at project start and end, which allows for 
triangulation of data at baseline and project end with concurrent agency leader and clinician. 
Triad interviews with matched clients, caregivers, and clinicians will be explored qualitatively to 
inform our understanding of the MBC mechanisms of action related to the CFIT theory (e.g., 
recognition of actionable feedback, cognitive dissonance, and clinical actions). The separate 
interviews with youth and caregivers will allow for a better understanding of the MBC 
mechanisms of action related to the therapeutic assessment theory (e.g., treatment engagement, 
therapeutic alliance). 
Sample size and power calculations. Given the preliminary nature of this pilot study, power 
calculations focused on power for the analyses of the group differences in outcomes (Hypothesis 
2c.1). Power calculations were conducted using Optimal Design 3.1. We assumed 8% of the 
outcome variance at the clinician level135 for an n of 22 clinicians (over the duration of the study) 
and 900 adolescents, β = .80, and p = .05.  Using these estimates, Aim 2c, examining the impact 
of feedback condition on treatment response (assessed by the SFSS) will be optimally powered 
to detect a small effect of d = .25. The only previous estimate of differences between 
multidimensional and unidimensional MBC focused on the NOT subgroup only, finding an 
effect of (d = .31); the overall effect size for this study is expected to be smaller because the 
analyses will focus on the NOT and OT groups combined. Using a more conservative small 
effect size of d = .20 yields power of approximately .75. With respect to power assessing 
differences by feedback condition in the Research Subsample (n = at least 22 clinicians and 400 
adolescents) will be optimally powered for a small-moderate effect of d = .30.  Assuming an 
effect size of d = .25 yields power of .62. 
DATA SECURITY, STORAGE, AND QUALITY CONTROL 
 
Quantitative data will be collected electronically via either the University of Miami 
REDCAP system or directly through Mirah. Mirah is a HIPAA compliant system for 
collecting and scoring clinical measures. It uses Secure Sockets Layer (SSL) encryption 
to protect all data during transfers and stores is on a secure server. Data access will be 
restricted to study personnel, with permissions set to only those necessary to perform 

 
 
Page 16 of 30 
  
tasks. Whenever possible, staff will download data directly from REDCAP or Mirah, but 
any file transfers between sites will be done via an approved secure platform such as 
secure send. Downloaded files will be stored on the psychology department’s server, 
box.com, one drive, and/or google drive. The project coordinator will be regularly 
reviewing data for threats to data quality, such as missing data. Built-in validation 
protocols in REDCAP and Mirah will prevent entry of invalid responses.  
 
Qualitative interview recordings, transcripts, and codes will also be stored securely on 
psychology department’s server, box.com, one drive, and/or google drive.  
13) Provisions to Monitor the Data to Ensure the Safety of Subjects* 
 
This study only presents minimal risks to participants.  
 
14) Withdrawal of Subjects* 
Participants and their parents/guardians may be withdrawn from the study without their consent 
if they are withdrawn from clinical care in the participating clinics. Should the client be 
terminated or transferred in accordance with the local clinic policy, he/she will be 
withdrawn from the study. In addition, participants may be withdrawn for the following 
reasons: 
● The participant is no longer able to attend clinic visits or complete research assessments. 
● There may be other reasons that are unforeseen at this time. 
 
Those participants who voluntarily withdraw from treatment may be asked to complete the 6 and 
12 week assessments for data collection purposes, if appropriate.   
 
Clinicians may be excluded from the study for any of the following reasons: 
● They leave the clinic and are no longer seeing clients at the agency. 
● They are unable to attend and/or participate in training and consultation meetings. 
● They are determined inappropriate for the research study (i.e. failure to adhere to 
treatment, failure to adhere to study procedures). 
 
 
15) Risks to Subjects* 
 
This study is considered to involve minimal risk for all participants. For the clinician and agency 
leader participants, participants, the main risk is that they will be asked information about their 
work in a group interview setting where their colleagues might hear their answers. In addition, 
the clinicians taking part in the pre-pilot and RCT may experience slight discomfort participating 
in MBC consultation calls or from seeing client progress reports. However, clinicians could 
experience this same discomfort in the routine supervision they receive. For youth and caregiver 
participants, there could be some risk of discomfort from disclosing information through the 
Mirah platform, although this discomfort is not anticipated to exceed that involved in typical 

 
 
Page 17 of 30 
  
clinical procedures. In addition, they may experience minor discomfort giving feedback about 
their experiences using the MBC system. 
 
For agency leader and clinician participants, the PIs will emphasize with participants and agency 
administrators that participation is voluntary and that clinics cannot penalize individuals who 
decline participation. All information obtained through individual and group interviews and 
surveys will be kept private, with only summaries of information relevant to implementation 
provided to clinics if it is possible to do so without violating participant confidentiality. To 
decrease risks to individuals participating in group interviews, we will: 1) ensure that none of the 
participants are in a supervisory role over other participants, and 2) begin the session by 
discussing the importance of confidentiality and the limits to confidentiality based on the group 
setting, and asking all participants to not discuss anything from the interview outside of the 
group.  
 
As detailed above, the primary strategy for protection against risk for youth is that the study will 
not collect any identifying information, which protects the privacy of clients and ensures the 
confidentiality of their data. Because the study activities present minimal risk and are similar to 
activities they are already choosing to engage in by enrolling in treatment, we do not anticipate a 
risk of adverse events. 
 
16) Potential Benefits to Subjects* 
 
Youth participants will potentially experience alleviation of distress associated with a mental 
health condition. In both study conditions, youth have access to MBC, an evidence-based 
practice that is not currently used in these agencies.  
 
Therapist participants will have access to training and consultation in a MBC.  
 
17) Vulnerable Populations* 
 
This research involves youth 17 and under. A number of safeguards have been put in place and 
careful consideration has gone in to the decision to conduct this research with this particular 
population. The PIs have extensive experience conducting research and clinical work with this 
population. All study staff involved will be trained in conducting research with vulnerable 
populations, including minors. All study therapists will have experience treating mental health 
concerns in youth.  The risks to the study are minimal, so the benefits outweigh the risks to 
participants.  
 
18) Multi-Site Research* 
 
Dr. Susan Douglas at Vanderbilt University and Dr. Jensen-Doss at the University of Miami as 
co-PIs for this grant and will be responsible for the management of the entire project including 
developing and implementing project policies and procedures. Both Drs. Douglas and Jensen-
Doss will share equal responsibility for implementation of the scientific agenda. Additionally, 
the PIs will both be responsible for ensuring that the project adheres to US Laws and NIH 
policies, including guidelines for the protection of human subjects. Dr. Douglas will take primary 

 
 
Page 18 of 30 
  
responsibility for the qualitative arm, implementation support and measurement, and quarterly 
site visits to clinics.  Dr. Jensen-Doss will take primarily responsibility for the quantitative arm. 
She will also oversee database management and data analyses, in collaboration with the 
statistical consultant. The PIs will share responsibility for overseeing the training and supervision 
of project staff. Dr. Jensen-Doss will serve as contact PI and will assume fiscal and 
administrative management, responsibility for communication with NIMH, and submission of 
annual reports. As the primary award site, the University of Miami (UM) will have primary 
administrative responsibility for the project, with funds to Vanderbilt awarded via a subcontract. 
The UM Institutional Review Board (IRB) will serve as the coordinating IRB- we will initiate 
the process to establish UM as the coordinating IRB after the grant is funded.  
 
Communication Schedule: The two PIs will have at least weekly phone or skype meetings to 
review progress towards study aims, address any problems that arise, and manage the project. 
The PIs will also hold joint phone or skype meetings weekly with the project coordinator to 
ensure fidelity to project procedures, with Dr. Jensen-Doss holding additional in-person meetings 
with the project coordinator as-needed. Funds are also budgeted for in-person meetings, 
including an initial start-up meeting at UM, six site visits across the life of the project, meetings 
at one conference per year, and a closing meeting at Vanderbilt in Year 3.  
 
19) Community-Based Participatory Research* 
 
N/A 
 
20) Sharing of Results with Subjects* 
 
The Mirah platform will be used to conduct the MBC, and clinicians will collect and view these 
data as part of their routine care. The research-specific instruments will not be shared with 
clinicians, and it will also not be possible for the study to share the results directly with 
participants, as the researchers will only receive limited identifier datasets from Mirah.  
 
21) Setting 
 
The study will take place in up to four locations of a community mental health agency serving a 
diverse client population in Tennessee. The four clinics are part of Health Connect America 
(HCA), which provides behavioral health treatment in 45 offices in five states. The corporate 
headquarters is located 20 minutes from Nashville, TN (location of PI Douglas). HCA has not 
previously routinely used MBC. They have agreed to conduct the study program-wide in five TN 
clinics with youth receiving intensive outpatient (two programs, referred to as intensive 
outpatient and intensive home-based services by the agency) and outpatient mental health 
services, with a client flow estimated at up to 2000 youth over an 18-month period. All 
recruitment and study activities will take place at HCA 
 
22) Resources Available 
 
Project staff will include:  

 
 
Page 19 of 30 
  
PIs (Jensen-Doss, Douglas at Vanderbilt): Both PIs have extensive experience conducting 
research with children, with clinical populations, and in clinical settings.  
Research Assistant: This individual will hold a bachelor’s degree.  This individual will be 
supervised closely by the PIs and the postdoctoral fellow.   
Co-I (Craig Henderson, Sam Houston State University): The Co-I is an experienced 
biostatistician with expertise in the quantitative methods to be utilized.  
Study resources include:  
Univeristy of Miami: PI Jensen-Doss 
The University of Miami provides Dr. Jensen-Doss with a dedicated faculty office, 
laboratory space, offices for postdoctoral associates and additional graduate student office space, 
all on the same floor in a relatively new psychology building completed in 2003.  This space is 
large enough to house all UM site staff working on this grant.  All of this space is wired for 
internet, wifi and phone access. The UM Department of Psychology has an in-house team of 
three consultants for hardware and software needs and an additional IT support technician.  Over 
a dozen servers provide separate functions including printing, e-mail, and software applications.  
Network directories are encrypted, backed up nightly and scanned in order to be virus-free. UM 
will provide access to tools (e.g., copying machine, fax, computers, phones, etc.) needed for 
project implementation.  
 
Vanderbilt University: PI Douglas 
Dr. Douglas has resources available to her to fully support her activities as a principal 
investigator on the project, including a dedicated faculty office and an office for a master’s 
student research assistant, all on the second floor of the building occupied by the Department of 
Leadership, Policy, and Organizations (LPO) on the Peabody College campus. The space is 
wired for internet, wifi and phone access. Vanderbilt will provide access to tools (e.g., copying 
machine, fax, computers, microphones and audio recorders, phones, etc.) needed for project 
implementation. Vanderbilt University’s Information Technology (VUIT) provides computing 
resources, services and support to all Vanderbilt faculty, staff, and students. VUIT maintains and 
supports all network operations. This includes access to the Internet, collaboration, storage, 
telecommunication and video networks. Technical support specialists help to facilitate access to 
statistical and research tools, and provide consultation on the appropriate uses of software and 
storage of data. Research relationship managers connect researchers with cutting-edge resources, 
such as the Advanced Computing Center for Research and Education (ACCRE), a 5,000 core 
research cluster. The Office of Research has shared research resources and facilities that offer 
cutting edge scientific services, enabling access to high-end equipment, advanced techniques and 
specialized expertise for all Vanderbilt investigators. 
 
Sam Houston State University: Co-I Henderson 
Dr. Henderson has resources available to him to fully support his activities as the statistician 
on the project, including desktop computers which are on a protected and firewalled network, 
and require secure login credentials. Software appropriate for word processing, data storage, 
retrieval and statistical analysis (SAS, SPSS, Mplus, Stata, HLM) are readily available to him 
along with several high-quality printers. Dr. Henderson has ample room to house the proposed 
research without acquiring additional space, including two research offices with computer 
equipment, file cabinets, book shelves, and meeting tables. 

 
 
Page 20 of 30 
  
 
Mirah 
Mirah is a software and services company founded in 2015 to make measurement-based care 
better and more available to mental health clinicians and patients. Mirah’s staff includes both 
technology and clinical experts to provide cutting-edge mental health products and clinical 
support. Mirah’s clinical team consists of licensed medical and mental health professionals 
representing the fields of psychiatry, psychology, and social work. Mirah has a software and 
product team with years of experience delivering world-class health IT products to market. 
Mirah’s flagship product, the Mirah platform, enables fast, robust, and regular measurement and 
tracking of patient symptoms in a mental health setting. The Mirah Platform is currently 
successfully used across community care, research hospital, government, and private practice 
settings.  
Mirah’s engineering team has the resources and capabilities to rapidly prototype, develop, 
and launch new features and content to support the changing needs of customers and the 
market. These resources will support this proposal by providing any necessary adaptations to the 
Mirah Platform necessary to achieve the study goals, specifically including the addition of 
research measures and the development of a randomization scheme that will automate random 
assignment to groups for new clients involved in research studies. Mirah’s clinical staff includes 
leaders and advisors from a broad spectrum of mental health institutions, including, among 
others, McLean Hospital, Harvard Medical School, and Vanderbilt University. The clinical team 
brings vast experience in implementing and practicing measurement-based care in a variety of 
settings, and can use that experience to support this effort.  
 
Health Connect America 
Health Connect America, Inc. (HCA) is a fully licensed and certified private, not-for-profit 
corporation with national headquarters in Franklin, Tennessee and 45 locations in five states 
(Tennessee, Mississippi, Virginia, Alabama, and Georgia). HCA provides a full continuum of 
services for individuals and families at risk. With roots going back to 2005, HCA provides 
community-based, office-based, and home-based therapy, case management, medication 
management and counseling for children, adolescents, adults, and families with the steadfast goal 
of promoting personal positive growth, healthy coping skills, preserving/repairing relationships 
and natural support systems and adding additional needed community support systems. HCA 
staff are trained in various evidenced based models of treatment, such as Cognitive Behavioral 
Therapy, Rational Emotive Therapy, Trauma-Focused Cognitive Behavioral Therapy and 
specific training to use models that address issues of domestic violence, substance abuse and 
sexual behavior problems. HCA serves as a subcontractor for both private and public agencies, 
and accepts self-pay, Medicaid, and third party payers, and is accredited by the Council on 
Accreditation (COA). 
Three programs (Intensive Outpatient Services, Intensive Home-based Services and 
Outpatient Services) serving youth aged 11 to 17 years and their families in five clinics in 
Tennessee will take part in this study. As detailed in the Recruitment and Retention Plan 
documents, HCA has ample access to clients and clinicians to support the proposed research. As 
detailed in their letter of support, HCA is enthusiastic about participating in the proposed project. 
At their national headquarters and each of the clinic locations, HCA has sufficient conference 
room and office space to support the proposed training and data collection activities. Agency 
leadership has agreed to support staff time for participation in training and ongoing 

 
 
Page 21 of 30 
  
implementation support activities. They have wireless internet access throughout the agency to 
support use of the tablet-based MBC data collection, and their electronic health record (EHR) 
system, CareLogic, will be partially integrated with the Mirah system to facilitate feasibility of 
use.  
23) Prior Approvals 
 
N/A 
 
24) 
Recruitment Methods 
 
Please see above in Section 7 (Study-Wide Recruitment Methods). 
 
25) 
Local Number of Subjects 
 
Up to 15 agency leaders, up to 200 clinicians, and up to 2000 youth will take part in the study. 
 
26) Confidentiality 
 
X  Data obtained or created for this research will be stored on an encrypted 
electronic device or system owned by the University of Miami or on a 
cloud storage system that has been approved by the University of Miami 
for storage or research data.  
 
X The Investigator (or research staff) will record (e.g. write down, abstract) 
data collected in a manner that does not include any indirect or direct 
identifiers and the recorded data will not be linked to the individual’s’ 
identity.   
 
☐ The investigator (or research staff) will record (e.g. write down, abstract) 
the data collected in a manner that does not include any direct identifiers 
of the subject. The investigator will assign a code to each subject and link 
the code to the subject’s identity.  The research team will maintain the link 
to the subject’s identity on a document separate from the research data.  
Both documents will be stored in separate files on a University of Miami 
encrypted device or on a University of Miami approved cloud storage 
system. The research team will destroy the identifiers at the earliest 
opportunity. 
 
X The research team will maintain the research data for at least three years.  
 
☐  Bio-Specimens obtained for this research will be stored without any direct 
or indirect identifiers.   
 

 
 
Page 22 of 30 
  
 ☐ Bio-Specimens obtained for this research will be stored in a de-identified 
coded manner.   
 
X  When required to transport data or bio-specimens for this research, the 
research team will transport the data and bio-specimens in a de-identified 
(or anonymous) manner with a link to the individual subject’s identity 
maintain separately from the data and/or bio-specimen. 
27) Provisions to Protect the Privacy Interests of Subjects 
 
All information obtained through individual and group interviews and surveys will be kept 
private, with only summaries of information relevant to implementation provided to clinics if it 
is possible to do so without violating participant confidentiality. To decrease risks to individuals 
participating in the group interviews, we will: 1) ensure that none of the participants are in a 
supervisory role over other participants, and 2) begin the session by discussing the importance of 
confidentiality and the limits to confidentiality based on the group setting, and asking all 
participants to not discuss anything from the interview outside of the group.  
As detailed above, the primary strategy for protection against risk for youth is that the study will 
not collect any identifying information, which protects the privacy of clients and ensures the 
confidentiality of their data. 
 
28) Compensation for Research-Related Injury 
 
N/A/ The study is minimal risk.  
 
29) Economic Burden to Subjects 
 
There is no economic burden to subjects, as the study procedures are part of the care they are 
already receiving or part of their regular workday.  
 
30) Consent Process 
 
There are four groups of participants with different consent procedures.  
 
Clinicians and Agency Leaders interviewed in person will provide written or online consent 
using a REDCap consent form, obtained by the study PIs. Consent may be explained in a group 
setting, but individuals will be allowed to talk with the study PIs privately if they have any 
questions or concerns about the consent form. For the agency leader and clinician group 
interviews, if individuals are interviewed via phone or video conference, they will be sent the 
consent form ahead of time, and provided the opportunity to ask questions before or at the 
interview. For the triad interview participants, we will obtain verbal consent at the start of the 
interview. We are requesting a waiver of written consent for these individuals, given the low risk 
nature of this data collection and logistic difficulties of obtaining written consent remotely.  
 

 
 
Page 23 of 30 
  
Clinicians for the pre-pilot will be contacted by the study PIs, who will do consent with 
them individually by phone or video chat. For the full trial, consent forms will be sent out 
by e-mail to all clinicians prior to the first training in Mirah and they will be asked to 
contact the study if they have any questions. If new clinicians join the agency during the 
study, they will be contacted by e-mail and consented at that point. A new consent will also 
be administered electronically together with the follow-up clinician survey. We are also 
requesting a waiver of signed consent for the clinicians taking part in the Aim 1 pre-pilot and the 
full trial because they may need to be consented remotely due to COVID-19 and because many 
activities involve electronic surveys. We will obtain consent via REDCAP.   
 
Clients and Caregivers from the General Patient Population. We are requesting a waiver of 
consent for the individuals receiving MBC as part of the agency-wide implementation of MBC. 
Because the agency is using MBC agency-wide as part of their routine services, is interested in 
randomization to two different forms of MBC as a program evaluation question, and is only 
providing limited identifier datasets to the researchers, the study presents minimal risk to these 
participants. Obtaining consent would increase the study risk because the study would receive 
identifying information from participants.  
 
Clients and Caregivers in the Research Subsample. At least 420 clients and caregivers will  
complete additional research measures via the Mirah system during the Aim 1c pre-pilot and the 
Aim 2 trial.  Consent procedures will differ slightly for the two phases of the study. For the pre-
pilot, caregivers will view a consent form and youth an assent form the first time they complete 
these measures in the Mirah system. If they have questions before participating, they will have 
the option to provide a phone number and ask for all call from study staff before deciding. For 
the aim 2 Research Subsample, which involves a much larger sample of youth being recruited 
when they start treatment, youth and caregivers will view recruitment information when they 
complete their clinic intake questionnaires via Mirah. If they click that they are interested in the 
study, they will proceed to a screen with the consent/assent document. After reading the 
document, they will indicate yes, no, or “I have questions.” If they have questions, they will next 
proceed to a Frequently Asked Questions (Research Subsample Frequently Asked Questions 
Page, uploaded in consent materials) page to see if their question is there. If not, they will be 
provided with the project phone number to call to ask questions. If they indicate at the intake that 
they have questions and do not consent, the study recruitment and consent/assent information 
will be presented one more time prior to their completion of the Mirah measures for their first 
session. If they do not agree to be in the study at that point, they will no longer be eligible to be 
part of the research subsample. Because youth and caregivers receive different links to complete 
their Mirah questionnaires, they will be asked to assent and consent separately, and it is possible 
that youth may consent prior to their caregivers consenting. However, neither youth nor 
caregivers will complete and research measures unless both have consented/assented. We are 
requesting a waiver of signed consent for both the pre-pilot and research subsample subsample 
participants. For the pre-pilot obtaining written consent would increase the study risk by 
providing identifying information to the study team. For the research subsample, the research 

 
 
Page 24 of 30 
  
team will obtain identifying information from participants in order to give them gift cards, but 
the study is online, so it is not feasible to get signed consent and the study is minimal risk.  
 
Clients and Caregivers Taking Part in the Qualitative Study Activities. Clients and caregivers 
taking part in the triad interviews will provide verbal consent and assent. Consent and assent will 
be obtained privately by study staff at the time of the interviews. We are requesting a waiver of 
written consent because interviews are taking place remotely.  
 
31) Process to Document Consent in Writing 
As detailed under #30, we are requesting a waiver of written consent for some 
participants and a waiver of consent for others.  
32) 
  Authorization for Use and Disclosure of Protected Health Information 
(HIPAA)  
 
 
Type of Request: 
☐ Waiver of Authorization for access to medical record for subject identification/recruitment. 
X Waiver of Authorization for access to medical record to obtain data for the research. 
 
Confirm that you will destroy or de-identify the information you collect at the earliest 
opportunity.  
X  I confirm 
 
Confirm that the information you collect will not be reused or disclosed to any other person or 
entity, except as required by law, for authorized oversight of the research study or for other 
research for which the use or disclosure of PHI is permissible.          X   I confirm          
 
 
33) Drugs or Devices 
N/A 
 
References 
 
1. 
National Research Council and Institute of Medicine. Preventing Mental, Emotional, and 
Behavioral Disorders Among Young People: Progress and Possibilities. Washington, 
DC2009. 
2. 
Eisenberg D, Neighbors K. Economics of preventing mental disorders and substance 
abuse among young people. Paper commissioned by the Committee on Prevention of 
Mental Disorders and Substance Abuse Among Children, Youth, and Young Adults: 
Research Advances and Promising Interventions, Board on Children, Youth, and 
Families, National Research Council and the Institute of Medicine, Washington, DC. 
2007. 

 
 
Page 25 of 30 
  
3. 
Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-
Use Conditions: Quality Chasm Series National Academies Press2006  
4. 
Weissman MM, Wolk S, Goldstein RB, et al. Depressed adolescents grown up. Jama. 
1999;281(18):1707-1713. 
5. 
McLeod JD, Uemura R, Rohrman S. Adolescent Mental Health, Behavior Problems, and 
Academic Achievement. Journal of health and social behavior. 2012;53(4):482-497. 
6. 
Caspi A, Wright BRE, Moffitt TE, Silva PA. Early failure in the labor market: Childhood 
and adolescent predictors of unemployment in the transition to adulthood. American 
sociological review. 1998:424-451. 
7. 
Farmer EMZ. Extremity of externalizing behavior and young adult outcomes. Child 
Psychology & Psychiatry & Allied Disciplines. 1995;36(4):617-632. 
8. 
Institute of Medicine. Psychosocial Interventions for Mental and Substance Use 
Disorders: A Framework for Establishing Evidence-Based Standards. Washington, DC: 
National Academies Press; 2015. 
9. 
Weisz JR, Kuppens S, Eckshtain D, Ugueto AM, Hawley KM, Jensen-Doss A. 
Performance of evidence-based youth psychotherapies compared with usual clinical care: 
A multilevel meta-analysis. JAMA Psychiatry. 2013;70(7):750-761. 
10. 
Scott K, Lewis CC. Using measurement-based care to enhance any treatment. Cognitive 
and Behavioral Practice. 2015;22(1):49-59. 
11. 
Bickman L. Why don't we have effective mental health services? Administration and 
Policy in Mental Health and Mental Health Services Research. 2008;35:437-439. 
12. 
Kelley SD, Bickman L. Beyond outcomes monitoring: Measurement feedback systems in 
child and adolescent clinical practice. Current Opinion in Psychiatry. 2009;22(4):363-
368. 
13. 
Knaup C, Koesters M, Schoefer D, Becker T, Puschner B. Effect of feedback of 
treatment outcome in specialist mental healthcare: Meta-analysis. The British Journal of 
Psychiatry. 2009;195(1):15-22. 
14. 
Krägeloh CU, Czuba KJ, Billington DR, Kersten P, Siegert RJ. Using feedback from 
patient-reported outcome measures in mental health services: A scoping study and 
typology. Psychiatric Services. 2015;66(3):224-241. 
15. 
Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it 
time for clinicians to routinely track patient outcome? A meta-analysis. Clinical 
Psychology: Science and Practice. 2003;10(3):288-301. 
16. 
Shimokawa K, Lambert MJ, Smart DW. Enhancing treatment outcome of patients at risk 
of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality 
assurance system. J Consult Clin Psych. 2010;78(3):298-311. 
17. 
Tam HE, Ronan K. The application of a feedback-informed approach in psychological 
service with youth: Systematic review and meta-analysis. Clinical Psychology Review. 
2017;55:41-55. 
18. 
De Jong K. Effectiveness of progress monitoring: A meta-analysis. The 48th 
International Annual Meeting for the Society for Psychotherapy Research; 2017; 
Toronto, CA. 
19. 
Carlier IVE, Meuldijk D, Van Vliet IM, Van Fenema E, Van der Wee NJA, Zitman FG. 
Routine outcome monitoring and feedback on physical or mental health status: evidence 
and theory. Journal of Evaluation in Clinical Practice. 2012;18:104-110. 

 
 
Page 26 of 30 
  
20. 
Gondek D, Edbrooke-Childs J, Fink E, Deighton J, Wolpert M. Feedback from outcome 
measures and treatment effectiveness, treatment efficiency, and collaborative practice: A 
systematic review. Administration and Policy in Mental Health and Mental Health 
Services Research. 2016;43(3):325-343. 
21. 
Douglas S, Button S, Casey SE. Implementing for sustainability: Promoting use of a 
measurement feedback system for innovation and quality improvement. Administration 
and Policy in Mental Health and Mental Health Services Research. 2016;43(3):286-291. 
22. 
Jensen-Doss A, Becker EM, Smith AM, et al. Monitoring treatment progress and 
providing feedback is viewed favorably but rarely used in practice. Administration and 
Policy in Mental Health and Mental Health Services Research. in press. 
23. 
Borntrager CF, Chorpita BF, Higa-McMillan C, Weisz JR. Provider attitudes toward 
evidence-based practices: are the concerns with the evidence or with the manuals? 
Psychiatric Services. 2009;60(5):677-681. 
24. 
Chorpita BF, Bernstein A, Daleiden EL. Driving with roadmaps and dashboards: Using 
information resources to structure the decision models in service organizations. 
Administration and Policy in Mental Health and Mental Health Services Research. 
2008;35(1-2):114-123. 
25. 
Douglas SR, Jonghyuk B, Andrade ARVd, Tomlinson MM, Hargraves RP, Bickman L. 
Feedback mechanisms of change: How problem alerts reported by youth clients and their 
caregivers impact clinician-reported session content. Psychotherapy Research. 
2015;25:678-693. 
26. 
Amble I, Gude T, Stubdal S, Andersen BJ, Wampold BE. The effect of implementing the 
Outcome Questionnaire-45.2 feedback system in Norway: A multisite randomized 
clinical trial in a naturalistic setting. Psychotherapy Research. 2015;25(6):669-677. 
27. 
De Jong K, Timman R, Hakkaart-Van Roijen L, et al. The effect of outcome monitoring 
feedback to clinicians and patients in short and long-term psychotherapy: A randomized 
controlled trial. Psychotherapy Research. 2014;24(6):629-639. 
28. 
de Jong K, van Sluis P, Nugter MA, Heiser WJ, Spinhoven P. Understanding the 
differential impact of outcome monitoring: Therapist variables that moderate feedback 
effects in a randomized clinical trial. Psychotherapy Research. 2012;22(4):464-474. 
29. 
Newnham EA, Hooke GR, Page AC. Progress monitoring and feedback in psychiatric 
care reduces depressive symptoms. Journal of Affective Disorders. 2010;127(1-3):139-
146. 
30. 
Yeung AS, Jing Y, Brenneman SK, et al. Clinical outcomes in measurement‐based 
treatment (COMET): A trial of depression monitoring and feedback to primary care 
physicians. Depression and Anxiety. 2012;29(10):865-873. 
31. 
Hansson H, Rundberg J, Österling A, Öjehagen A, Berglund M. Intervention with 
feedback using Outcome Questionnaire 45 (OQ-45) in a Swedish psychiatric outpatient 
population. A randomized controlled trial. Nordic Journal of Psychiatry. 2013;67(4):274-
281. 
32. 
Hawkins EJ, Lambert MJ, Vermeersch DA, Slade K, Tuttle KC. The therapeutic effects 
of providing patient progress information to therapists and patients. Psychother Res 
2004;14(3):308-327. 
33. 
Lambert MJ, Hansen NB, Finch AE. Patient-focused research: Using patient outcome 
data to enhance treatment effects. Journal of Consulting and Clinical Psychology. 
2001;69(2):159-172. 

 
 
Page 27 of 30 
  
34. 
Lambert MJ, Whipple JL, Vermeersch DA, et al. Enhancing psychotherapy outcomes via 
providing feedback on client progress: A replication. Clinical Psychology and 
Psychotherapy. 2002;9:91-103. 
35. 
Mathias SD, Fifer SK, Mazonson PD, Lubeck DP, Buesching DP, Patrick DL. Necessary 
but not sufficient: The effect of screening and feedback on outcomes of primary care 
patients with untreated anxiety. Journal Of General Internal Medicine. 1994;9(11):606-
615. 
36. 
Schmidt U, Landau S, Pombo-Carril MG, et al. Does personalized feedback improve the 
outcome of cognitive-behavioural guided self-care in bulimia nervosa? A preliminary 
randomized controlled trial. British Journal of Clinical Psychology. 2006;45(1):111-121. 
37. 
Bickman L, Douglas SR, De Andrade ARV, et al. Implementing a measurement feedback 
system: A tale of two sites. Administration and Policy in Mental Health and Mental 
Health Services Research. 2016;43(3):410-425. 
38. 
Bickman L, Kelley SD, Breda C, De Andrade ARV, Riemer M. Effects of routine 
feedback to clinicians on mental health outcomes of youths: Results of a randomized 
trial. Psychiatric Services. 2011;62(12):1423-1429. 
39. 
Kluger AN, DeNisi A. The effects of feedback interventions on performance: A historical 
review, a meta-analysis, and a preliminary feedback intervention theory. Psychological 
Bulletin. 1996;119:254-284. 
40. 
Priebe S, McCabe R, Bullenkamp J, et al. The impact of routine outcome measurement 
on treatment processes in community mental health care: approach and methods of the 
MECCA study. Epidemiology and Psychiatric Sciences. 2002;11:198-205. 
41. 
Riemer M, Rosof-Williams J, Bickman L. Theories related to changing clinician practice. 
Child and Adolescent Psychiatric Clinics of North America. 2005;14(2):241-254. 
42. 
Crawford MJ, Robotham D, Thana L, et al. Selecting outcome measures in mental health: 
The views of service users. Journal Of Mental Health (Abingdon, England). 
2011;20(4):336-346. 
43. 
Eisen SV, Ranganathan G, Seal P, Spiro A. Measuring clinically meaningful change 
following mental health treatment. The journal of behavioral health services & research. 
2007;34:272–289. 
44. 
Fossey EM, Harvey CA. A conceptual review of functioning: Implications for the 
development of consumer outcome measures. Australian & New Zealand Journal of 
Psychiatry. 2001;35:91-98. 
45. 
Rickwood DJ, Mazzer KR, Telford NR, Parker AG, Tanti CJ, McGorry PD. Changes in 
psychological distress and psychosocial functioning in young people visiting headspace 
centres for mental health problems. The Medical Journal of Australia. 2015;202:537-542. 
46. 
Wyrwich KW, Bullinger M, Aaronson N, Hays RD, Patrick DL, Symonds T. Estimating 
clinically significant differences in quality of life outcomes. Quality of Life Research. 
2005;14:285-295. 
47. 
Bergman H, Kornør H, Nikolakopoulou A, et al. Client feedback in psychological 
therapy for children and adolescents with mental health problems. Cochrane Database of 
Systematic Reviews. 2018(8):Art. No.: CD011729. 
48. 
Anker MG, Duncan BL, Sparks JA. Using client feedback to improve couple therapy 
outcomes: A randomized clinical trial in a naturalistic setting. Journal of Consulting and 
Clinical Psychology. 2009;77(4):693-704. 

 
 
Page 28 of 30 
  
49. 
Crits-Christoph P, Ring-Kurtz S, Hamilton JL, et al. A preliminary study of the effects of 
individual patient-level feedback in outpatient substance abuse treatment programs. 
Journal of Substance Abuse Treatment. 2012;42(3):301-309. 
50. 
Connolly Gibbons MB, Kurtz JE, Thompson DL, et al. The effectiveness of clinician 
feedback in the treatment of depression in the community mental health system. Journal 
of Consulting and Clinical Psychology. 2015;83(4):748-759. 
51. 
Harmon SC, Lambert MJ, Smart DM, et al. Enhancing outcome for potential treatment 
failures: Therapist–client feedback and clinical support tools. Psychotherapy Research. 
2007;17(4):379-392. 
52. 
Murphy KP, Rashleigh CM, Timulak L. The relationship between progress feedback and 
therapeutic outcome in student counselling: A randomised control trial. Counselling 
Psychology Quarterly. 2012;25(1):1-18. 
53. 
Priebe S, McCabe R, Bullenkamp J, et al. Structured patient-clinician communication and 
1-year outcome in community mental healthcare: Cluster randomised controlled trial. The 
British Journal of Psychiatry. 2007;191(5):420-426. 
54. 
Probst T, Lambert MJ, Loew TH, Dahlbender RW, Göllner R, Tritt K. Feedback on 
patient progress and clinical support tools for therapists: Improved outcome for patients 
at risk of treatment failure in psychosomatic in-patient therapy under the conditions of 
routine practice. Journal of Psychosomatic Research. 2013;75(3):255-261. 
55. 
Reese RJ, Norsworthy LA, Rowlands SR. Does a continuous feedback system improve 
psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training. 
2009;46(4):418-431. 
56. 
Rise MB, Eriksen L, Grimstad H, Steinsbekk A. The short-term effect on alliance and 
satisfaction of using patient feedback scales in mental health out-patient treatment. A 
randomised controlled trial. BMC Health Services Research. 2012;12:348-348. 
57. 
Simon W, Lambert MJ, Harris MW, Busath G, Vazquez A. Providing patient progress 
information and clinical support tools to therapists: Effects on patients at risk of treatment 
failure. Psychotherapy Research. 2012;22(6):638-647. 
58. 
Simon W, Lambert MJ, Busath G, et al. Effects of providing patient progress feedback 
and clinical support tools to psychotherapists in an inpatient eating disorders treatment 
program: A randomized controlled study. Psychotherapy Research. 2013;23(3):287-300. 
59. 
Slade K, Lambert MJ, Harmon SC, Smart DW, Bailey R. Improving psychotherapy 
outcome: The use of immediate electronic feedback and revised clinical support tools. 
Clinical psychology & psychotherapy. 2008;15(5):287-303. 
60. 
Whipple JL, Lambert MJ, Vermeersch DA, Smart DW, Nielsen SL, Hawkins EJ. 
Improving the effects of psychotherapy: The use of early identification of treatment and 
problem-solving strategies in routine practice. Journal of Counseling Psychology. 
2003;50(1):59-68. 
61. 
Fortney J, Sladek R, Unutzer J, et al. Fixing Behavioral Health Care in America: A 
National Call for Measurement-Based Care in the Delivery of Behavioral Health 
Services. The Kennedy Forum; 2015 2015. 
62. 
Fortney JC, Unützer J, Wrenn G, et al. A tipping point for measurement-based care. 
Psychiatric Services. 2017;68(2):179-188. 
63. 
Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome 
monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy 
research. 2015;25(1):6-19. 

 
 
Page 29 of 30 
  
64. 
Greenhalgh J, Dalkin S, Gooding K, et al. Functionality and feedback: a realist synthesis 
of the collation, interpretation and utilisation of patient-reported outcome measures data 
to improve patient care. Health Services and Delivery Research. 2017;5:1-280. 
65. 
Mellor-Clark J, Cross S, Macdonald J, Skjulsvik T. Leading Horses to water: Lessons 
from a decade of helping psychological therapy services use routine outcome 
measurement to improve practice. Administration and Policy in Mental Health and 
Mental Health Services Research. 2016;43(3):279-285. 
66. 
Gleacher AA, Olin SS, Nadeem E, et al. Implementing a measurement feedback system 
in community mental health clinics: A case study of multilevel barriers and facilitators. 
Administration and Policy in Mental Health and Mental Health Services Research. 
2016;43:426-440. 
67. 
Duncan EA, Murray J. The barriers and facilitators to routine outcome measurement by 
allied health professionals in practice: a systematic review. BMC health services 
research. 2012;12(1):96. 
68. 
Fitzpatrick M. Blurring practice–research boundaries using progress monitoring: A 
personal introduction to this issue of Canadian Psychology. Canadian 
Psychology/Psychologie canadienne. 2012;53(2):75. 
69. 
Martin A-M, Fishman R, Baxter L, Ford T. Practitioners’ attitudes towards the use of 
standardized diagnostic assessment in routine practice: a qualitative study in two child 
and adolescent mental health services. Clinical child psychology and psychiatry. 
2011;16(3):407-420. 
70. 
Wolpert M, Curtis-Tyler K, Edbrooke-Childs J. A qualitative exploration of patient and 
clinician views on patient reported outcome measures in child mental health and diabetes 
services. Administration and Policy in Mental Health and Mental Health Services 
Research. 2016;43(3):309-315. 
71. 
Lewis CC, Puspitasari A, Boyd MR, et al. An implementation-science informed narrative 
review of measurement based care in mental and behavioral health. . JAMA Psychiatry. in 
press. 
72. 
Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing 
clinical innovations in health and mental health. Medical Care Research and Review. 
2012;69(2):123-157. 
73. 
Powell BJ, Proctor EK, Glass JE. A systematic review of strategies for implementing 
empirically supported mental health interventions. Research on Social Work Practice. 
2014;24(2):192-212. 
74. 
Proctor EK, Powell BJ, Feely MA. Measurement in dissemination and implementation 
science. In: Beidas RS, Kendall PC, Beidas RS, Kendall PC, eds. Dissemination and 
implementation of evidence-based practices in child and adolescent mental health. New 
York, NY, US: Oxford University Press; 2014:22-43. 
75. 
Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O. Diffusion of Innovations 
in Service Organizations: Systematic Review and Recommendations. Milbank Quarterly. 
2004;82(4):581-629. 
76. 
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering 
implementation of health services research findings into practice: a consolidated 
framework for advancing implementation science. Implement Sci. 2009;4(1):50. 

 
 
Page 30 of 30 
  
77. 
Liberati EG, Ruggiero F, Galuppo L, et al. What hinders the uptake of computerized 
decision support systems in hospitals? A qualitative study and framework for 
implementation. Implementation Science. 2017;12(1):113.