Increasing Accuracy and Efficiency of Fidelity Measurement in CBT
Phase: NA
Conditions: Assess Fidelity to Cognitive-behavioral Therapy for Youth
Sponsor: Northwestern University

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Official Study Title: Increasing Accuracy and Efficiency of Fidelity Measurement in CBT 
 
Brief Title: Project FACTS (Fidelity Accuracy: Comparing Three Strategies) 
 
NCT02820623 
 
May 11, 2020 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 2 of 13 
 
Executive Summary 
 
1. Description of theory, methodology, intent of study 
Our objective is to compare the accuracy, costs, and cost-effectiveness of three fidelity 
measurement methods to assess fidelity to cognitive-behavioral therapy for youth. We will 
randomize up to 170 therapists, implementing cognitive-behavioral therapy to 3 conditions: self-
report, chart stimulated recall, and behavioral rehearsal (N = about 56 for each group). To calculate 
our outcomes of interest, each condition will be compared to the gold-standard fidelity 
measurement method, direct observation. 
 
OBJECTIVES 
Objective 1: To identify the most accurate fidelity measurement method. 
Objective 2: To estimate the economic costs and cost-effectiveness of the fidelity measurement 
methods. 
Objective 3: To compare stakeholders’ motivation to use each method, as well as identify their 
perceived barriers and facilitators to use of each method. 
 
This study will have a significant positive impact in two ways. First, it will validate fidelity 
measurement methods that can then be used by implementation scientists for research. Second, it 
will produce tools that can be used by community mental health clinics to monitor therapist 
fidelity, an indicator of therapy quality. 
 
BACKGROUND 
The significance of this study is its potential to reduce the research-to-practice gap by identifying 
accurate and cost-effective fidelity measurement methods that both researchers and stakeholders 
in community mental health clinics can use to monitor fidelity of evidence-based practices. This 
will allow for the public health impact of evidence-based psychosocial interventions to reach its 
potential. Specifically, we use the exemplar treatment, cognitive-behavioral therapy (CBT), an 
established evidence-based practice for a wide range of child and adult psychiatric disorders, to 
test the three fidelity measurement methods. The proposed research is based on a framework which 
identifies fidelity as a key implementation outcome.  
 
Therapist fidelity is frequently measured due to its mediating role between implementation of 
evidence-based practices and client outcomes. Fidelity can also be conceptualized as an indicator 
of quality of care, which is a priority for many national organizations such as the Institute of 
Medicine. Fidelity includes adherence (how closely the components of a protocol are followed) 
and competence (how skillfully the components are implemented and therapist responsiveness to 
the child). The gold-standard fidelity measurement technique is direct observation, which is 
infeasible in community agencies due to lack of recording devices, time, and trained coders. 
Community mental health clinics sorely need methods of measuring fidelity that are accurate and 
cost-effective. Accurate methods refer to those with strong reliability and validity, whereas cost-
effective methods refer to those that can be used with the limited resources in routine practice 
settings while providing benefit.   
 
Therapists’ self-report of their own fidelity is easy to administer, brief, and low-cost. However, 
the correspondence between self-report and independent observers is low (e.g., kappa of .22 in one 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 3 of 13 
 
study). Leading scholars have suggested that self-report is frequently inaccurate because self-
report measures typically include many terms that are unfamiliar to therapists; and because 
therapists are not trained in rating their own behavior (i.e., how to rate high vs. low behavior). One 
way to address these concerns is to ensure that the operational definitions of each item are clear 
and to train therapists in how to rate their own behavior. Additionally, two innovative methods 
from the medical literature, chart-stimulated recall and behavioral rehearsal, hold promise for 
fidelity measurement. Chart-stimulated recall entails a brief interview with a clinician about the 
care provided to a client, while allowing the clinician to review the client’s file to assist recall. 
Behavioral rehearsal refers to a simulated interaction between a clinician and another individual 
around treatment delivery. Our goal is to test the accuracy, costs, and cost-effectiveness of each of 
these three methods in comparison to direct observation.  
 
STUDY SAMPLE AND DATA SOURCE 
The City of Philadelphia Department of Behavioral Health and Intellectual disAbility Services 
(DBHIDS) has sponsored multiple initiatives to train therapists at community mental health clinics 
in the City of Philadelphia to administer CBT for a range of youth psychiatric disorders (e.g., Beck 
initiative, trauma initiative). As part of these initiatives, more than 300 therapists from 30 agencies 
have been trained in CBT over the past seven years, and 40-60 new therapists in two to five new 
agencies are added annually. Therapists participate in training as part of their agency employment; 
they will not be required to participate in this study to receive training.  
 
Organizations. To identify organizations who have participated in these initiatives, we worked 
with DBHIDS leadership. We then reached out to each agency to ascertain their interest in 
potentially participating in this study (see attached Letters of Support from 12 agencies). Every 
agency that we reached out to agreed to participate. The study sample will include therapists 
employed by these 12 agencies who implement CBT.  
 
Therapists. We will recruit up to 170 therapists who provide mental health treatment services in at 
least 12 community mental health agencies in the City of Philadelphia who (1) have been trained 
in CBT through the Beck Initiative and/or the trauma initiative, (2) respond to a brief screening 
survey that they use CBT; and/or (3) are nominated by a supervisor as a therapist who uses CBT. 
We anticipate that therapists will be 80% female and will be 55% Caucasian, 25% African 
American, 10% Latino, 5% Asian and 5% of other races/ethnicities. 
 
Supervisors. We will recruit 12 individuals in leadership positions (supervisors and administrators) 
in community mental health agencies in the City of Philadelphia.  
 
Children, Youth, and their legal guardian(s). We will recruit approximately 3 youth clients per 
therapist for a total of 405 youth clients (if younger than 18 years old, we will also obtain written 
or verbal consent from the client’s legal guardian)*. We base our projections for youth clients on 
administrative data from Medicaid claims. Youth clients range in age from 7-24 with a variety of 
presenting disorders. Based on these data, we anticipate that youth will be 55% male, and primarily 
African American or Hispanic/Latino. Any randomly selected child aged 7-24 who is presenting 
for treatment for any psychosocial difficulties may participate if appropriate assent and/or consent 
is ascertained. Exclusion criteria include if the child does not have a legal guardian who is able to 
consent (e.g., DHS is the guardian).  

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 4 of 13 
 
 
*Note, our intention is to enroll three youth clients per therapist. However, we will ask therapists 
immediately following the session if this was a “crisis” session or therapy as usual. We are looking 
for at least three samples of behavior that are not crisis sessions so we can code their behavior; 
therefore, we may need to enroll more than three clients per therapist if there are sessions that are 
deemed to not meet criteria for inclusion. Sessions need to include at least ten minutes where the 
client is in the room (i.e., guardian only sessions will not be counted towards the three sessions 
needed for this study).  
 
The study will begin human subject activities in Fall 2016 (following full approval from the City 
of Philadelphia IRB) and continue enrolling human subjects until mid-2020.  
 
PROCEDURES 
Aim 1: We will recruit therapists in the City of Philadelphia who (1) have been trained in CBT 
through a city sponsored CBT initiative, (2) respond to a brief screening survey that they use CBT; 
and/or (3) are nominated by a supervisor as a therapist who uses CBT. More than 300 therapists 
from 30 agencies have been trained over the past 7 years, and 40-60 new therapists in 2-5 new 
agencies are added annually. Therapists participate in training as part of their agency employment; 
they will not be required to participate in this study to receive training. Therapists, agency 
leadership, and DBH have expressed considerable enthusiasm for the proposed study. We have 
included letters of support from agency administrators as well as the DBHIDS Commissioner, Dr. 
Arthur Evans.  
 
Therapists will be consented during a study meeting at each agency where we will provide lunch 
and information about the study. At that meeting, the manager of research projects, with the 
assistance of the biostatistician, Dr. Marcus, will randomize therapists into one of the three 
measurement arms. To maintain balance, we will ensure that we have equal numbers of therapists 
in each study arm in each site.  
 
Therapists report approximately 30 client encounters per week. We base our projections for youth 
clients on administrative data from Medicaid claims. On average, youth attend 5.7 sessions (SD = 
8.6). Our recruitment goal is 3 unique encounters (i.e., separate clients who each had a CBT 
session) per therapist for a total of 405 youth client encounters. We will use sequential sampling 
to identify client encounters to include in the trial. We will sample as a function of individual client 
session number to ensure that we enroll clients receiving services across the treatment span because 
CBT techniques used in sessions early on in treatment are different than those used later in 
treatment. For example, treatment of anxious youth would typically begin with psychoeducation 
and rapport building in initial sessions before progressing to exposure strategies later in treatment. 
Once therapists are enrolled in the study, our research staff will work with the therapists directly 
each week to provide the information they need to check in with eligible families for participation 
(e.g., recruitment notice, script to obtain verbal assent/consent for the research team to approach 
the family). Because of our desire to randomly sample treatment sessions from different treatment 
stages (2-9 (early treatment) and 10+ (late treatment), we will first randomly select the days the 
research staff will be present at the agency and then randomly select the blocks of time within each 
day (per each therapist’s schedule) during which the research staff will enroll clients for each 
therapist. For instance, if we randomly select Monday, Tuesday and Wednesday as the days the 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 5 of 13 
 
research staff will be at the agency that week and a therapist participant planned to be at the agency 
from 11 AM to 7 PM each of those days, then we would randomly select either the first block (11 
AM to 3 PM) or the second block (3 PM to 7 PM) of time in which to approach clients of that 
therapist. Before each block of time begins, the research staff will check in with the therapist to 
determine how many of the clients scheduled for that block of time are eligible (i.e. youth clients 
7-24 years with whom the therapist uses CBT strategies). We will also ask the therapist to indicate 
if the client is in early or late treatment stages. We will ask therapists to introduce the study to 
eligible clients within each randomly selected block of time until three clients have been enrolled. 
We will provide $25 compensation to complete this process until 3 separate clients who each had 
a CBT session are enrolled (we will not include if the therapist identifies the session as a crisis of 
the week session). Based on conversations with our community advisory board, we have been 
advised that we will need to be flexible at each agency with regard to how we ascertain this 
information.  
 
We will not have research staff directly approach families to avoid confidentiality concerns. 
Therapists will approach the family and obtain verbal consent and assent for the research team to 
contact the family. Any randomly selected child aged 7-24 who is presenting for treatment for any 
psychosocial difficulties may participate. Exclusion criteria include if the child does not have a 
legal guardian who can consent to their participation (e.g., DHS-involved youth). Children and 
their families will be recruited and retained with the help of study team members who will be 
embedded within agencies where enrolled therapists are seeing eligible clients. When study team 
members learn of a consent to contact from the therapist, they obtain verbal or written consent and 
assent from the guardian and client to record the encounter. If the therapist and client both request 
that a research team member sit with the recorders during the session, then a research team member 
will sit in the room and hold the recorder. Youth will be paid $10 for allowing their therapy session 
to be audio-taped. Study participation for youth only entails recording their one session.  
 
Due to COVID-19, we have also decided to implement remote recruitment of therapists and 
clients. Research staff will recruit therapists by emailing them directly and introducing the study. 
If therapists are interested in participating in the study, research staff will consent therapists over 
the phone. Once consented, research staff will work with therapists to identify eligible clients using 
sequential sampling, which will ensure that we enroll clients receiving services across the 
treatment span. Therapists will introduce the study to eligible clients (and if necessary, guardians) 
via a virtual platform to obtain verbal consent (and if necessary, assent). If the client (and if 
necessary, guardian) are interested, research staff will enroll and consent clients through a virtual 
platform. Research staff will place the audio recorder next to the computer in a private room, press 
“record” on the audio recorder, close the door and leave the room. Therapists will notify the 
research staff through text message once the session is over. Youth will be paid with an electronic 
$10 Amazon gift card for allowing their therapy session to be audio-taped.  
 
All three conditions (described below) will be compared with direct observation (i.e., audio-
recordings of each session). Therapists will be randomized to one of the three experimental 
measurement conditions. Therapists in the self-report condition will be asked to complete the self-
report measure within 48 hours of the encounter, to ensure appropriate recall and also because that 
is typically when therapists are expected to complete their notes (personal communication, 
Gretchen Murchison, Community Behavioral Health). Therapists in the other two conditions will 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 6 of 13 
 
complete chart-stimulated recall or behavioral rehearsal within one week of the encounter. We 
selected one week because therapists report receiving one hour of weekly supervision, suggesting 
that this time period is best in planning for future potential trials integrating these methods into 
community supervision. Therapists will be paid upon completion of their fidelity method (i.e., self-
report, chart stimulated recall, behavioral rehearsal). Once a therapist enrolls three separate clients 
who each had a CBT session and has completed the fidelity measurement for each client, they have 
completed participation in this aim of the study (we estimate it will take a month to enroll three 
clients who each had a CBT session and complete fidelity measurements). More than three clients 
may be enrolled if a session was recorded and we find out that CBT was not used (e.g., the therapist 
thought CBT would be used in the session but it ended up being a crisis management session).  
 
If assigned to the self-report condition, therapists will be asked to complete a brief self-report 
measure on their own time within 48 hours after each recorded session (once for each client; three 
times total; anticipated time = 30-60 minutes). Therapists can complete this measure via REDCap, 
a HIPAA compliant platform, or using paper-and-pencil, whichever is their preference. Therapists 
will be compensated $50 for the completion of the self-report measure for their three clients. 
Additionally, to learn how to use this self-report measure, the research staff will ask therapists to 
engage in up to 30 minutes of training, for which they will be compensated an additional $25.  
If assigned to the chart-stimulated recall condition, therapists will be asked to meet with research 
staff for one hour.  During that hour, therapists will bring the chart of each client that was recorded 
for a brief interview (approximately 20 minutes per client session). In this interview, therapists 
will be asked questions about different techniques they may have used in session with their client. 
Therapists will be allowed to use the chart for the client to spur their memory of the session but 
the research staff will not be allowed to view the chart. The research staff will ask to complete this 
interview with therapists within one week after the session was recorded with the client. Therapists 
will be compensated $50 for the completion of chart-stimulated recall for three clients. 
 
If assigned to the behavioral rehearsal condition, therapists will be asked to meet with the research 
staff for one hour.  During that hour, they will participate in a role-play demonstrating the strategies 
they used in session with the three clients that were recorded. Research staff will ask to complete 
this meeting within one week after the session was recorded with the client. Therapists will be 
compensated $50 for the completion of behavioral rehearsal for three clients. If behavioral 
rehearsal takes longer than one hour, then the therapist will be compensated $25 for each additional 
half hour it takes to complete the condition.  
Aim 2: The research team will work closely with clinic leadership to collect cost data. The process 
begins by presenting the brief Drug Abuse Treatment Cost Analysis Program (DATCAP) materials 
to clinic leadership most familiar with the operations and financing of the program. After these 
personnel have reviewed the materials, conference calls are conducted between them and the 
researchers to formulate strategies for preliminary data collection and to answer questions. The 
researchers will provide clinic leadership with guidance regarding the type and source of resource 
use and cost information to gather. The researchers then conduct a site visit at each agency, which 
includes face-to-face meetings with clinic leadership and instrument completion. Cost analysis 
items include information about how much time is spent on each fidelity measurement method, 
the time it takes to develop the materials for the method, and the resources that are used for each 
condition (e.g., office room, computers). Please note, we have not included a consent form for Aim 2 
of the study  (i.e., assessing the cost-effectiveness of each fidelity measurement method) as this will be 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 7 of 13 
 
accomplished by using the DATCAP to collect administrative level data on each organization; absolutely 
no individual level or identifying information will be collected as part of this economic evaluation.  
 
Aim 3: Up to 170 therapists and 12 individuals in leadership positions (i.e., supervisors and 
administrators) will be recruited to participate in a brief quantitative survey during data collection 
in Aim 1 that will address Aim 3 objectives. For the qualitative data collection, we will use 
purposive sampling to recruit 48 individuals. The sample will include 36 therapists (12 therapists 
from each of the three conditions) and 12 individuals in leadership positions (supervisors and 
administrators). We selected 12 for each group of interest given research that suggests this is the 
ideal number for thematic saturation.  
 
Motivation will be quantitatively measured using a 7-point scale. This item will be compared for 
each of the three methods. We will ask therapists about their current and future motivation to use 
their assigned fidelity method. Because of concerns that willingness to adopt could be influenced 
by social interaction between therapists in various conditions, we will measure knowledge of the 
other conditions in the willingness to adopt measure as a potential covariate to be included in the 
mixed-effects multivariate regression models for Aim 3.  
 
Qualitative data will be collected using validated, standard open-ended questions that have been 
developed to elicit the beliefs underlying one’s motivation. For example, we will ask advantages 
and disadvantages the stakeholders attribute to using the particular method they were assigned to, 
as well as direct observation. We will also ask who they believe will approve and disapprove of 
them using a particular method. These and other questions will be asked to identify perceived 
barriers and facilitators influencing use of that method. We will also ask therapists and leadership 
to elaborate upon the quantitative scores provided on the willingness measure, and to generate 
information on their beliefs about the integration of these techniques in supervision in the future. 
All of the questions will be compiled in a questionnaire that will undergo cognitive response 
testing, which is a strategy that identifies ambiguous or awkward wording. Once the participant 
completes the qualitative interview (est. time, 1 hour), they have completed participation for this 
study.  
 
All research activities for Aims 1 and 3, including data collection and engagement with human 
subjects, will be conducted by members of the University of Pennsylvania research team. All Aim 
2 research activities will be conducted by members of the University of Pennsylvania research 
team as well as by co-investigators from Temple University. 
 
ANALYSIS PLAN  
All analyses are by intent-to-treat and will occur at the University of Pennsylvania Center for Mental 
Health Policy and Services Research under the direction of study biostatistician, Steven Marcus, PhD. 
Differences across the three arms of the study will be analyzed using SAS PROC MIXED. We 
will conduct three mixed-effects multivariate regression models. The dependent variable is 
difference in score between direct observation and the specific fidelity method under 
consideration. We will measure the dependent variables in three ways: raw score adherence, raw 
score competence, and total fidelity t-score (composite of adherence and competence raw score). 
We will conduct two sets of analyses. In the first set, the model will include two fixed effects 
(fidelity measurement method and site) and a random effect for therapist to account for the nesting 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 8 of 13 
 
of patients within therapists. Our primary interest is in the beta coefficient for the fidelity 
measurement method, which tells us whether there is any difference in accuracy between the 
intervention arms. Our first hypothesis is that both chart-stimulated recall and behavioral rehearsal 
(i.e., total fidelity t-score) will be more accurate than self-report. Our second hypothesis is that 
chart-stimulated recall will capture adherence best. Our third hypothesis is that behavioral 
rehearsal will capture competence best. To test these hypotheses, we will use the model described 
above to conduct custom contrasts in the regression models to examine whether the mean of the 
dependent variable differs for each pair-wise comparison of our intervention arms. We will then 
repeat that analysis after transforming the dependent variable into t-scores to allow computation 
of standardized differences for each pairwise comparison, so that we can rank accuracy. The 
second model will include an additional fixed-effect term for time and an interaction (time X 
fidelity measurement method), allowing us to examine if there are changes in outcome over time 
by measurement method. This is important in case there are differential practice effects across 
condition over time. 
 
2. Description of Philadelphia Department of Public Health, Department of Behavioral 
Health or Risk Management involvement 
 
This project is investigator-initiated. However, it is a result of many conversations with community 
stakeholders, including DBHDIS leadership. DBHIDS is not formally involved with this project 
and will not receive any identified data; however, we will share study findings to ensure that policy 
is shaped by the research, as consistent with best practices (e.g., present at Research Grand 
Rounds).  
 
DURATION OF STUDY 
Therapists will be randomized to one of the three experimental measurement conditions (self-
report, chart-stimulated recall, and behavioral rehearsal). Therapists in the self-report condition 
will be asked to complete the self-report measure within 48 hours of the encounter, to ensure 
appropriate recall and also because that is typically when therapists are expected to complete their 
notes (personal communication, Gretchen Murchison, Community Behavioral Health). Therapists 
in the other two conditions will complete chart-stimulated recall or behavioral rehearsal within one 
week of the encounter. We selected one week because therapists report receiving one hour of 
weekly supervision, suggesting that this time period is best in planning for future potential trials 
integrating these methods into community supervision. We anticipate that we will complete data 
collection for three therapists and their clients (three each; nine total clients) encounters each 
month. Therefore, each therapist participant will likely be enrolled in the study for a month; client 
participants will be enrolled for the one day that their therapy session is taped. Based on these 
projections, this study will take 45 months to complete. The study will likely begin human subjects 
activities in Fall 2016 and run until mid-2020. See a timeline of the proposed study below: 
 
Table 3.  
Activity 
Year 1 
(2016) 
Year 2 
Year 3 
Year 4 
Year 5 
Quarter 
Quarter 
Quarter 
Quarter 
Quarter 
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 
Preparation (i.e., hire staff, IRB approval, recruit agencies) 
• • 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Aim 1: Accuracy 
 
 
• 
• 
• 
• 
• 
• 
• 
• 
• 
 
 
 
 
 
 
 
 
 
Aim 2: Cost-Effectiveness 
 
 
 
 
• 
• 
• 
• 
• 
• 
• 
• 
• 
 
 
 
 
 
 
 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 9 of 13 
 
Aim 3: Motivation and Barriers/Facilitators 
 
 
 
 
 
 
 
 
 
 
 
• 
• 
• 
• 
• 
 
 
 
 
Data analysis 
 
 
 
 
 
 
 
 
 
 
 
• 
• 
• 
• 
• 
• 
• 
 
 
Manuscript preparation  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
• 
• 
• 
 
3. Aspects of research 
 
The findings from the research will be reported to the City of Philadelphia DBHIDS through 
Research Grand Rounds and agencies that provide child mental health services so that the 
stakeholders will have the opportunity to review and discuss this information. Further, we will 
prepare findings for publication.  
 
4. Risks to subjects 
 
SUBJECT RISKS 
There are no known physical or legal risks to participating in the study. Clinicians will be asked 
to answer questions about their practice and will also be observed in session. Leadership will be 
asked to report on cost information, as well as their perceptions of the fidelity measurement 
techniques. Self-report, brief observation, and semi-structured interviews could lead participants 
to feel temporarily uncomfortable. Dr. Beidas, a licensed clinical psychologist, will be available 
to speak with any participants who feel unduly distressed, and appropriate referrals will be made. 
Participants will not be required to participate and lack of participation will not impact their 
employment. Our experience in similar previous research is that therapists who are not interested 
in participating in the research will leave after hearing about the study. This does not impact their 
employment because we ask managers/leadership to leave the room after introducing us during the 
launch meeting. Supervisors will know who is participating but they will not explicitly know who 
declined to participate since they will not be present at the launch meeting. 
 
In the event that child abuse/neglect or suicidality/homicidality is identified during consent/assent 
or while listening to a session (either in person or during the coding of a session which typically 
occurs one month after the session), Dr. Beidas or a clinically trained member of the research team 
will follow up with the client’s therapist to ensure that a safety plan is in place and to work with 
the therapist to ensure that any information about child abuse/neglect or intent to harm self or 
others has been reported to the authorities, as required by law.  
 
In the event that serious concerns about client safety or therapist misconduct is developed during 
consent/assent or while listening to a session (either in person or during the coding of a session,  
i.e., the therapist promises not to disclose information that must be reported to authorities as 
required by law), Dr. Beidas or a clinically trained member of the research team will follow up 
with the therapist within two business days of becoming aware of the event to discuss the situation 
and develop a plan to ensure client safety and compliance with the law. If the study team is unable 
to reach the therapist after multiple attempts, they might also follow up with the therapist’s 
supervisor determine whether the event is reportable and to ensure that the report is filed if deemed 
appropriate.  
 
SUBJECT CONFIDENTIALITY 
As part of informed consent and assent, participants (i.e., therapists, legal guardians, and youth) 
will be informed that all information they provide will be kept confidential (unless it is determined 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 10 of 13 
 
that it must be reported as required by law), within the personnel of the Center for Mental Health 
Policy and Services Research (CMHPSR). No information gathered as part of this research will be 
shared with agency executive directors, supervisors, or DBHIDS (unless it is determined that 
certain information must be reported as required by law). Participants will be asked to sign a 
consent form that thoroughly describes the procedures to be followed in the study and the type of 
assessments involved. The Philadelphia Department of Public Health Institutional Review Board 
will have approved the protocol and consent prior to the initiation of the study. The following 
institutions will sign institutional authorization agreements so that the Philadelphia Department of 
Public Health Institutional Review Board can serve as the IRB of record: University of 
Pennsylvania, Temple University, University of Washington IRB, and the Medical University of 
South Carolina. The University of Pennsylvania has already agreed to cede to the Philadelphia 
Department of Public Health Institutional Review Board. We will submit these signed 
authorization agreements as a modification. Participants will be provided with copies of the signed 
consent forms while original copies of the signed consent forms will be kept in locked files at the 
CMHPSR which no one will have access to other than Dr. Beidas and the research team. All data 
will be coded with a subject identification number. The names that correlate to those numbers will 
be kept separate, such that any identifying information will be stored in one file, while 
experimental data will be kept in a separate file. Only Dr. Beidas and the research team will have 
access to these files. All project staff will complete training on confidentiality through the 
Collaborative IRB Training Initiative (CITI) course.  
 
All data, including the master list linking identifiers to the ID number and recordings will be 
destroyed in 2024. NIH policy requires that data be retained for a period of three years from the 
date of the final Federal Financial Report. The award period is from 2016-2021. That means that 
we must retain all study data until 2024, eight years following the start of the study. Assuming that 
data collection begins as planned in 2016, that means that the data of a youth who consented in 
2016 will be kept until 2024 (eight years later). 
 
Privacy and confidentiality is of the utmost importance given that a breach in privacy and/or 
confidentiality could result in serious consequences for participating clinicians (e.g., employment 
implications). Thus, breach of privacy and/or confidentiality is the biggest risk of participating in 
this study.  
 
To ensure privacy, self-report measures will be completed either by hand in the privacy of the 
therapist’s office or using the HIPAA compliant web-based survey platform, REDCap, hosted by 
the University of Pennsylvania. Therapists will have the option of completing measures using 
paper-and-pencil or via web given our previous experiences that therapists prefer different options. 
Paper-and-pencil measures, only identified with ID numbers, will be sealed in a manila envelope 
and handed to the embedded research staff person in the agency who will immediately transport 
the data to Penn. Semi-structured interviews will be conducted in a private location of the 
therapist’s preference. In our previous studies, this has typically been a closed room in the agency 
where the therapist works. The therapist meets with the interviewer one-on-one in this room and 
the content of the interview is recorded. To ensure recordings are kept confidential, the procedures 
described below will be followed. Once a session is audio-recorded, the research staff on site will 
take the recording device and plug it into a laptop that is connected to an encrypted external hard 
drive that is password-protected and helps make sure that only our team can access the information. 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 11 of 13 
 
The audio file will be transferred directly to the encrypted external hard drive and erased from the 
recording device. The research staff will transport the encrypted external hard drive back to the 
University of Pennsylvania, where the audio file will be uploaded to REDCap, a password-
protected HIPAA-compliant server. The audio file will be maintained in REDCap and on an 
encrypted external hard drive (inside a locked filing cabinet in a locked office) for three years after 
the study has ended. The audio file is later transcribed and de-identified by individuals who are 
trained in human subjects protection. Our research team will provide written feedback to each 
therapist on his/her recorded session behavior. This feedback will include a de-identified summary 
of the session, the areas of strength and potential growth, and the scores earned by the therapist as 
coded using the Therapy Process Observational Coding System for Child Psychotherapy. The 
research team will not share this individualized feedback with anyone else besides the therapist; 
the therapist is welcome to use the feedback in whatever manner they choose.  
 
If a therapist is randomized to chart-stimulated recall, a member of the study team will sit with the 
therapist while the therapist reviews the client’s mental health chart. The study team will not access 
any information from the client’s mental health chart. However, there is a small chance that the 
study team may see parts of the client’s mental health records during chart-stimulated recall. The 
study team does not intend to look at any parts of the client’s mental health records and will take 
steps to minimize seeing the record (e.g. choose not to sit facing the chart, ask the therapist not to 
show the chart to the study team).  
 
To ensure confidentiality, we will use the following methods: (a) all self-report data, rating scales, 
observations, and interviews will be kept in a locked filing cabinet; (b) subject identity will be 
masked using numeric codes and password-protected master list which only Dr. Beidas and the 
manager of research projects will have access to; (c) data will be entered directly into password-
protected files which only Dr. Beidas and the manager of research projects will know, and (d) files 
kept on the computer will only be identified with subject numbers and will not contain identifying 
information. 
 
DATA DISCLOSURE 
The proposed research will include data from up to 170 therapists and 12 administrators who work 
in community-based settings that have participated in the City of Philadelphia DBHIDS training 
initiatives. Data from approximately 405 youth receiving treatment from these therapists will also 
be included. The final dataset will include quantitative data (accuracy, cost-effectiveness, 
willingness) and qualitative data (motivation, barriers, facilitators) for each clinician and 
supervisor who participated. The final dataset will be stripped of identifiers prior to sharing. 
Release of data for data sharing will occur after publication of the main findings from the dataset. 
We will make the data and associated documentation available to users under our own auspices by 
mailing an encrypted hard-drive to users. A data-sharing agreement must be signed that accounts 
for (1) commitment to using the data only for research, (2) IRB approval at the host institution, (3) 
a plan for securing the data using appropriate technology, and (4) an agreed upon plan to destroy 
or return the data upon completion. 
 
5. Numbers of participants 
 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 12 of 13 
 
We will recruit up to 170 therapists who provide mental health treatment services in community 
mental health agencies in the City of Philadelphia who (1) have been trained in CBT through a 
city-sponsored initiative (2) respond to a brief screening survey that they use CBT; and/or (3) are 
nominated by a supervisor as a therapist who uses CBT. We will recruit approximately 3 youth 
clients receiving CBT in their session per therapist for a total of 405 youth clients (if younger than 
18 years old, we will also obtain consent from the client’s legal guardian).  
 
*Note, our intention is to enroll 3 youth clients per therapist. However, we will ask therapists 
immediately following the session if this was a “crisis” session or therapy as usual. We are looking 
for at least three samples of behavior that are not crisis sessions so we can code their behavior; 
therefore, we may need to enroll more than three clients per therapist if there are sessions that are 
deemed to not meet criteria for inclusion. There is often no way for a therapist to know when a 
crisis session will occur until the session has already started, by which point the family will have 
already consented and assented to being in the study. If it is deemed that a session does not meet 
criteria for inclusion, we will still pay the family for participation.  
 
6. Contact information for Principal Investigator 
 
Name:     Rinad S. Beidas      Degree(s): PhD 
Title:       Assistant Professor of Psychology in Psychiatry  
Agency:  University of Pennsylvania Perelman School of Medicine 
Address: 3535 Market Street, Room 3015 
               Philadelphia, PA 19104 
Phone:    215-746-1759 
E-mail:   rbeidas@upenn.edu 
 
7. Notation if requesting exemption from or alteration of written consent documentation 
and/or waiver of HIPAA authorization   
 
We will obtain consent from therapists and supervisors to participate in the study, and research staff will 
obtain consent from young adults aged 18-24, as well as assent from youth ages 7-17 and consent from 
their parents/legal guardians.  
 
We have requested waiver of documentation of parental consent when a youth client 7-17 
years presents to their treatment session without a parent/legal guardian. While piloting our 
procedures at our first agency, we have identified a major barrier to enrollment of these clients. 
Many youth clients present to the community mental health agencies for their treatment session without 
a parent/legal guardian, particularly adolescents who can come to session on their own using public 
transportation. We have discussed potential solutions to this barrier with therapist stakeholders and have 
learned that a parent/legal guardian can often be reached by phone during the client’s session even if they 
do not accompany youth to session. As such, we have requested waiver of documentation of parental 
consent when a youth client 7-17 years presents to their treatment session without a parent/legal guardian. 
Our justification is as follows: 
- 
The research involves no more than minimal risk to the subjects, as we are asking youth clients to 
agree to the audio recording of their session. The primary risk related to study participation is the 
breach of confidentiality of the audio recording. We minimize this risk by immediately transferring 

2016-24 FACTS protocol (PI: Beidas)  
Version date 05/11/2020 
 
Page 13 of 13 
 
the audio recording to an encrypted hard drive.  
- 
The waiver or alteration will not adversely affect the rights and welfare of the subjects, since we will 
still obtain assent in person from the youth client and consent from the parent/guardian over the 
phone in the presence of the youth client. The only difference from our currently approved 
consenting process is that the parent/legal guardian will not be physically present in the room during 
parental consent and a signed consent form will not be obtained from the parent/legal guardian.  
- 
The research could not practicably be carried out without the waiver or alteration, because many 
youth clients 7-17 years present to their treatment session without a parent/legal guardian. If we do 
not obtain this waiver of documentation of consent, then we would have to significantly increase the 
amount of time needed to enroll three clients per therapist, which could be perceived as too 
burdensome on an agency and could influence agencies and therapists not to participate in this study.  
Additionally, this research could not practicably be carried out without waiver of documentation of 
parental consent for youth who want to participate but whose parent/legal guardian cannot feasibly 
attend the session. We want to be able to provide these youth with a fair opportunity to participate if 
they express interest in participating in this study.     
 
Our process to obtain verbal consent from parents/legal guardians is as follows: when a youth client 
7-17 years present to their treatment session without a parent/legal guardian, then the therapist will 
call the parent/legal guardian and if s/he agrees to hear more about the study, the research team will 
speak to the parent/legal guardian on the phone in the presence of the therapist and client. If the 
parent/legal guardian verbally consents to allow their child to participate in the study and the client 
assents to participate in the study, the research team will obtained signed assent from the youth client 
and write a note documenting the consenting process of the parent/legal guardian. This note will be 
maintained with the client’s assent form in a locked filing cabinet in a locked office room.  
 
We have requested alteration of documentation of youth, caregiver, and young adult 
assent/consent when participants cannot sign consent/assent forms in-person (as in the 
case of the COVID-19 outbreak). Specifically, when a client and/or caregiver is 
interested in participating in the study, we will obtain verbal assent and/or consent 
from the client and/or caregiver. Research staff typically meet with clients and caregivers 
around their scheduled therapy sessions at community mental health agencies to minimize 
burden to families. However, when it is not possible to meet with clients and caregivers in-
person, we will enroll them through a virtual platform. Not allowing eligible clients and 
caregivers to participate in the study virtually during times when in-person enrollments are 
not possible presents a recruitment barrier for the study. As such, we are requesting alteration 
of documentation of caregiver consent, youth (7-17) assent, and young adult (18-24) consent 
when a virtual enrollment is to be conducted. When this occurs, the therapist will contact the 
client and/or caregiver ahead of time to see if they are interested in participating in the study. 
If the client and/or caregiver is interested, research staff will obtain verbal assent/consent via 
a virtual platform. As with in-person consents, assent will be obtained from youth ages 7-17, 
consent will be obtained from young adults age 18-24, and consent will be obtained from 
caregivers of clients age 7-17.  If verbal assent/consent is not obtained from all necessary 
parties, study participation will not take place.