App for Strengthening Services In Specialized Therapeutic Support
Phase: NA
Conditions: Autism Spectrum Disorder
Sponsor: University of Pennsylvania

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 1 of 11 
 
 
Protocol ID: 84995 
ClinicalTrials.gov ID: NCT05207956 
Title: App for Strengthening Services In Specialized Therapeutic Support (ASSISTS) 
849995 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 2 of 11 
 
 
 
1. Description of theory, methodology, intent of study 
Philadelphia’s Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) is 
about to make substantial investments in establishing and enforcing standards for autism 
intervention. Data collection will be an important part of these new standards. The objective of the 
proposed study is to learn about the barriers and facilitators of support staff’s data collection in 
their work with children with autism. The main objective of the clinical trial is to test the refined 
app with 30 support staff from three agencies. Successful completion of the proposed study will 
lay the foundation to develop an intervention to increase data collection using behavioral economic 
principles. 
 
BACKGROUND 
Elementary school children with autism often need intensive support throughout the day. Concerns 
about safety, behavioral challenges, and need for a highly structured environment have resulted in 
an increased use of one-to-one aides at home, school and in the community. These aides, referred 
to as ‘behavioral health technicians (BHTs)’ in Philadelphia, usually have a bachelor’s or 
associate’s degree and receive limited training and supervision. (Please note that the terminology 
used to refer to these aides has changed from ‘therapeutic support staff” [TSS]; all subsequent 
references to these aides in the document will be as BHTs.) Some BHTs have received additional 
training and are referred to as ‘registered behavior technicians (RBT)’. Ideally, aides would use 
evidence-based interventions in the family of applied behavior analysis (ABA) to help children 
reduce problem behaviors and increase adaptive behaviors. 
 
Philadelphia’s Medicaid system spends ~$80 million a year on BHTs, about a third of the 
children’s mental health services budget. While children with autism comprise 7% of all children 
served through this system, they account for 40% of BHT services. Administrators, advocates and 
parents have decried the poor or unknown quality of care and outcomes associated with it; yet the 
very nature of the work they do makes it difficult to monitor. Our structured observations of BHTs 
in the community, combined with our interviews with administrators and clinicians suggest that 
BHTs are not supported in providing high-quality, evidence-based care, in large part because of 
the isolating nature of their work and limited opportunity for feedback and acknowledgement. 
 
This project builds on our preliminary work to develop and test strategies to increase support staff’s 
self-efficacy, supervision, and sense of belonging to a professional community, with opportunities 
for peer comparison and supervisor recognition as mechanisms to increase the use of evidence- 
based practice. The target practice for this study is data collection. We chose data collection 
because: 1) it is a component of every evidence-based intervention for children with autism and is 
common to many mental health interventions for other children as well; 2) this foundational 
practice is essential to measuring and monitoring outcomes and has been associated with more 
positive outcomes in and of itself; and 3) it makes possible objective assessment of the 
effectiveness of implementation strategies and supports iteration and improvement. 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 3 of 11 
 
Our clinician-focused implementation strategy is based on three psychological principles that 
inform behavioral economics. The first is bounded rationality, defined as limited information, 
cognitive capacity and willpower. Support staff may find data collection difficult because they are 
not sure what information to collect or how to collect it easily. The second is perceptions of social 
norms. Based on typical practice, support staff may believe that their supervisors do not expect 
them to collect data and that none of their peers collect data. The third is hyperbolic discounting, 
in which people prefer more immediate gratification (not expending effort to collect data) at the 
expense of long-term outcomes (data ultimately used to show child progress and inform future 
interventions). 
 
STUDY SAMPLE AND DATA SOURCE 
The study will begin human subject activities in summer 2019 (pending full approval from the 
City of Philadelphia IRB as the IRB of record, and signature of a reliance agreement from the 
School District of Philadelphia Research Review Committee and the Penn IRB).) We will continue 
enrolling human subjects until 2021. 
 
Sample. Participants will include 30 BHTs who provide individual support to children with autism 
in Philadelphia preschools, schools, daycares, or homes. 
 
Data Source. 
 
Data in this objective will come from: 
 
1. In-person observations (up to two hours each), or if infeasible due to COVID-19 
restrictions, one-hour interviews by phone or video conference to compare BHTs’ data 
collection with data collected by an observer from the research team. Observations that take 
place via videoconference or phone will not be recorded in any form. 
2. Two 10-minute follow-up questionnaires about demographic information, perceptions of app 
usability and information regarding their intentions to collect data on their students. 
3. Qualitative interviews (in person, by video conference, or by phone; 30-60 minutes) to 
elicit contextual information about BHTs’ experiences using the app and collecting data. The 
researcher will take notes during these interviews and they will be audio-recorded. 
4. Data collection consistency, completeness and timeliness of BHT’s to understand the 
acceptability and utility of a mobile app developed in a partnership between study leads and 
the Mozzaz Corporation. 
The study team will not collect any identifiable student data. The study team will not share data 
collected through this study with BHTs’ supervisors. 
 
PROCEDURES 
We continue to partner with the Mozzaz Corporation, a digital health software company, to create 
two versions of a mobile application to help BHTs who work with students with autism collect 
data. The first version comprises a basic electronic platform for data collection. The second version 
has the same basic electronic platform for data collection, plus additional features designed to 
increase motivation to collect data and ease the burden of data collection. We will randomize 
BHTs (n = 30) to use one of the two app versions to evaluate the effect of including these additional 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 4 of 11 
 
features on BHTs’ rates of data collection. In the event that participants are recruited from different 
types of settings, we will work with the study statistician to implement appropriate stratified 
sampling strategies. 
 
We will recruit BHTs from agencies whose leadership have agreed to allow BHTs to participate 
in this project. Research staff will contact leadership at community agencies with an overview of 
the study and confirm a plan to distribute information about the study in the following ways 1) 
leaders will email their team describing the study and inviting them to participate and/or, 2) agency 
leaders and participating BHT’s will share BHT emails with research staff who will email BHT’s 
directly to invite them to participate. The email will contain an electronic link to an electronic 
consent to be contacted form in REDCap. 
 
After BHTs consent to participate in the trial, we will randomize them in pairs to one of the two 
study arms. Members of the research team will teach BHTs how to use the app during a brief 
individual meeting that will occur at the school where the BHT works, at their agency, or remotely, 
depending on BHTs’ preference and possible COVID-19 restrictions. BHTs also will complete 
baseline questionnaires on REDCap on their demographic information, intentions to collect data, 
and the determinants of intention: attitudes, norms, and self-efficacy. BHTs will be compensated 
$25 for these baseline activities, which we anticipate will take approximately 30-60 minutes. 
 
We then will ask BHTs to use their assigned app to collect data daily for 3 weeks, or up to 15 
sessions. All data collection will occur via the BHT’s smartphone, or if preferred, through the 
research team’s iPad. The Mozzaz app and associated server is HIPAA compliant. BHTs will only 
record de-identified data in the mobile app; neither the app nor study staff will collect identifiable 
information about the students with whom the BHTs work. As originally written, this project 
included research staff conducting in-person observations of BHTs’ use of the app to verify BHTs’ 
self-report data. Given COVID-19 restrictions, we propose multiple options to replace in-person 
observations: 
 
1. In the event that the BHT is working in person and in-person observation is possible: 
Research staff will coordinate with the BHT to facilitate in-person observations over the 
course of approximately three weeks. Research staff would complete up to observations 
over the course of approximately three weeks (each up to two hours long.) 
2. In the event that the BHT is working in person but in-person observation is not possible: 
In lieu of in-person observations, research staff will administer weekly one-hour interviews 
with the BHT about their experience using the app. Research staff would complete up to 
three one-hour interviews over the course of approximately three weeks. 
3. In the event that the BHT is working remotely: Research staff will coordinate with the BHT 
to facilitate remote observations through their video conferencing platform. Research staff 
would complete up to three one-hour observations over the course of approximately three 
weeks. 
 
In each scenario, BHTs will be compensated $25/hour per observation or interview (up to six 
hours, up to $150) and $10 per ten-minute follow-up questionnaire ($20). 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 5 of 11 
 
After the three-week trial period concludes, BHTs will return their iPad to the study team if they 
were loaned one. Participants will complete ten-minute questionnaires about perceptions of app 
usability, their intentions to collect data, and determinants of their intentions to collect data 
(attitudes, norms, and self-efficacy) via REDCap. We also invite all participants to participate in 
an additional qualitative phone interview to probe deeper into the user experience and intentions 
to collect data. This optional interview will take about 30 to 60 minutes and BHTs will be 
compensated $25 for completion. 
 
In total, participants will be compensated up to $220 for their participation in the study. 
Specifically, they will be paid: 
• $25 for the initial training and baseline questionnaires (about 30-60 minutes) 
• $25 for each set of observations or interviews (up to six hours, up to $150) 
• $10 for two ~10 minute follow-up questionnaires ($20) 
• $25 for qualitative interview about experience using the app (30-60 minutes) 
 
Timeline 
Measures 
Baseline 
• Demographic questionnaire 
• Intentions and determinants questionnaire 
Trial Period 
Mobile application analytics 
Post-Trial Period 
• Intentions and determinants questionnaire 
• System Usability Scale (SUS) 
• ASSISTS app feedback form 
o Basic app arm 
o Behavioral economics-informed app arm 
• Post-trial qualitative interview guide 
 
ANALYSIS PLAN 
There are three data collection quality metrics of interest: data collection consistency, timeliness, 
and completeness. Consistency was the percentage of intervals in which aides entered data per 
session. Timeliness was the delta (time difference in minutes) between behavior occurrences and 
the aides’ data entry related to those behavior occurrences. Completeness was the percentage of 
metrics such as frequency, duration, severity, or accuracy that aides logged on behavior form 
entries out of the total number of metrics for that behavior. These metrics were captured via web 
analytics in partnership with our digital health company.  The research team will also conduct 
two ten-minute follow-up questionnaires with BHTs. 
 
The secondary outcomes of interest are: 1) the change in intentions to collect data and the 
determinants of intention (attitudes, norms, and self-efficacy), and 2) the usability of the app. We 
will calculate the change in intentions and determinants of intention by comparing the 
questionnaires distributed at baseline and after the trial. We will assess usability using the 
System Usability Scale (SUS) post-observation and by conducting qualitative interviews. 
We will analyze outcomes as continuous variables at the BHT level across time points. Initially, 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 6 of 11 
 
all outcomes will be described and compared between groups using summary statistics (means, 
medians, and estimates of variance). For each measure, the analysis data set will contain one 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 7 of 11 
 
observation with each participant’s outcome measure at baseline and post-trial. To determine the 
preliminary efficacy of our implementation strategy, the behavioral economics enhanced app, we 
will compare the two study arms (basic app vs. enhanced app) on the primary outcome 
(proportion of time intervals BHTs take data) and the secondary measures (change in intentions 
to collect data and the determinants of intention [attitudes, norms, and self-efficacy], and app 
usability), using linear regression with random effects for BHT and time (baseline, post-trial 
period). The regression will include treatment arm, time (baseline and endpoint), and the 
interaction between group and time. The coefficient associated with the interaction will indicate 
the effect of one treatment arm versus the other on data collection. 
 
We also will examine appropriateness, acceptability and feasibility through qualitative analysis 
of the interviews conducted with all participating BHTs. Audio recordings of interviews will be 
transcribed and all transcripts will be loaded into qualitative analysis software for data 
management and analysis. Thematic analysis will be guided by an integrated approach that 
includes identification of the three priori attributes of interest (i.e., appropriateness, acceptability 
and feasibility), and grounded theory, which provides a systematic approach to identifying 
emergent themes and is used to generate robust theoretical models of social behavior in 
healthcare settings. This integrated approach uses an inductive process of iterative coding to 
identify recurrent themes, categories, and relationships in qualitative data. After exploring the 
data, a comprehensive coding scheme will be developed and applied to produce a fine-grained 
descriptive analysis. Study team members will separately code the transcripts and compare the 
results to assess the reliability of the coding scheme. Any disagreements in coding will be 
resolved through team discussion. After the coding scheme is finalized, coders will be required 
to code two transcripts in a row at high level of agreement (κ >.8) with the lead coder before 
independent coding of the rest of the transcripts. A subset of transcripts will be double-coded for 
the reporting of inter-rater reliability. 
 
Data analysis will inform future refinements needed to optimize the development of the app and 
provide the basis for a peer-reviewed publication on the outcomes of the randomized controlled 
trial, as well as a subsequent grant application to test the effects of our implementation strategy 
in a fully-powered trial. 
 
2. Description of Philadelphia Department of Public Health, Department of Behavioral 
Health or Risk Management involvement 
This project is investigator-initiated. However, it is a result of many conversations with community 
stakeholders, including DBHIDS leadership. DBHIDS is not formally involved with this project 
and will not receive any identified data; however, we will share study findings to ensure that policy 
is shaped by the research, as consistent with best practices (e.g., present at Research Grand 
Rounds). 
 
3. Duration of Study 
 
The study plans to begin human subjects activities in Summer 2019 and run until May 2022. The 
end date has been amended in response to a pause in recruitment in schools caused by COVID- 19, . 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 8 of 11 
 
4. Aspects of research 
 
Potential Benefits. There are no direct benefits to the participant; however, participants may find 
satisfaction in sharing their ideas and experiences with research staff as their feedback may be 
utilized to inform future funding. The benefit to the community is that the results may ultimately 
help enhance the implementation of EBP. The ratio of risks to benefit is positive, considering that 
we do not think that the study poses considerable risks and that the scientific yield from the study 
could be greatly beneficial. 
 
Dissemination. The findings from the research will be reported to the City of Philadelphia 
DBHIDS through Research Grand Rounds, agencies that provide child mental health services, and 
to the BHRS system that provides support to children with autism in schools so that the 
stakeholders will have the opportunity to review and discuss this information. Further, we will 
prepare findings for publication. 
 
5. Risks to subjects 
 
SUBJECT RISKS 
There are no known physical or legal risks to participating in the study. Participants will be asked 
to answer questions about their practice and will also be observed. Self-report, brief observation, 
and semi-structured interviews could lead participants to feel some temporary emotional 
discomfort. The investigator or a member of our clinically trained team will be available to speak 
with any participants who feel unduly distressed, and appropriate referrals will be made. 
Participants will not be required to participate and lack of participation will not impact their 
employment. Our experience in similar previous research is that support staff who are not 
interested in participating in the research will leave after hearing about the study. This does not 
impact their employment because we ask managers/leadership to leave the room after introducing 
us during the consenting process, so leadership is not aware if any staff declined participation. 
 
In the event that child abuse/neglect or suicidality/homicidality is identified upon observation, a 
member of our clinically trained team at the Penn Center for Mental Health (CMH) will follow- 
up with the support staff to ensure that a safety plan is in place to ensure that any information about 
child abuse/neglect or intent to harm self or others has been reported to the authorities, as required 
by law. In the event that staff misconduct is identified during observation, a member of our 
clinically trained team will follow-up with the support staff and/or his/her supervisor to determine 
whether the event is reportable and to ensure that the report is filed if deemed appropriate. 
 
All research staff at CMH will be trained by a member of our clinically trained team on mandated 
reporting requirements. In the event of any of the above circumstances, research staff will report 
events to a member of our clinically trained team and they will follow-up with the support staff 
within 2 business days of becoming aware of any event to ensure that any necessary safety and/or 
reporting measures have been taken and that the child’s support staff is knowledgeable about the 
occurrence and has taken action to address it. 
 
SUBJECT CONFIDENTIALITY 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 9 of 11 
 
As part of informed consent and assent, participants will be informed that all information they 
provide will be kept confidential (unless it is determined that it must be reported as required by 
law), within the personnel of the CMH. No information gathered as part of this research will be 
shared with agency executive directors, supervisors, or DBHIDS (unless it is determined that 
certain information must be reported as required by law). Participants will be asked to sign a 
consent form that thoroughly describes the procedures to be followed in the study and the type of 
assessments involved. The Philadelphia Department of Public Health Institutional Review Board 
will have approved the protocol and consent prior to the initiation of the study. The University of 
Pennsylvania and the School District of Philadelphia Research Review Committee will sign 
authorization agreements so that the Philadelphia Department of Public Health Institutional 
Review Board can serve as the IRB of record. We will submit these signed authorization 
agreements as a modification. Participants will be provided with copies of the signed consent forms 
while original copies of the signed consent forms will be kept in locked files at the CMH which 
no one will have access to other than Dr. Mandell and the research team. All data will be coded 
with a subject identification number. The names that correlate to those numbers will be kept 
separate, such that any identifying information will be stored in one file, while experimental data 
will be kept in a separate file. Only Dr. Mandell and the research team will have access to these 
files. All project staff will complete training on confidentiality through the Collaborative IRB 
Training Initiative (CITI) course. 
 
All data, including the master list linking identifiers to the ID number and recordings will be 
destroyed in 2027. NIH policy requires that data be retained for a period of 3 years from the date 
of the final Federal Financial Report. The award period is from 2019-2021. That means that we 
must retain all study data until 2027, thus 8 years following study start. Assuming that data 
collection begins as planned in 2019, that means that the data of a participant who consented in 
2019 will be kept until 2027 (8 years later). 
 
Privacy and confidentiality is of the utmost importance given that a breach in privacy and/or 
confidentiality could result in serious consequences for participating clinicians (e.g., employment 
implications). Thus, breach of privacy and/or confidentiality are the biggest risk of participating 
in this study. 
 
To ensure privacy, self-report measures will be completed either by hand in the classroom or using 
a HIPAA compliant web-based survey platform, e.g., REDCap or Qualtrics, hosted by the 
University of Pennsylvania. Participants will have the option of completing measures using paper- 
and-pencil or via web given our previous experiences that support staff prefer different options. 
Paper-and-pencil measures, only identified with ID numbers, will be sealed in a manila envelope 
and transported to Penn in a locked briefcase. Semi-structured interviews will be conducted in a 
private location of the participant’s preference. In our previous studies, this has typically been over 
the phone or in-person in an open office in the school or agency where the support staff works. 
The participant interviews with the research staff and the content of the interview is recorded and 
then later transcribed and de-identified by individuals who are trained in human subjects 
protection. 
 
To ensure confidentiality, we will use the following methods: (a) all self-report data, rating scales, 
observations, and interviews will be kept in a locked filing cabinet and/or on password-protected 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 10 of 11 
 
servers, that can only be accessed by study personnel; (b) subject identity will be masked using 
numeric codes and password-protected master list which only Dr. Mandell and the research team 
will have access to; (c) data will be entered directly into password-protected files which only Dr. 
Mandell and the research team will know, and (d) files kept on the computer will only be identified 
with subject numbers and will not contain identifying information. 
 
DATA DISCLOSURE 
The proposed research will include data from support staff and supervisors who work in school 
settings in the City of Philadelphia. The final dataset will include quantitative and qualitative data 
for each support staff and supervisor who participated. The final dataset will be stripped of 
identifiers prior to sharing. Release of data for data sharing will occur after publication of the main 
findings from the dataset. We will make the data and associated documentation available to users 
under our own auspices by mailing an encrypted hard-drive to users. A data-sharing agreement 
must be signed that accounts for (1) commitment to using the data only for research; (2) IRB 
approval at the host institution, (3) a plan for securing the data using appropriate technology, and 
(4) an agreed upon plan to destroy or return the data upon completion. 
 
6. Numbers of participants 
30 BHTs will be enrolled in the RCT 
 
7. Contact information for Principal Investigator 
 
Name: 
David Mandell 
Degree(s): 
ScD 
Title: 
Professor and Director, Penn Center for Mental Health 
Agency: 
University of Pennsylvania Perelman School of Medicine 
Address: 
3535 Market Street, Room 
Philadelphia, PA 19104 
Phone: 
215- 573-7494 
E-mail: 
David.Mandell@pennmedicine.upenn.edu 
8. Notation if requesting exemption from or alteration of written consent documentation 
and/or waiver of HIPAA authorization 
We are requesting a waiver of written documentation of consent from the PDPH IRB for supervisors and 
BHTs who participate in the qualitative interview over the phone part of this research project. Because 
these interviews will be done over the phone, we will not have an opportunity to collect a signed copy of 
the consent form. Before the scheduled interview takes place, research staff will send a copy of the consent 
form to the participant via email. During the call but before the interview begins, the research staff will go 
through the required elements of informed consent and answer any questions the supervisor or BHT might 
have. If they agree to participate, then the member of the research team conducting the call will note the 
participant’s name and the date the individual consented to participate on a consent form. 
 
This research component presents no more than minimal risk of harm to subjects and involves no 
procedures for which written consent is normally required outside of the research context. The 
primary risk related to study participation is the breach of confidentiality of the audio recording. We 

ASSISTS 2019-32 
IRB Protocol, version 4/11/2024 
Page 11 of 11 
 
minimize this risk by immediately transferring all data from the interview to a HIPAA-compliant platform 
(REDCap) and a HIPAA-compliant drive. The waiver of documentation of consent will not adversely 
affect the rights and welfare of the subjects, since we will still go through all required elements of 
informed consent over the phone and send the consent form to the participant via email prior to the phone 
call.