Supporting the Implementation of a State Policy on Screening for Adverse Childhood Experiences (ACEs) in Federally Qualified Health Centers (FQHC)
Phase: NA
Conditions: Adverse Childhood Experiences
Sponsor: University of Colorado, Denver

Supporting the implementation of a state policy on screening for Adverse Childhood 
Experiences (ACEs) in Federally Qualified Health Centers. 
Study Protocol Document 
NCT: 04916587 
Date: 07.20.2024 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
1 
 
COMIRB Protocol 
COLORADO MULTIPLE INSTITUTIONAL REVIEW BOARD 
CAMPUS BOX F-490    TELEPHONE:  303-724-1055  Fax:  303-724-0990 
Protocol #: 
22-0547 
Project Title: 
Supporting the implementation of a state policy on screening for Adverse Childhood 
Experiences (ACEs) in Federally Qualified Health Centers. 
Principal Investigator: Mónica Pèrez Jolles, PhD. 
Version Date: 05/10/2024 
 
I. 
Background/Rationale [previously iStar section 11 (exempt) or 12 (expedited/full board)] 
Background 
• 
Adverse Childhood Experiences (ACEs) refer to traumatic events occurring before age 18, such as 
maltreatment, family separation and exposure to violence. 
• 
ACEs are pervasive among children ages 0 – 17 years old in the United States with 45% experiencing 
at least one ACE and 10% experiencing three or more ACEs. These events place children at high risk 
for negative life outcomes if left undetected and untreated. 
• 
Early screening for ACEs is a national public health priority but health care providers seldom use these 
screenings during routine wellness checks. 
• 
This study leverages the Surgeon General of the state of California’s 2020 ACEs screening policy (ACEs 
Aware), which is supported by an annual $40.8 million budget. Starting January 2020, the state’s 
Medicaid health care program will reimburse primary care settings for implementing ACEs screenings 
annually during well-child visits. 
 
Justification 
• 
There is evidence that increased mental health screening campaigns alone often do not translate into 
higher access to services for children and they may in fact exacerbate disparities by increasing stigma 
and reinforcing a deficit view of marginalized groups. 
• 
One reason why these efforts do not meet expected outcomes is the fact that they lack an evidence-
based implementation plan to successfully deliver the screening programs/policies in local contexts. 
• 
This study is needed to fill that gap as, in partnership with a trauma-informed workgroup, we will 
develop and test implementation strategies suited for pediatric screening in community-based health 
centers in California. 
• 
Results of our study will inform our clinical partners and state policy leaders on their efforts to 
promote the successful adoption and implementation of the ACEs Aware state policy.  
 
II. Purpose/Objectives/Aims/Research Questions  
• 
This study will test the implementation strategy and examine its impact on the fidelity and reach of 
ACEs screenings (implementation outcomes), the resulting linkage to support services, and any 
changes in children’s clinical screening scores (i.e., Pediatric Symptoms Checklist) as a result of families 
being linked to support services. 
• 
The implementation strategy is comprised of the following activities: 
o Activity 1: Short video-trainings for clinic personnel (care team staff and providers) on the 
administration of caregiver-reported screening tools through a trauma-informed care lens and 
implementation protocols. The trainings will complement the current state-sponsored 2-hour on-
line provider training and related trainings currently implemented at the FQHC. 
o Activity 2: Technical implementation support using a combined approach comprised of external 
academic consultants and internal FQHC personnel to increase its internal capacity. Each clinic 
will receive intense implementation support from the FQHC implementation coach (about 1 hour 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
2 
per week virtual or in person in the first 10-week trial crossover period), and less intense support 
in the succeeding 10-week periods (45-60 minutes every other week visit and phone consult). In 
addition, the Trauma Informed Care (TIC) workgroup, comprised of FQHC leadership (i.e., clinical 
and quality departments), service providers, communications director, information technology 
(IT) coordinator, and academic collaborators will support clinics.  
o Activity 3: Use of a validated clinical screening tool — The Pediatric Symptoms Checklist (PSC) is 
used in pediatric primary care settings to assess behavioral and social/emotional development. 
For this study, we will use the PSC tool tailored to children ages 0 to 5 years old. The PSC screening 
tool is needed, as the ACEs screening called PEARLS only assesses ACEs exposure and not mental 
health needs, which is a common ACEs sequela. 
o Activity 4: Use of a technology-based tailored screening algorithm that will be made available to 
ACEs screeners using computer tablets. The algorithm will be hosted in a HIPAA compliant REDCap 
(Research Electronic Data Capture) secure web application, hosted at the PI’s institution, to 
facilitate the screening process.  
• 
Examine the value of pairing the delivery of a state policy with an implementation plan tailored to 
community-based clinics.  
• 
Convene a diverse workgroup comprised of researchers, health care personnel and patients to refine 
an implementation strategy designed to support the ACEs state screening policy in community-based 
clinics. 
• 
Pilot test the feasibility, acceptability, fidelity, and reach (i.e., number of ACEs screenings delivered 
divided by the total number of eligible child visits to their clinic in a 10-week period) of the developed 
implementation plan.  
• 
Compare changes in child referrals to mental health services, after caregivers’ participation in the 
screenings, between intervention and comparison clinics. 
• 
Compare changes in PSC scores before and after caregivers’ participation in the screenings:  
Ten weeks after child’s initial screening, a member of the study research team will contact a 
group of caregivers (n=100) on the phone to implement the Pediatric Symptoms checklist (post) 
and assess any changes from scores obtained during the initial screening at the clinic (pre). The 
study statistician will randomly select 35 caregivers from each study clinic who participated in 
the study (the goal is to interview 20 caregivers x 5 clinical sites), and create a separate database 
in REDCap with the list of selected caregivers and the phone number provided in the consent 
form. Research study team members will use the list to contact caregivers to schedule phone 
interviews. We will use this database to track completed interviews, declines, and not being able 
to contact. The statistician will replenish the list as needed. 
 
III. Participants (sample) (Consistent with iStar sections 10 (expedited and full board) and 22 (for all 
submission types) 
a. Inclusion and exclusion criteria for enrollment:  
Inclusion criteria. Clinic eligibility is defined by: provision of pediatric care (i.e., family medicine, pediatric, 
primary care) for child patients ages 0-5 years old. Caregiver eligibility for reporting of the PEARLS and 
Pediatric Symptoms Checklist (PSC) on their child’s behalf and interviews: Adults ages 18 years and older 
with legal custody or with the authority to arrange care for the child. Clinic staff eligibility for interviews: 
Clinic personnel involved directly or indirectly in the planning and/or implementation of the ACEs Aware 
policy. 
  
Exclusion criteria. Clinics not providing pediatric care (i.e., family medicine, pediatric, primary care) for 
child patients ages 0-5 years old. Caregivers that are not a legal guardian or someone with written 
authorization at the clinic to make decisions for the child. Clinic staff being a new hire and/or not having 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
3 
participated directly or indirectly in the implementation of the ACEs Aware screening at their clinic.  
 
b. Rationale for excluding children: We are not recruiting children for data collection during screenings 
at the clinic visit because their caregiver will report on their behalf based on two screenings (PEARLS and 
PSC). Child data on referrals to mental health services and socio-demographic characteristics will be 
obtained directly from the child’s clinical chart (EMR system) and after obtaining signed consent and 
HIPAA authorization forms from each caregiver. The ACEs screenings are happening at the clinics 
regardless of this study.  
 
c. Participants are part of two groups of special populations:  
1) Clinic Employees: For Aim 2, we will invite clinic personnel to be part of interviews to evaluate the 
success of our implementation strategies based on their experiences. That means, we will survey 
them and ask follow-up interview questions assessing their acceptability, feasibility and lessons 
learned from their experience with the screening process at the clinic.  
2) Non-English-Speaking Caregivers: For Aim 2, we will randomly select caregivers at each clinic 
whose children were screened for ACEs to participate in phone interviews about their perceptions 
of the screening process. We will also gather basic caregiver sociodemographic information (age, 
gender, education). Spanish-speaking caregivers will be interviewed by a fluent Spanish speaker 
(e.g., The PI is Latinx scholar and Spanish is her first language). For the child screenings (Aim 1), we 
will follow the clinics’ procedure. That is, when the caregiver makes an appointment for their child, 
they are asked if they would like a translator. Every clinic has someone who speaks Spanish and 
two have someone who speaks Arabic. In the event that there would not be a Spanish-speaking 
staff member available, translation services are available.  
 
IV. Recruitment/Screening Process (sampling strategy)  
a. Recruitment will take place at five Federally Qualified Health Center (FQHC) clinics located in 
under-resourced communities in California. We do not have exact locations yet because these 
five clinics will be randomly selected from a total of 28 clinics located in multiple counties (e.g., 
Riverside, San Bernardino, San Diego). 
 
b. Recruitment of caregivers for participation in the ACEs screenings and to allow access to their 
child’s EMR data: Once the caregiver checks-in at the front desk, the medical assistant (MA) 
individual will meet the family and will walk them to the examination room for their wellness visit 
(part of routine care procedure). As part of this event, the MA will introduce the study and obtain 
consent from the caregiver for participation. Here, caregivers can decline, in which case the MA 
will continue with the wellness visit procedure that includes screening for ACEs using the PEARLS 
tool (this screening is happening at the clinics and regardless of this study). The FQHC’s data 
analyst will receive the list of children who had been screened and consented to be in the study 
and will provide EMR data to the PI based on the study design schedule. Note: If the caregiver 
declines to participate in this study, the screener will not use the REDCap programmed tablets as 
we will not have that caregiver’s screening data at all.  
 
Recruitment of caregivers at each clinic for interviews (Aim 2): The caregivers would be informed 
during initial consent that someone may contact them for a follow up interviews (consent form 
includes a space to add their phone number for this follow-up contact). A member of our research 
team will contact selected caregivers to invite them to participate in the follow-up phone 
interview. Caregivers can decline this follow-up interview at any point. We know caregivers move 
often or change their phone numbers. We will ask help from the clinic manager in identifying a 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
4 
current phone number for a caregiver if the one provided in the consent is no longer working. 
Three attempts will be made to schedule the phone interview with caregivers.  
 
The caregiver consent form will also state that for a group of caregivers selected at random 
(n=100) they will be contacted 10-weeks after their participation in the study to be asked to follow 
up questions from the Pediatric Symptoms Checklist (PSC), and about their opinions and 
experiences with the study, ACEs screening process and suggestions for improvement. Each 
caregiver will receive a $50 incentive for participating in the interview. The caregiver consent form 
also includes language stating that there is a data sharing requirement using a National Data 
Archive or NDA and per requirement from the federal funding agency.  
 
We will offer the consent form electronically as an option using REDCap to reduce the clinical 
burden.  
 
From the group of caregivers who are consented at the clinic and who agree to participate in the 
follow-up interview, the additional recruitment strategies, informed by our clinical partners, are 
as follows: 
• 
Call caretakers from a local number by obtaining a Google number with each of the 
clinic’s location area code 
• 
The study team members will text caregivers in advance of our interview call so they 
expect the call. They can reply at that time with “yes” or “no” to confirm. We will add a 
picture of the flyer. 
• 
Script for the text 
o “Phone call for ACEs interview soon from this number. Text back ‘Yes’, ‘No’, or a 
date to schedule. Flyer attached”. 
 
Recruitment of clinic staff for surveys/interviews (Aim 2): The Data manager at Borrego Health 
FQHCs will send an internal email asking clinic personnel who directly (e.g., performs screenings) 
or indirectly (e.g., management and front desk person who is aware of this initiative at their clinic) 
participate in the ACEs screenings to complete an on-line (10-minute) survey to assess their 
opinions and experiences with the implementation strategies supporting the ACEs screening 
program at their clinic. The survey will have at the end a question asking if they can be contacted 
for a follow up 25-minute phone call interview. If they agree, the PI will contact them. The clinic 
manager will not know exactly who accepted the invitation and who did not to preserve 
confidentiality and undue influence to participate (this will be listed in the online survey). The PI 
will ask each clinic to provide a higher number of staff names than planned to allow for declines.  
 
c. Materials that will be used for recruitment of caregivers will be a flyer about the study that will 
be given to them by the front desk as soon as they check in at the clinic so they can read in advance 
while they wait to be seen by the provider. In addition, we will use the consent form and verbal 
explanation from the clinic ACEs screener.  Materials used for staff recruitment is an email script 
(uploaded in the application) sent internally.  
 
d. Recruitment of caregivers for participation in the REDCap screenings and to allow access to their 
child’s EMR data: Children who are scheduled for a wellness visit during the trial period will be 
identified using the clinic’s electronic medical record system. Each eligible child will be given a 
unique ID for the study, which is linked to the child’s medical chart record ID. The FQHC’s data 
analyst will provide to the PI this list and de-identified by matching patient chart ID with our study 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
5 
fake ID and providing only the latter.  
 
e. Recruitment of clinic staff for surveys/interviews (Aim 2): The PI will ask the study contact person 
(program co-lead for the study) at Borrego Health FQHCs for a suggested list of staff members at 
the clinic who have been directly or indirectly involved in the ACEs screening program. In the past, 
this individual has worked with each clinic’s manager to obtain suggested names. From that list, 
the internal data manager will send an internal email asking them to complete an on-line (10-
minute) survey. 
 
V. Methods  
1. Study steps 
1. Hiring of personnel (Months 1 – 2) 
2. Kick-off meeting with clinical partners (Month 2): The PI will update a trauma-informed care 
(TIC) workgroup, already convened in 2019 and meeting bi-monthly, about funding of this 
project. The group will plan for the implementation mapping sessions that will be scheduled 
(Months 2-5) of the study.  
3. Kick-off meeting with research team (Month 2): The PI will hold a kick-off meeting with the 
research team (including consultants)  
4. Preparation for Step 1 of trial period (Months 4-6):  
4.1 
Month 1 or 2: The Research Director at the FQHC will receive the implementation 
coaching training online (scheduled for May of 2021 by the organization providing the 
national certification training). Due to very high clinic staff turnover, we are broadening 
the potential training options for implementation coaching and by including more 
individuals (e.g., clinic screeners, managers, director of Healthy Steps program). 
4.2 
Months 3-5: Dr Mack will randomly select five clinics (from a pool of 27 clinics) as 
research sites for the study.  
4.3 
Months 2-4: Dr. Mack will create a secure Research Electronic Data Capture (REDCap) 
web application database, which is HIPAA-compliant, and develop an algorithm that will 
include the questions for each ACEs screening tools (PEARLS and PSC) and an ACEs-
related health conditions checklist provided by the state of California as a checklist. The 
algorithm will automatically provide an assessment of the child patient’s risk level for 
ACEs-related health issues, as well as indicate whether a referral to mental-health 
services is indicated. 
4.4 
Months 5 - 7: Pilot testing of the REDCap algorithm using computer tablets by the PI, 
Dr. Mack and the FQHC’s data analyst and with the use of dummy data. We will also use 
simulations/demos with the Community Advisory Board to reduce clinical burden. Test 
data will be entered into the REDCap system to identify glitches and refine the system 
as necessary.  
4.5 
Months 5-7: ACEs computer tablets delivered to the Research Director (Ms. Reaves) via 
FedEx and for distribution to each of the selected clinics. Each clinic’s manager will be 
responsible for delivering the tablets to the assigned ACEs screener for the study. Zoom 
conferences will be held with the principal investigator (PI), FQHC implementation 
coach and screeners to familiarize them with the tablets and how to operate them, and 
troubleshoot any potential problems. 
4.6 
Month 4: The PI and consultant (Stadnick) will pre-record 3 video-training sessions on 
study procedures and protocols, caregiver consent process and administration of the 
PSC survey (in 2020, all clinic personnel at the FQHC partner have already completed a 
2-hour online training on how to administer the PEARLS screener)  

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
6 
4.7 
Month 5: Screeners at each selected clinic will go over the videos for training. The PI 
and Dr. Stadnick will schedule a follow up 45-min call with each screener to discuss 
videos and learning/questions.  
4.8 
Months 6 – 7.5: Baseline starts in all five selected clinics. Pre-intervention data will be 
collected from the clinics over a 10-week period. Clinic staff will be told to follow the 
ACEs state requirements, which include the PEARLS screenings, ACEs-related health 
conditions checklist, wellness exam, follow-up education, and mental-health referrals if 
deemed appropriate. Mediators data will be collected at the end of this period.  
4.9 
Months 7.5 to 30 / Trial starts:  
4.9.1 
During a 10-week trial period for each clinic (data collection will continue until 
the end of the trial), each clinic switching from control to intervention status 
will start using the implementation strategies (i.e., support from the trained 
implementation coach, TIC workgroup guidance/advise, use of electronic 
tablets for screenings and service referrals).  
4.9.2 
Due to the human and financial impact of COVID-19 epidemic, the randomized 
control clinics had not implemented ACEs screening at the time of the pilot 
starting. We have identified two clinics within the healthcare system, not part 
of the trial, that have started implementing ACEs screening. These two clinics 
will serve as non-randomized comparison groups for this study. We will 
compare our study clinics and these comparison clinics on ACEs screenings and 
on referrals to mental health services.  
4.9.3 
Steps for ACEs screenings with caregivers: 
4.9.3.a. When a child and their caregiver come to the clinic for the wellness 
visit, screening staff will conduct the study screenings using a tablet 
device that has the REDCap database on it. This should take 10 
minutes. 
4.9.3.b. As part of the screenings for the study, screening staff will use either 
the Baby Pediatric Symptoms Checklist (BPSC) or the Preschool PSC 
based on the child’s age to help caregivers more accurately identify 
perceived behavioral and/or emotional concerns they have about 
their child. Due to low reading levels among some of our participants, 
the screener will ask caregivers if they prefer the questions to be read 
to them by the screener.  
4.9.3.c. Once the staff has finished conducting the screenings (i.e., PEARLS 
and the BPSC or PPSC), the physician will come in to conduct the 
wellness visit as part of routine care. The physician will use the tablet 
provided by the screening staff with the REDCap database to check 
ACES-related health conditions from an automated checklist. ‘ 
4.9.3.d. Based on our REDCap preprogrammed algorithm, which takes the 
information from the PEARLS, PSC, and the ACES-related health 
condition checklist, a pop-up box will appear indicating the child’s risk 
level (low/medium/high). If the child’s risk level is determined to be 
high, the physician will be prompted on the tablet to begin a mental-
health referral for the child and the caregiver.  
4.9.3.e. Screening staff will document when any caregivers decline 
screenings, or when there is an incomplete screening for any reason.  
4.9.3.e.1 Mediators and primary and secondary outcomes will be 
collected by the co-leads at the FQHC, the PI and the study 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
7 
coordinator.  
4.9.3.e.2 During this time period, EMR data will be pulled by the 
FQHC’s data manager and provided to the PI as excel tables 
and using a password protected and encrypted email. The 
following data will be collected from REDCap and from the 
data manager (EHR records): number of eligible child visits, 
number of PEARLS and BPSC/PPSC screenings and scores, 
eligible children who are referred to services, child 
characteristics, and number of mental health referrals.  
4.9.3e.3 At the end of each 10-week trial period, the principal 
investigator and study statistician will meet to review the 
REDCap system and will follow up with clinical partners to 
discuss any questions and unexpected data patterns. 
4.10 
Months 12-30: An online survey with demographic and survey questions on 
feasibility, acceptability, and implementation leadership and climate measures 
will be sent out to clinic staff, managers, and primary-care providers using work 
emails. Follow up interviews will be conducted on the phone.  
4.11 
Months 8 – 36: The study’s data analyst will lead data preparation and analysis 
and will meet weekly with the PI, research team and clinical partners to go over 
progress and troubleshoot.  
4.12 
Months 22-42: Preparation of the R01 proposal and sharing of results with clinical 
partners. Dissemination of results.  
 
See a timeline summary table of the trial activities below: 
 
2. The intervention will be implemented at five partner FQHC clinics. Handheld computer tablets will 
be provided to clinic screening staff with the REDCap web application database (implementation 
strategy). The tablets will enable clinic screening staff to administer the PEARLS and BPSC/PPSC 
assessments by the medical assistant (screener) to the caregiver of the child patient prior to the 
wellness visit. After these assessments have been conducted using the tablet, the screening staff 
will hand the tablet to the primary-care provider. The tablet will then guide the primary-care 
provider to complete the ACEs-related conditions checklist provided by the state. The REDCap 
web application algorithm will use the information from the PEARLS, PSC and ACEs-related health 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
8 
conditions checklist to determine whether the child is deemed to be at low, medium, or high risk 
for toxic stress and/or caregiver concerns about the child’s emotional/behavioral development. If 
the child is determined to be at high risk, the web application will prompt the primary-care 
provider to make a referral to mental-health and support services.  
 
3. Steps that will be taken to protect participant privacy and confidentiality:   
 
For child data - Anonymous: A unique study identifier will be linked to child patient’s electronic medical 
record chart ID. The REDCap secure web application will be used to implement the screening process and 
will store data associated with screening tools, using study identifiers. In addition, in order to provide 
patient information to NIMH for the data sharing requirement, we will collect the following information 
from each child patient: child's date of birth (we will obtain this information from the EMR records and 
per caregiver signed consent and HIPAA released forms), and the child's city of birth and name (asked of 
caregiver during the screening and using the REDCap system). This information will only be used to meet 
NIMH data sharing requirement and destroyed after we meet this requirement. This information will not 
be retained by NIMH because their system will use the identifiers to create a unique participant number. 
We will use study ID for each participant for most of the work. We will keep identifiers in a separate 
password-protected folder in the study OneDrive. This information will not be kept in the REDCap forms. 
We will retrieve additional information from children screened in REDCap by producing a report in REDCap 
with a list of children screened by record number and by medical chart number. Then, the Borrego Health 
co-lead persons will access that list of children directly from REDCap using a password-protected user 
account, and will share the list internally with the data manager co-lead for linkage to additional data. 
Then, the data manager will share with the study PI the data for each child on that list using a password-
protected excel file, and send using a password-protected and encrypted work email. Data on the 
control/comparison clinics will be reported using aggregated numbers and using the same data retrieval 
procedure. Recordings of interviews will be kept in a password-protected computer drive and they will be 
destroyed after the transcription has been finalized. All transcriptions will not include individual names or 
identifiable information. All data, including transcriptions, will be kept for 3 years, and then erased per 
NIMH policy.  
 
Reporting on adverse events: This study does not involve physical/psychological harm. Screeners at the 
clinics will receive training in-person or virtual on identifying adverse events related to the consenting of 
participants and adherence to study activities and procedures. Written information is also included in the 
study manual shared with the care team at each participating clinic. Any deviations from study protocol 
will be reported to the PI within 24 hours.  
 
For child patient and clinic staff information  
 
1. Data storage plans: All data will be entered into a password protected VPN Egnyte system at the 
University of Colorado, Denver (UCD) and only clinic and participant IDs will be used to protect 
confidentiality. EMR data will be entered by medical personnel at each clinic and into a secured 
i2iTracks/Tableau centralized dashboard within the FQHC secured network. To protect confidentiality, 
all data will be shared with the UCD research team following guidelines from a signed Data Transfer 
Agreement (DTA), and after IRB approval. The PI and the data analyst at UCD will log in into a secured 
shared drive to access the data (i.e., Egnyte). The clinic personnel survey and interview transcripts will 
not contain identifying information. The survey will be distributed through a HIPAA compliant REDCap 
account or through a UCD’s Qualtrics account. No participant data will be stored in clinics’ iPads. Data 
will automatically be stored in UCD’s REDCap system. The EMR data will contain the child’s chart ID 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
9 
so we can link that data with the child’s ACEs screening data entered into the REDCap tablets. Once 
the files are linked, we will only use the child’s study ID and will delete the child’s chart ID from the 
files. When an internet connection is available, data is transmitted directly to REDCap using a SSL (TLS 
v1.2) connection. A hash-based message authentication code (HMAC) is used to verify the integrity of 
the data and to authenticate the sender. Participant entered data (i.e. task responses) are not stored 
or sent anywhere else. Data exists on the REDCap tablet device or on the UCD server. Data is wiped 
from the device after the MyCap app verifies that data has been successfully transmitted. By default, 
data is wiped.  
 
We are clarifying that clinic staff access to the REDCap system for the study will be done through 
remote access by using a link and a password protected login system and only for data entry. No data 
will be saved in the computer tablet used to access REDCap and clinic personnel will not have access 
to all study data. 
 
Results of the REDCap screenings will be kept in clinical records and for a group of children (n= 450) it 
will be kept in the REDCap system for research.  
 
2. Plans for dissemination of findings/results: The PI will ensure the clinical trial under this R21 award 
upon funding is registered and results information is submitted to ClinicalTrials.gov and to meet NIH 
policy on dissemination of NIH-funded clinical trial information (https://grants.nih.gov/policy/clinical-
trials/reporting/understanding/nih-policy.htm). The UCD system also has internal policies in place to 
ensure that clinical trials registration and results in reporting occur in compliance with NIH policy 
requirements. The PI will be responsible for submitting the protocol and the results to the website. 
Informed consent documents for the clinical trial will include a specific statement relating to the 
posting of clinical trial information at ClinicalTrials.gov. In December of 2021, we published the study 
protocol in a peer-reviewed journal - Implementation Science Communications. To facilitate 
replication in other studies, most measures we propose to use are available in the public domain as 
standard measures in pediatric settings (i.e., PSC), as well as implementation measures (leadership, 
climate, reach, feasibility, and acceptability). In addition, we will submit the study results to 
ClinicalTrials.gov and any updates to the study description of the page as needed. The PI will present 
findings to leading conferences from academic societies such as: Academy Health, the Society for 
Implementation Research Collaboration (SIRC), the Society for Social Work and Research (SSWR), and 
the Science of Dissemination and Implementation in Health (D&I). Besides conference presentations, 
we will draft and submit at least three manuscripts for peer review in professional publication: (1) 
study protocol paper (completed), (2) conceptual paper on the use of EPIS framework to guide an 
intervention mapping process (under review), (3) pilot stepped-wedge randomized trial study main 
effects, and (4) mechanisms of change of policy interventions in Federally Qualified Health Centers 
manuscripts. 
 
b. Instrumentation 
b.i. Questionnaires/Survey Measures for the study (names and citations):  
 
• 
The Pediatric Symptoms Checklist is comprised of two measures based on age: 
o The Baby Pediatric Symptom Checklist (BPSC) is a 12-item screening measuring children’s 
psychosocial function validated for children ages 0-18 months. Cronbach’s alpha adequate for 
subscales (alpha=0.75 to 0.83). 
o The Preschool Pediatric Symptom Checklist (PPSC) is an 18-item screening for measuring 
children’s psychosocial functioning validated for children ages 18-60 months. (alpha=0.92) 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
10 
• 
The Feasibility of Intervention Measure (FIM) is a self-reported 4-item instrument to evaluate 
feasibility of implementation efforts. Good internal consistency (alpha =0.89). Test-retest 
reliability (r=0.88). 
• 
The Acceptability of Intervention Measure (AIM) is a 4-item instrument to evaluate acceptability 
of implementation efforts. Good internal consistency (alpha =0.83). Test-retest reliability (r=0.83).  
• 
Fidelity checklist form will be developed by principal investigator and Dr. Greg Aarons. 
• 
Implementation Leadership Scale is a 12-item scale with four subscales measuring proactive 
leadership, knowledgeable leadership, supportive leadership, and perseverant leadership. 
Reliability for the total scale is strong (alpha=0.98). 
• 
Implementation Climate Scale is a 6-item scale measuring the strategic climate for the 
implementation of interventions.  
• 
Readiness for Change is a 12-item scale measuring clinic staff and supervisors’ perceptions of how 
prepared they are to start the ACEs screenings. This measure is only used at baseline for all clinics 
(Step 1 in the Stepped Wedge Design schedule).  
 
b.ii. Qualitative instruments:  
We will use a semi-structure interview protocols for Aim 2 in interviews with clinic personnel and with 
caregivers. We will allow participants to add in relevant areas of discussion. Each interview protocol 
has been uploaded to the IRB application.  
 
c. Data Analysis  
 
Quantitative data: We will use descriptive statistics (i.e., frequencies, means and standard deviation 
estimates) to describe the analytical sample. We will examine whether the implementation strategies (i.e., 
advisory board, use of REDCap handheld tablets, use of the PSC tool and an implementation coach) 
increases the number of children screened and referred to support services, compared to baseline. To do 
that, we will use methodology that accounts for repeated measures over time and that accounts for both 
within-person and across-person variation (i.e., Generalized linear mixed model (GLMM)) while 
controlling for clustering of patients within clinics and possible confounding by calendar time. Our analysis 
will include sex as a biological variable, testing for a differential effect on outcomes by child sex. We will 
report the magnitude of missing data and compare demographic and clinic characteristics and outcomes 
between observations with versus without missing data. In sensitivity analyses, we will perform multiple 
calculations (i.e., imputations) using fully conditional specification to obtain 10 complete datasets; 
summary estimates of intervention effects with 95% confidence limits over the 10 datasets will 
incorporate within- and between-dataset sampling variability. 
 
Comparison of randomized intervention sites to non-randomized control sites 
Due to the human and financial impact of COVID-19 epidemic, the randomized control periods had not 
implemented ACEs screening at the time of the pilot starting. We have identified two clinics that have 
implemented ACEs screening that are not part of the trial. 
 
To obtain an additional non-randomized estimate of the intervention effect, we will compare: 
 
(1) the control periods in the randomized clinics to comparable time periods in the non-randomized 
clinics; 
(2) the intervention periods in the randomized clinics to comparable time periods in the non-randomized 
clinics. 
 

PI: Monica Perez Jolles 
COMIRB # 22-0547 
Revised 05.10.24 
11 
We will use the same GLMM models as described above, with indicator variables for non-randomized vs 
randomized clinic, and randomized control vs intervention periods. Groups will be compared on ACEs 
screenings, and mental health referral rates. 
 
Qualitative data: All interviews will be audio-taped and professionally transcribed. Data analysis will 
include coding transcripts to identify common themes, capture analytical categories from the emerging 
themes, and depict associations between categories. Interviews will be transcribed using pseudonyms to 
increase confidentiality. Two coders (study team member and graduate student) will analyze data. Inter-
rate reliability (IRR) will be calculated as the proportion of agreed codes over the total number of codes in 
the document using Microsoft Word (Word) and Excel. Rigor strategies include documenting coding 
decisions and thematic development. A detailed codebook will be developed.