Eliminating Monitor Overuse (EMO) Hybrid Effectiveness-Deimplementation Trial
Phase: NA
Conditions: Bronchiolitis Acute Viral
Sponsor: Children's Hospital of Philadelphia

 
Template version: January 30, 2020 
Title 
Eliminating Monitor Overuse (EMO) Hybrid Effectiveness-Deimplementation Trial 
Short Title 
EMO Trial 
eIRB Number 
21-018560 
Protocol Date 
December 12, 2024 
CHOP Principal Investigator: 
Chris Bonafide, MD, MSCE 
Children’s Hospital of Philadelphia 
Buerger Center for Advanced Pediatric Care 
3500 Civic Center Blvd, 12th Floor 
Philadelphia, PA 19104 
Office Phone: 267-426-2901 
email: bonafide@chop.edu 
 
 
Overall Study Co-Principal Investigators: 
Chris Bonafide, MD, MSCE 
Children’s Hospital of Philadelphia 
Buerger Center for Advanced Pediatric Care 
3500 Civic Center Blvd, 12th Floor 
Philadelphia, PA 19104 
Office Phone: 267-426-2901 
email: bonafide@chop.edu 
 
 
Rinad Beidas, PhD 
Northwestern University  
Feinberg School of Medicine 
625 N. Michigan Avenue, 21st Floor 
Chicago, Illinois 60661 
Office Phone: 312-503-0546 
email: Rinad.beidas@northwestern.edu 
 

ii 
 
 
 
 
 
 
 
 
 
 
TABLE OF CONTENTS 
TABLE OF CONTENTS II 
ABSTRACT 
VII 
1 
BACKGROUND INFORMATION AND RATIONALE 1 
1.1 
INTRODUCTION 
1 
1.2 
RELEVANT LITERATURE AND DATA 
1 
1.3 
COMPLIANCE STATEMENT 
3 
2 
STUDY OBJECTIVES 4 
2.1 
SPECIFIC AIM 1 
4 
2.2 
SPECIFIC AIM 2 
4 
2.3 
SPECIFIC AIM 3 
4 
3 
INVESTIGATIONAL PLAN 4 
3.1 
GENERAL SCHEMA OF STUDY DESIGN 4 
3.1.1 
BASELINE PHASE 
5 
3.1.2 
ACTIVE DEIMPLEMENTATION PHASE 
5 
3.1.3 
SUSTAINABILITY PHASE 
5 
3.1.4 
EXPLORATORY PHASE 
5 
3.2 
ALLOCATION TO TREATMENT GROUPS (RANDOMIZATION) AND BLINDING ........ 5 
3.3 
STUDY DURATION, ENROLLMENT AND NUMBER OF SITES .......................................... 6 
3.3.1 
DURATION OF SUBJECT STUDY PARTICIPATION 6 
3.3.2 
TOTAL NUMBER OF STUDY SITES/TOTAL NUMBER OF SUBJECTS PROJECTED .... 6 
3.4 
STUDY POPULATIONS WITH INCLUSION AND EXCLUSION CRITERIA ...................... 6 
3.4.1 
POPULATION 1: HOSPITAL STAFF 6 
3.4.2 
POPULATION 2A 
7 
3.4.3 
POPULATION 2B 
8 
3.4.4 
POPULATION 3 
9 
4 
STUDY PROCEDURES 
9 
4.1 
DESCRIPTION OF INTERVENTIONS (DEIMPLEMENTATION STRATEGIES) .............. 9 
4.1.1 
UNLEARNING (BOTH STUDY ARMS) 
9 
4.1.2 
SUBSTITUTION (EXPERIMENTAL ARM ONLY) 
10 
4.2 
PROCEDURES WITHIN EACH STUDY PHASE (SEE ALSO TABLE 3) ............................ 12 
4.2.1 
BASELINE PHASE (P1) 
12 
4.2.2 
ACTIVE DEIMPLEMENTATION PHASE (P2) 
12 
4.2.3 
SUSTAINABILITY PHASE (P3) 
13 
4.2.4 
EXPLORATORY PHASE (P4) 13 
5 
STUDY EVALUATIONS AND MEASUREMENTS 
14 
Deleted: VI
Deleted: 11
Deleted: 11
Deleted: 12
Deleted: 13

iii 
 
 
 
 
 
 
 
 
 
 
5.1 
MEASURE USE OF CONTINUOUS SPO2 MONITORING IN PATIENTS WITH 
BRONCHIOLITIS WHO ARE IN ROOM AIR USING DIRECT OBSERVATION ............................... 14 
5.1.1 
MEDICAL RECORD REVIEW 
14 
5.2 
QUESTIONNAIRES 
16 
5.2.1 
MEASURE ACCEPTABILITY USING QUESTIONNAIRE .................................................... 17 
5.2.2 
MEASURE ROUTINIZATION & INSTITUTIONALIZATION USING QUESTIONNAIRE
 
17 
5.2.3 
MEASURE IMPLEMENTATION CLIMATE AND LEADERSHIP USING 
QUESTIONNAIRE 
17 
5.2.4 
MEASURE PSYCHOLOGICAL REACTANCE USING QUESTIONNAIRE ....................... 17 
5.3 
MEASURE FIDELITY TO EACH DEIMPLEMENTATION STRATEGY ........................... 17 
5.4 
ASSESS COSTS ASSOCIATED WITH DELIVERING EACH DEIMPLEMENTATION 
STRATEGY 
18 
5.5 
CLINICAL OUTCOMES 
18 
5.5.1 
MEASURE LENGTH OF HOSPITAL STAY (LOS) IN BRONCHIOLITIS PATIENTS ..... 18 
5.5.2 
MEASURE OXYGEN SUPPLEMENTATION DURATION IN BRONCHIOLITIS 
PATIENTS 
18 
5.6 
QUALITATIVE INTERVIEWS OF STAFF AIMING TO BETTER UNDERSTAND 
MECHANISMS OF PRACTICE CHANGE IN RESPONSE TO EACH DEIMPLEMENTATION 
STRATEGY 
18 
5.7 
QUALITATIVE INTERVIEWS OF PARENTS/GUARDIANS OF CHILDREN 
HOSPITALIZED WITH BRONCHIOLITIS TO EXPLORE THEIR PERCEPTIONS OF, AND 
REACTIONS TO, DEIMPLEMENTATION ACTIVITIES 
19 
5.8 
UNDERUSE OF CONTINUOUS SPO2 MONITORING IN PATIENTS WITH 
BRONCHIOLITIS WHO ARE RECEIVING ≥2L/MIN OXYGEN (ARE INAPPROPRIATELY 
UNMONITORED). 
19 
5.9 
MONITORING FOR EQUITABLE DEIMPLEMENTATION ................................................ 20 
6 
STATISTICAL CONSIDERATIONS 20 
6.1 
PRIMARY ENDPOINT 20 
6.2 
SECONDARY ENDPOINTS 20 
6.3 
STATISTICAL METHODS 
20 
6.3.1 
PRIMARY ENDPOINT: SUSTAINABILITY OF GUIDELINE-CONCORDANT 
DEIMPLEMENTATION OF CONTINUOUS SPO2 MONITORING ........................................................ 20 
6.3.2 
THE CHANGES IN PENETRATION OF GUIDELINE-CONCORDANT CARE WITHIN 
EACH STUDY ARM 
21 
6.3.3 
THE ACCEPTABILITY OF DEIMPLEMENTATION AMONG STAFF, COMPARED 
BETWEEN NURSES AND PHYSICIANS 
21 
6.3.4 
THE MEDIATION EFFECTS OF ROUTINIZATION AND INSTITUTIONALIZATION OF 
THE PRACTICE CHANGE ON THE RELATIONSHIP BETWEEN THE DEIMPLEMENTATION 
STRATEGIES AND SUSTAINABILITY, PENETRATION 
21 
6.3.5 
THE MODERATION EFFECTS OF IMPLEMENTATION CLIMATE, 
IMPLEMENTATION LEADERSHIP, AND PSYCHOLOGICAL REACTANCE AMONG STAFF ON 
Deleted: 13
Deleted: 15
Deleted: 15
Deleted: 15
Deleted: 16
Deleted: 16
Deleted: 16
Deleted: 16
Deleted: 17
Deleted: 17
Deleted: 17
Deleted: 17
Deleted: 17
Deleted: 18
Deleted: 18
Deleted: 18
Deleted: 18
Deleted: 18
Deleted: 19
Deleted: 19
Deleted: 19
Deleted: 20
Deleted: 20

iv 
 
 
 
 
 
 
 
 
 
 
THE RELATIONSHIP BETWEEN THE DEIMPLEMENTATION STRATEGIES AND 
SUSTAINABILITY, PENETRATION 
22 
6.3.6 
FIDELITY TO EACH DEIMPLEMENTATION STRATEGY ................................................ 22 
6.3.7 
COSTS ASSOCIATED WITH DELIVERING EACH DEIMPLEMENTATION STRATEGY
 
22 
6.3.8 
THE RELATIONSHIP BETWEEN PENETRATION OF GUIDELINE-CONCORDANT 
CARE AND LENGTH OF HOSPITAL STAY (LOS) IN BRONCHIOLITIS PATIENTS IN EACH 
STUDY PHASE 22 
6.3.9 
THE RELATIONSHIP BETWEEN PENETRATION OF GUIDELINE-CONCORDANT 
CARE AND OXYGEN SUPPLEMENTATION DURATION IN BRONCHIOLITIS PATIENTS IN 
EACH STUDY PHASE 22 
6.3.10 
THEMES FROM QUALITATIVE INTERVIEWS OF STAFF, PARENTS/GUARDIANS .. 23 
6.3.11 
UNDERUSE OF CONTINUOUS SPO2 MONITORING IN PATIENTS WITH 
BRONCHIOLITIS WHO ARE RECEIVING ≥2L/MIN OXYGEN (ARE INAPPROPRIATELY 
UNMONITORED) WITHIN EACH STUDY ARM, ACROSS ALL STUDY PHASES ........................... 23 
6.3.12 
EXPLORATION OF DEIMPLEMENTATION INEQUITIES ................................................. 23 
6.4 
SAMPLE SIZE AND POWER 23 
6.4.1 
AIM 1 SAMPLE SIZE AND POWER 23 
6.4.2 
AIM 2 SAMPLE SIZE AND POWER 24 
6.4.3 
AIM 3 SAMPLE SIZE AND POWER 24 
6.5 
INTERIM ANALYSIS 24 
7 
SAFETY MANAGEMENT 
25 
7.1 
CLINICAL ADVERSE EVENTS 
25 
7.2 
ADVERSE EVENT REPORTING 
25 
8 
STUDY ADMINISTRATION 25 
8.1 
TREATMENT ASSIGNMENT METHODS 
25 
8.1.1 
RANDOMIZATION 
25 
8.1.2 
BLINDING 
25 
8.2 
DATA COLLECTION AND MANAGEMENT 
25 
8.2.1 
DATABASE 26 
8.2.2 
DATA ENTRY AND DATA QUALITY MODULE 
26 
8.2.3 
REPORTS DASHBOARD MODULE 26 
8.2.4 
TESTING 
26 
8.2.5 
OPERATIONS AND ADMINISTRATION 
27 
8.2.6 
MAINTENANCE 
27 
8.2.7 
USER TRAINING 
27 
8.2.8 
USER SUPPORT 
27 
8.2.9 
REPORTS 
27 
8.2.10 
SHARING DATA WITH ANALYTIC CORE 27 
Deleted: 20
Deleted: 20
Deleted: 21
Deleted: 21
Deleted: 21
Deleted: 21
Deleted: 21
Deleted: 21
Deleted: 21
Deleted: 22
Deleted: 22
Deleted: 23
Deleted: 23
Deleted: 23
Deleted: 23
Deleted: 23
Deleted: 24
Deleted: 24
Deleted: 24
Deleted: 24
Deleted: 24
Deleted: 24
Deleted: 24
Deleted: 25
Deleted: 25
Deleted: 25
Deleted: 25
Deleted: 25
Deleted: 25
Deleted: 25
Deleted: 26

v 
 
 
 
 
 
 
 
 
 
 
8.2.11 
DATABASE LOCK 
27 
8.3 
CONFIDENTIALITY 27 
8.3.1 
RECORD RETENTION POLICY RECONCILIATION 28 
8.4 
REGULATORY AND ETHICAL CONSIDERATIONS 28 
8.4.1 
DATA AND SAFETY MONITORING PLAN 28 
8.4.2 
RISK ASSESSMENT 
32 
8.4.3 
POTENTIAL BENEFITS OF TRIAL PARTICIPATION ......................................................... 33 
8.4.4 
RISK-BENEFIT ASSESSMENT 
33 
8.5 
RECRUITMENT STRATEGY 33 
8.5.1 
RECRUITMENT OF HOSPITAL SITES 
33 
8.5.2 
RECRUITMENT OF INDIVIDUALS 34 
8.6 
INFORMED CONSENT/ASSENT AND HIPAA AUTHORIZATION .................................... 35 
8.6.1 
POPULATION 1: HOSPITAL STAFF 35 
8.6.2 
POPULATIONS 2A AND 2B 36 
8.6.3 
POPULATION 3 
37 
8.7 
PAYMENT TO SUBJECTS/FAMILIES 
38 
9 
PUBLICATION 
39 
10 
REFERENCES 
39 
 
 
 
Deleted: 26
Deleted: 26
Deleted: 26
Deleted: 26
Deleted: 26
Deleted: 31
Deleted: 32
Deleted: 32
Deleted: 32
Deleted: 32
Deleted: 33
Deleted: 34
Deleted: 34
Deleted: 34
Deleted: 36
Deleted: 37
Deleted: 37
Deleted: 37

vi 
 
 
 
 
 
 
 
 
 
 
 
 
ABBREVIATIONS AND DEFINITIONS OF TERMS 
A&F 
 
Audit and feedback 
AE 
 
Adverse event 
AIM 
 
Acceptability of Implementation Measure 
BEEP 
 
Best Evidence for Effective monitoring Practice (guideline) 
CDS 
 
Clinical Decision Support 
CFIR 
 
Consolidated framework for implementation research 
CHOP 
 
Children’s Hospital of Philadelphia 
CI 
 
Confidence interval 
COI 
 
Conflict of interest 
CRCU 
 
Clinical Research Computing Unit at the University of Pennsylvania 
DCC 
 
Data Coordinating Center 
DMS 
 
Data management system 
DSMB 
 
Data and safety monitoring board 
EHR 
 
Electronic Health Record 
EMO 
 
Eliminating Monitor Overuse 
FDA 
 
Food and Drug Administration 
ICU 
 
Intensive Care Unit 
IT 
 
Information technology 
LOS 
 
Length of (hospital) stay 
MIS-C 
 
Multisystem inflammatory syndrome in children  
MOXY 
 
Monitoring OXYgen in Infants Hospitalized With Bronchiolitis (clinical trial) 
NHLBI 
 
National Heart, Lung, and Blood Institute 
P1,2,3,4 
 
Phase 1, 2, 3, or 4 
PASS 
 
Power and Sample Size (software) 
PCC 
 
Pediatric Clinical Decision Support Collaborative 
Penn 
 
University of Pennsylvania 
PHI 
 
Protected Health Information 
PRIS 
 
Pediatric Research in Inpatient Settings Network 
RDC 
 
Remote data entry module 
RTD 
 
Research Technologies Department 
SARS-CoV-2  
 
Severe Acute Respiratory Syndrome Coronavirus 2 
SD 
 
Standard deviation 
SpO2 
 
Pulse oximetry 
UK 
 
United Kingdom 
US 
 
United States 
 
 
 

 
vii 
  
 
 
 
ABSTRACT 
Context:   
Bronchiolitis, an infectious lung disease, is the leading cause of infant hospitalization and is responsible 
for over 100,000 admissions and $1.7 billion in hospital charges annually. Continuous pulse oximetry 
monitoring of oxygen saturation (SpO2) is a common intervention in bronchiolitis, yet it does not improve 
outcomes when it is used during periods of the hospitalization when supplemental oxygen is not being 
administered (i.e. when the patient is “in room air”). The American Academy of Pediatrics, Choosing 
Wisely, and the Best Evidence for Effective monitoring Practice guidelines all discourage continuous 
SpO2 monitoring in children with bronchiolitis who are in room air, making it an opportune target for 
deimplementation. 
Objectives:  
Primary: To test a “usual approach” unlearning deimplementation strategy (educational outreach with 
audit & feedback) vs. an enhanced unlearning + substitution deimplementation strategy (adding an 
electronic health record-integrated clinical pathway). The primary outcome is sustainability of guideline-
concordant deimplementation in bronchiolitis patients who are in room air. 
Secondary: (a) Identify deimplementation strategy mechanisms. (b) Examine the effects of 
deimplementation on clinical outcomes and unintended consequences. 
Study Design:  
Multicenter hybrid type III effectiveness-deimplementation trial with a longitudinal cluster-randomized 
design. 
Setting/Participants: 
The trial will be conducted on PRIS Network hospital inpatient units that care for children with 
bronchiolitis, excluding ICUs, emergency departments, and step-down units. Subject populations include 
(1) bronchiolitis patients, (2) parents or guardians of bronchiolitis patients who participate in qualitative 
interviews, (3) hospital staff who care for bronchiolitis patients and participate in questionnaires and 
interviews, and (4) Clinical Decision Support (CDS) Coaches. 
Study Interventions and Measures:  
The study interventions are directed toward hospital staff caring for bronchiolitis patients and include 
educational outreach, audit & feedback, and an electronic health record-integrated clinical pathway. 
Sustainability, the primary outcome, will be assessed as a longitudinal difference in differences 
comparison of the penetration of guideline-concordant care during at baseline (percentage of bronchiolitis 
patients in room air who are appropriately not being continuously SpO2-monitored at baseline, before any 
deimplementation strategies are deployed) compared to the penetration 1 year after withdrawal of 
educational outreach and A&F, and contrasted between study arms (see study diagram). We will also 
assess other deimplementation outcomes (e.g. fidelity and cost), assess practice routinization and 
institutionalization as potential mediators, and examine clinical outcomes such as length of hospital stay 
and unintended monitoring underuse that could result from deimplementation. 
 

 
viii 
  
 
 
 
 
 
Gap between Phase 1 and 2: Minimum of 6 months 
Can be extended beyond 6 months at discretion of Steering Committee in any of the following conditions: 
• 
≥20% of sites in either arm is unprepared to start active deimplementation 
• 
National RSV percent positivity is <2% 
• 
Or for other reasons, with DSMB and NHLBI approval 
 
Washout between Phase 2 and 3: Minimum of 6 months 
Can be extended beyond 6 months at discretion of Steering Committee in any of the following conditions: 
• 
National RSV percent positivity is <2% 
• 
July (new residents) falls within proposed Phase 3 
• 
Or for other reasons, with DSMB and NHLBI approval 
 
Figure 1. Trial diagram. The trial includes 3 main phases (baseline, active deimplementation, and 
sustainability) and an exploratory fourth phase. During the baseline phase (P1), we will measure baseline 
penetration of guideline-concordant care in approximately 45 hospitals and confirm that sites have 
adequate bronchiolitis volumes, infrastructure, and capability to complete the trial. Penetration is defined 
as the percentage of bronchiolitis patients who are in room air (no supplemental oxygen) and are 
receiving guideline-concordant care (this means they are not being continuously SpO2-monitored). Based 
on the P1 data, we will exclude hospitals with feasibility challenges and those with already high 
penetration of 80% or higher. We anticipate that this will result in 32-38 randomizable hospitals. If more 
than 38 hospitals remain after those exclusions, we will then randomize the 38 hospitals with the lowest 
baseline penetration. Of those that remain, we will randomize no fewer than 32 and no more than 38 
hospitals to either: the unlearning only arm (educational outreach with A&F) or the unlearning + 
substitution arm (adding an EHR-integrated clinical pathway to facilitate the transition from continuous 
SpO2 monitoring to intermittent SpO2 measurement when clinically appropriate). In the active 
deimplementation phase (P2), these strategies will be deployed in the 32-38 sites. At the end of P2, 
unlearning (educational outreach with A&F) will be withdrawn from both arms. In the sustainability phase 
(P3), substitution (EHR-integrated pathway) will be maintained exclusively in the unlearning + substitution 
arm and the primary outcome (sustainability) will be contrasted between arms as a longitudinal difference 
in differences comparison of penetration at baseline (P1) versus in the sustainability phase (P3). In the 
exploratory phase (P4), we will assess longer-term sustainability in a subgroup of up to 16 hospitals, 2 
years after withdrawal of education and A&F.
Baseline
measurement
R
Unlearning Only
Unlearning
+
Substitution
Withdraw Unlearning
Measure primary endpoint
Withdraw Unlearning
Maintain Substitution
Measure primary endpoint
Phase 1
Baseline phase
Phase 2
Active deimplementation phase
Phase 3
Sustainability phase
Phase 4
Exploratory phase
Explore long-term sustainability 
2 years after active 
deimplementation in a subgroup 
of hospitals
P1
P2
P3
P4
Hospitals exit study if:
a. Problem already solved: Penetration ≥80%
c. Feasibility challenges: <15 patients
d. Choose to drop out

 
 
 
1 
 
 1 
BACKGROUND INFORMATION AND RATIONALE 
1.1 
Introduction 
Deimplementing overused medical interventions is an essential step to maximize quality and 
minimize waste in our health care system.1 Bronchiolitis, an infectious lung disease, is the leading 
cause of infant hospitalization and is responsible for over 100,000 admissions and $1.7 billion in 
hospital charges annually.2,3 Continuous pulse oximetry monitoring of oxygen saturation (SpO2) is a 
common intervention in bronchiolitis, yet it does not improve outcomes when it is used during periods 
of the hospitalization when supplemental oxygen is not being administered (i.e. when the patient is 
“in room air”).4,5 In addition to being ineffective, its overuse can prolong the hospital stay,6–8 increase 
risk of harm,9 and contribute to alarm fatigue, which threatens patient safety beyond the monitored 
patient.10,11 Thus the American Academy of Pediatrics,12 Choosing Wisely,13 and the Best Evidence 
for Effective monitoring Practice (BEEP) guidelines14 all discourage continuous SpO2 monitoring in 
children with bronchiolitis who are in room air, making it an opportune target for deimplementation.15 
1.2 
Relevant Literature and Data 
Deimplementing overused health interventions that are not supported by evidence is 
essential to maximizing quality and value while minimizing harm, waste, and inefficiencies in 
the health care system.1,16 Medical overuse is defined by the Institute of Medicine as care in the 
absence of a clear medical indication, or when the benefit does not outweigh the risk.17 Overuse can 
be measured directly when evidence-based guidelines (a) specify conditions in which a practice is 
appropriate and (b) also consider the balance between benefits and harms.18 Deimplementation is 
the systematic divestment of overused practices that do not improve outcomes.19,20 Recently, 
experts have called for deimplementation research to identify the best strategies for minimizing low-
value care delivery to children, the focus of this proposal.21  
Medical overuse is a documented problem during inpatient treatment of acute viral 
bronchiolitis (“bronchiolitis”), a common acute lung disease caused by respiratory viral 
infection in children under 2 years old.2,3,12 In the US, bronchiolitis leads to over 100,000 
hospitalizations annually22 in a seasonal pattern, with most cases occurring between December and 
March.23 Treatment often includes feeding support, nasal suctioning, and supplemental oxygen.12 
Bronchiolitis patients are often continuously SpO2-monitored despite evidence that it does not 
improve outcomes when it is used during periods of the hospitalization when supplemental oxygen is 
not being administered (i.e. patient is “in room air”).24 Rather, in those patients continuous SpO2 
monitoring may overidentify brief, self-limited desaturations that do not impact patient outcomes.25 As 
a result, three sets of national guidelines now discourage the use of continuous SpO2 monitoring in 
hospitalized children with bronchiolitis who are in room air.12–14 
Overuse of continuous SpO2 monitoring is associated with increased oxygen administration, 
prolonged length of stay, and increased costs in retrospective studies.7,8 In addition, in a 
clinical trial, infants with bronchiolitis randomized to oximeters that inflated SpO2 readings to appear 
slightly better than the actual values were discharged 10 hours sooner than a control group, with no 
differences in clinical outcomes and lower hospital costs.6,26 This finding suggests that the SpO2 
value may be over-weighted in clinician decision making. Unfortunately, longer hospital stays 
translate into increased risk of injury: one study reported 12 unnecessary harms inflicted per 100 
bronchiolitis admissions (e.g. falls from cribs, medication errors).9 In addition, longer hospitalizations 
prolong the stress and anxiety experienced by parents,27 who also suffer financial impacts.28 Given 
that nearly all children hospitalized with bronchiolitis recover fully,22 and deteriorating after showing 
clinical improvement occurs in just 4% of cases,29 excess monitoring causes unnecessary burden in 
a high prevalence disease for which experts have called on clinicians to “safely do less.”30 
Overmonitoring also contributes to alarm fatigue, which threatens patient safety beyond the 
monitored patient. Alarm fatigue is a well-described result of exposure to non-actionable alarms, 
leading to desensitization.31,32 In children’s hospital wards, monitors alarm 42-155 times per patient 
day.33 SpO2 alarms are the most frequent but 99% are non-actionable; this “crying wolf” effect leads 

 
 
 
2 
 
 clinicians to distrust alarms and slows their response times to all alarm types, including those that 
are potentially critical. Response delays averaged 7 minutes in a study we conducted using 
video.10,11,34 One way to reduce alarm fatigue is to reduce monitoring in populations in which it has 
no benefit,35 including bronchiolitis patients in room air.  
Two clinical trials randomized at the patient level demonstrated that intermittent SpO2 
measurement is an equally safe alternative to continuous SpO2 monitoring for children in 
room air. A 4-hospital trial found no significant difference in safety,4 and the 6-hospital Monitoring 
OXYgen in Infants Hospitalized With Bronchiolitis (MOXY) trial also showed no significant 
differences in adverse safety outcomes, with nursing satisfaction higher in the intermittent monitoring 
group.5  
Specific attention to sustainability is needed, as sustaining guideline-concordant care over 
time is challenging, important, and understudied.36 Across a broad range of clinical specialties, 
fewer than half of studies measuring clinicians’ guideline-concordant care over time report successful 
sustainment >1 year after implementation.37 For example, in a cluster-randomized trial of educational 
outreach with A&F to deimplement inappropriate antibiotic prescribing to children with acute 
respiratory tract infections, the immediate post-implementation results showed a nearly 50% 
reduction in inappropriate prescribing.38 However, 18 months after withdrawal of A&F, inappropriate 
prescribing practices had returned to baseline. No studies have rigorously evaluated sustainability of 
deimplementation in pediatric hospital settings. 
When aiming to align clinical practice with guidelines in pediatric hospital medicine, the usual 
approach involves unlearning-based strategies.39–41 Enhancing unlearning with electronic 
health record (EHR)-based substitution may boost sustainability of deimplementation. In a 
special report, the NHLBI Implementation Science Work Group concluded that A&F and educational 
outreach are generally effective in improving outcomes.42 A&F in particular is so widely accepted that 
experts consider studies of A&F vs. no A&F controls “unlikely to provide valuable insights” and 
“research waste,” instead calling for trials of A&F alone vs. A&F + co-interventions as we propose in 
this application.43,44 Helfrich’s Dual Process Theory-based deimplementation framework45 offers a 
useful basis for experimentally testing A&F co-interventions, separating strategies into two 
categories targeting: (1) unlearning the ineffective practice using knowledge-based methods (e.g., 
presenting clinicians with evidence and guidelines, conducting A&F) and (2) substituting the 
ineffective practice with an alternative practice that becomes integrated into everyday work (e.g., 
using EHR-integrated clinical pathways to support intermittent SpO2 measurement). This is 
consistent with evidence that, when aiming to reduce low-value care, (a) multicomponent 
interventions have the greatest potential for success, (b) education is necessary but rarely sufficient, 
and (c) A&F and EHR-based “clinical decision support” (CDS) are the most promising strategies to 
address overuse.46–48 Two small-scale continuous SpO2 monitoring deimplementation efforts in 
bronchiolitis that used enhanced approaches showed short-term success. Schondelmeyer et al. 
used educational outreach, audit & feedback (A&F), and EHR guidance and found a significant 
reduction in continuous SpO2 monitoring duration.49 Mittal et al. used a clinical pathway that guided 
staff to limit continuous SpO2 monitoring to only patients with severe bronchiolitis, resulting in length 
of stay reduction from 53 to 33 hours.50 EHR-based strategies also offer promise of sustainability. In 
a 60-hospital collaborative focused on infant sepsis guideline adherence, unlearning strategies were 
not sustained due to time constraints and competing priorities.39,51 However, EHR-based strategies 
were not only sustained but increased in use 11 months after the active intervention period.51 The 
authors concluded that EHR-based strategies as well as clinical pathways and policies were the 
most feasible for sustaining guideline-concordant care.51 Randomized trials are needed to evaluate 
this conclusion.  
 
 

 
 
 
3 
 
  
NHLBI U01-Funded Preliminary Studies. First, our 
team led a 56-hospital, 3,612-patient observational study 
of SpO2 monitor overuse.24 Across all hospitals, 46% of 
bronchiolitis patients in room air were continuously SpO2-
monitored, discordant with guidelines. The remaining 
54% were appropriately unmonitored. In other words, at 
baseline, the penetration of guideline-concordant care 
was just 54%, with strikingly wide variation in penetration 
ranging from 8% to 98% between hospitals, not 
attributable to differences in patient populations. This 
variation suggests that achieving high penetration is 
feasible. Second, we conducted qualitative interviews 
with clinicians and administrators from 12 hospitals to 
understand barriers to deimplementation, guided by the Consolidated Framework for Implementation 
Research (CFIR).52,53 Key barriers included educational gaps, lack of clear instructions of when to 
monitor, and normative culture of monitoring. Third, we convened 39 stakeholders from 15 hospitals 
to develop deimplementation strategies using implementation mapping.54 Applying a 
deimplementation framework,45 we categorized strategies into: (1) unlearning (educational outreach 
with A&F) and (2) substitution (replacing continuous monitoring with intermittent measurement, 
supported by an EHR-integrated clinical pathway). Fourth, we performed a 6-hospital, single arm trial 
of educational outreach with A&F, using historical controls.55 Each hospital increased penetration 
compared to baseline (Table 1, manuscript forthcoming). In a survey, 1,193 nurses and physicians 
considered education and A&F to be feasible and acceptable.56  
Building on this body of evidence and our previous NHLBI-funded preparatory work, we 
propose to conduct the Eliminating Monitor Overuse (EMO) SpO2 hybrid type III clinical trial. 
Despite the evidence and guidelines, continuous SpO2 monitoring continues to be overused in 
hospitalized children with bronchiolitis, making it a prime target for deimplementation efforts. In the 
56-hospital observational study we conducted, 46% of bronchiolitis patients who were in room air 
were continuously SpO2-monitored, discordant with guidelines.24 In the EMO SpO2 trial, we will test 
the effects of a “usual approach” unlearning strategy (educational outreach with A&F) vs. an 
enhanced unlearning + substitution strategy (adding an EHR-integrated clinical pathway) on the 
sustainability of guideline-concordant deimplementation in children with bronchiolitis who are in room 
air. The trial will allow us to determine if the EHR-integrated clinical pathway enhances sustainability, 
with a focus on its effects 1 year after withdrawal of educational outreach and A&F, an outcome 
highly relevant to clinical practice and implementation science. 
1.3 
Compliance Statement 
This study will be conducted in full accordance all applicable Children’s Hospital of Philadelphia 
Research Policies and Procedures and all applicable Federal and state laws and regulations. All 
episodes of noncompliance will be documented. 
The investigators will perform the study in accordance with this protocol, will obtain consent as 
detailed herein, and will report unanticipated problems involving risks to subjects or others in 
accordance with Children’s Hospital of Philadelphia IRB Policies and Procedures and all federal 
requirements. Collection, recording, and reporting of data will be accurate and will ensure the 
privacy, health, and welfare of research subjects during and after the study.  
Table 1. 6-hospital pilot trial results.  
Penetration of guideline-concordant care 
Hospital 
Baseline 
% (95% CI) 
Intervention 
% (95% CI) 
P 
1 
22% (13-30%) 
57% (50-64%) 
<.001 
2 
31% (17-44%) 
91% (84-99%) 
<.001 
3 
34% (26-42%) 
73% (65-81%) 
<.001 
4 
35% (24-46%) 
70% (61-79%) 
<.001 
5 
37% (27-46%) 
80% (75-86%) 
<.001 
6 
78% (72-84%) 
90% (87-93%) 
.001 
Overall 
47% (43-51%) 
77% (75-80%) 
<.001 

 
 
 
4 
 
 2 
STUDY OBJECTIVES 
2.1 
Specific Aim 1 
To compare the effects of an unlearning-based deimplementation strategy alone versus a 
deimplementation strategy that combines unlearning and substitution-based approaches on 
deimplementation outcomes. 
The objective of this aim is to test the effects of unlearning only (educational outreach with A&F) vs. 
unlearning + substitution (educational outreach with A&F plus an EHR-integrated clinical pathway) 
on the primary trial outcome: sustainability of guideline-concordant deimplementation. 
Hypothesis. Compared to the unlearning only strategy, the unlearning + substitution strategy will 
result in better sustainability of guideline-concordant deimplementation 1 year after withdrawal of 
educational outreach and A&F. 
2.2 
Specific Aim 2 
To identify deimplementation strategy mechanisms linked to penetration and sustainability using 
mixed methods. 
The objective of this aim is to identify, test, and deeply explore mediators and moderators of the 
relationship between strategies and outcomes. We will also use qualitative inquiry of clinicians to 
explore mechanisms, and qualitative inquiry of parents to understand perceptions of 
deimplementation. 
Hypothesis. Primary. The unlearning + substitution strategy will result in better penetration and 
sustainability than the unlearning only strategy because the EHR-integrated pathway will produce 
better routinization57 and institutionalization57 of the change in practice (mediators). Secondary. 
Strategies in both study arms will be more effective in settings with more positive implementation 
climate58 and leadership,59 and where clinicians respond to deimplementation messaging with less 
psychological reactance (moderators).60 
2.3 
Specific Aim 3 
To examine the effects of deimplementation on clinical outcomes and unintended consequences. 
The objective of this aim is to examine clinical outcomes and unintended adverse consequences for 
patients associated with the deimplementation strategies themselves and/or from the resulting 
increases in penetration. 
Hypothesis. Primary. Increased penetration of guideline-concordant care will be associated with 
decreased hospital length of stay (LOS) in bronchiolitis patients. Secondary. Increased penetration 
will also be associated with decreased duration of oxygen supplementation in bronchiolitis patients. 
 
3 
INVESTIGATIONAL PLAN 
3.1 
General Schema of Study Design 
We will conduct a hybrid type III effectiveness-deimplementation trial61 with a longitudinal cluster-
randomized design within the Pediatric Research in Inpatient Settings (PRIS) Network.62 The trial 
includes 3 main phases (baseline, active deimplementation, and sustainability) and an exploratory 
fourth phase. Given the seasonal pattern of bronchiolitis, each phase was originally designed to take 
place during a 4-month winter period (December-March) spanning across 4 years. Due to the 
changes in bronchiolitis seasonality post-Covid, we uncoupled the study phases from winters. Each 
phase is still intended to be 4 months, however each phase may be extended beyond 4 months at 

 
 
 
5 
 
 the discretion of the Principal Investigators if challenging circumstances arise that limit sites’ abilities 
to collect sufficient data (e.g. staffing challenges or differences in seasonal bronchiolitis patterns due 
to the Covid pandemic). We also set parameters around the required gaps that must occur between 
each phase. See Figure 1 for those details. 
3.1.1 
Baseline Phase 
During the baseline phase (P1), we will measure baseline penetration of guideline-concordant care 
in approximately 45 hospitals and confirm that sites have adequate bronchiolitis volumes, 
infrastructure, and capability to complete the trial. Penetration is defined as the percentage of 
bronchiolitis patients who are in room air (no supplemental oxygen) and are receiving guideline-
concordant care (this means they are not being continuously SpO2-monitored). Based on the P1 
data, we will exclude hospitals with inadequate patient volumes or data collection challenges. We will 
then cluster-randomize the 32-38 hospitals with the lowest baseline penetration to either: the 
unlearning only arm (educational outreach with A&F) or the unlearning + substitution arm (adding an 
EHR-integrated clinical pathway to facilitate the transition from continuous SpO2 monitoring to 
intermittent SpO2 measurement when clinically appropriate).  
3.1.2 
Active Deimplementation Phase 
In the active deimplementation phase (P2), the above strategies are deployed in the 32-38 sites. 
At the end of P2, unlearning (educational outreach with A&F) is withdrawn from both arms.  
3.1.3 
Sustainability Phase 
In the sustainability phase (P3), substitution (EHR-integrated pathway) is maintained exclusively in 
the unlearning + substitution arm and the primary outcome (sustainability) is contrasted between 
arms as a longitudinal difference in differences comparison of penetration at baseline (P1) versus in 
the sustainability phase (P3).  
3.1.4 
Exploratory Phase 
In the exploratory phase (P4), we assess longer-term sustainability in a subgroup of up to 16 
hospitals, 2 years after withdrawal of education and A&F.  
3.2 
Allocation to Treatment Groups (Randomization) and Blinding 
This cluster-randomized trial uses covariate-constrained randomization63–65 to randomize at the 
hospital level, accounting for important hospital-level covariates and minimizing differences in these 
covariates between study arms. Following the baseline measurement phase in approximately 45 
hospitals during P1 (described above), we will exclude hospitals with inadequate patient volumes or 
data collection challenges from further trial participation and then cluster-randomize the 32-38 
hospitals with the lowest baseline penetration to the study arms (unlearning only [n=16] or unlearning 
+ substitution [n=16]). Cluster randomization is well-suited to strategies applied to groups, as it 
avoids the potential for contamination that would occur with randomization at the unit or clinician 
level.66 We will use covariate-constrained randomization63 methods to achieve optimal balance 
between arms for 3 important hospital characteristics: hospital type (2 groups: freestanding children’s 
hospitals vs. general or community hospitals), presence of pre-existing EHR CDS for bronchiolitis 
that promotes use of intermittent “spot checks” instead of continuous pulse oximetry in patients not 
requiring supplemental oxygen, and baseline penetration in P1. Assessment of pre-existing EHR 
CDS will be conducted in P1 and assessed using screenshots of the CDS from each institution and 
brief descriptions of the functionality from the site teams. Randomization will not be blinded, and will 
be performed by the Analytic Core. The combination of balanced allocation of hospitals in covariate-
constrained randomization and additional patient-level covariate adjustment in the analysis (see 
below) will limit bias and increase statistical power. There is no blinding in the trial. 

 
 
 
6 
 
 3.3 
Study Duration, Enrollment and Number of Sites 
3.3.1 
Duration of Subject Study Participation 
This is a cluster-randomized clinical trial with interventions assigned at the hospital level. There are 
several categories of subjects: 
• 
Hospital staff who are exposed to the interventions, complete study questionnaires, and/or 
participate in qualitative interviews 
o 
Maximum anticipated duration: Trial duration (4 years) 
 
• 
Bronchiolitis patients who are directly observed  
o 
Maximum anticipated duration: their length of hospital stay 
 
• 
Parents or guardians of bronchiolitis patients who participate in qualitative interviews 
o 
Maximum anticipated duration: 1 day 
 
• 
Clinical Decision Support Coaches who participate in qualitative interviews 
o 
Maximum anticipated duration: 1 day 
 
3.3.2 
Total Number of Study Sites/Total Number of Subjects Projected 
The study will be conducted at approximately 45 investigative sites in the United States and Canada. 
We will initially seek IRB approval to enroll up to 55 sites in order to allow for attrition during the IRB 
reliance, data use agreement, and study startup processes based on our prior experience with these 
processes with smaller hospitals in the PRIS Network. 
Total number of subjects are provided in Section 3.4 broken down by specific subpopulations. 
3.4 
Study Populations with Inclusion and Exclusion Criteria 
3.4.1 
Population 1: Hospital Staff 
Population 1 encompasses all hospital staff who are exposed to the interventions by being present 
on participating hospital units or overseeing bronchiolitis care during active deimplementation, 
sustainability, and exploratory phases. These hospital staff members are considered third parties to 
the research. They are not considered research subjects unless they participate in study 
questionnaires or qualitative interviews (and thereby fall into hospital staff research subject 
subpopulations 1a or 1b, outlined below). 
3.4.1.1 
Population 1a 
Hospital staff who complete study questionnaires. We will administer questionnaires to hospital staff 
in approximately 32-38 hospitals at 3 different time points (post-randomization, during P2, and during 
P3) to measure acceptability, potential mediators, and potential moderators. We estimate based on 
our pilot study distributing the questionnaires to 200 staff per hospital at each time point and 
achieving a maximum 80% response rate (we achieved a 71% response rate in our pilot).  Therefore 
maximum anticipated enrollment is 200*3*32*0.8= 15,360. 
3.4.1.1.1 Eligible nurses and physicians will meet the following inclusion criteria: 
• 
Employed full-time by the hospital, affiliated practice, or affiliated university in a role that 
involves the care of children with bronchiolitis hospitalized on non-intensive care 
inpatient units 
• 
Fluent in English 
3.4.1.1.2 Eligible administrators will meet the following inclusion criteria: 

 
 
 
7 
 
 
• 
Oversee the care of bronchiolitis on a local level (e.g. nurse manager) or a hospital level 
(e.g. Chief Quality and Safety Officer)  
• 
Fluent in English 
3.4.1.1.3 Nurses, physicians, and administrators will be excluded if they meet any of the 
following exclusion criteria: 
• 
No exclusion criteria 
 
3.4.1.2 
Population 1b 
Hospital staff who participate in qualitative interviews In Aim 2, we will conduct semi-structured 
interviews with physicians and nurses who provide care to bronchiolitis patients in participating units 
with a maximum anticipated enrollment of 64. 
3.4.1.2.1 Eligible nurses and physicians will meet the following inclusion criteria: 
• 
Directly or indirectly involved in the care of bronchiolitis patients or EMO Trial operations 
at the site of interest during the trial period  
• 
Fluent in English 
3.4.1.2.2 Nurses and physicians will be excluded if they meet any of the following exclusion 
criteria: 
• 
No exclusion criteria 
 
3.4.2 
Population 2a 
Bronchiolitis patients directly observed while not receiving supplemental oxygen (“in room air,” for 
primary trial outcome). Since the goal of the deimplementation strategies is to align practice with 
existing national guidelines, we aim to improve the care of children age 2-23 months old with 
bronchiolitis who are in room air, have not received supplemental oxygen in the preceding hour, and 
do not have history of apnea or cyanosis during the current illness. 
We calculate maximum anticipated enrollment as follows based on our target of 90 
patients/participating hospital/phase plus a 10% margin: 
Phase 1: 90*45 = 4050 
Phase 2: 90*32 = 2880 
Phase 3: 90*32 = 2880 
Phase 4: 90*16 = 1260* 
*despite increasing the sites to 16 from 14, we are intentionally not changing the previously 
calculated 1260 maximum anticipated enrollment, as we do not expect to exceed this number.  
Total maximum anticipated enrollment = 11,070*1.10= 12,177 
3.4.2.1 
Eligible patients will meet the following inclusion criteria: 
• 
Infants and children 2 months through 23 months old 
• 
Hospitalized on non-ICU, non-emergency department, non-step down units participating 
in the trial 
• 
Cared for by generalist inpatient services (e.g. general pediatrics, hospital medicine) 
• 
Primary diagnosis of bronchiolitis in most recent physician progress note 
• 
Not actively receiving supplemental oxygen (“in room air”) 

 
 
 
8 
 
 
• 
Last documented receipt of supplemental oxygen >1 hour prior to direct observational 
data collection  
3.4.2.2 
Patients will be excluded if they meet any of the following exclusion criteria:  
• 
Documented apnea or cyanosis during the current illness 
• 
Extreme prematurity (<28 weeks completed gestation) 
• 
Cardiac disease 
• 
Pulmonary hypertension 
• 
Chronic lung disease 
• 
Home oxygen requirement 
• 
Neuromuscular disease  
• 
Immunodeficiency 
• 
Cancer 
• 
Covid-19 / SARS-CoV-2-related illness (known or suspected, including multisystem 
inflammatory syndrome in children [MIS-C]) 
 
3.4.3 
Population 2b 
Bronchiolitis patients directly observed while receiving supplemental oxygen (for underuse 
evaluation). In Aim 3, we are seeking to identify potential unintended consequences of 
deimplementation strategies aiming to reduce continuous SpO2 monitoring in bronchiolitis patients 
who are in room air. One potential unintended consequence is underuse of monitoring in higher risk 
patients. In Aim 3, we define underuse as failing to continuously monitor bronchiolitis patients 
receiving ≥2L/min supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory 
support device (a marker of more severe disease). We will measure it using the same in-person 
bedside observational data collection methods as in Aim 1, but use them in this distinct population of 
patients receiving oxygen. 
We calculate maximum anticipated enrollment as follows based on 35 patients/participating 
hospital/phase plus a 10% additional margin: 
Phase 1: 35*45 = 1575 
Phase 2: 35*32 = 1120 
Phase 3: 35*32 = 1120 
Phase 4: 35*16 = 490 
*despite increasing the sites to 16 from 14, we are intentionally not changing the previously 
calculated 490 maximum anticipated enrollment, as we do not expect to exceed this number.  
 
Total maximum anticipated enrollment = 4305*1.10 = 4736 
3.4.3.1 
Eligible patients will meet the following inclusion criteria: 
• 
Infants and children 2 months through 23 months old 
• 
Hospitalized on non-ICU, non-emergency department, non-step down units participating 
in the trial 
• 
Cared for by generalist inpatient services (e.g. general pediatrics, hospital medicine) 
• 
Primary diagnosis of bronchiolitis in most recent physician progress note 
• 
Actively receiving ≥2L/min supplemental oxygen or 21% room air flow 
3.4.3.2 
Patients will be excluded if they meet any of the following exclusion criteria: 
• 
Extreme prematurity (<28 weeks completed gestation) 

 
 
 
9 
 
 
• 
Cardiac disease 
• 
Pulmonary hypertension 
• 
Chronic lung disease 
• 
Home oxygen requirement 
• 
Neuromuscular disease  
• 
Immunodeficiency 
• 
Cancer 
• 
Covid-19 / SARS-CoV-2 (known or suspected) 
 
3.4.4 
Population 3 
Parents or guardians of bronchiolitis patients who participate in qualitative interviews. In Aim 2, we 
will conduct semi-structured interviews with parents or guardians of children hospitalized with 
bronchiolitis who were found to be in room air during Aim 1 data collection. Maximum anticipated 
enrollment 20. 
3.4.4.1 
Eligible parents or guardians will meet the following inclusion criteria: 
• 
Their child was hospitalized for bronchiolitis on a unit participating in the trial during the 
most recent bronchiolitis season 
• 
Their child was found to be in room air during Aim 1 data collection 
• 
Fluent in English 
3.4.4.2 
Parents or guardians will be excluded if they meet any of the following exclusion 
criteria: 
• 
They are an employee of the hospital or a hospital volunteer 
3.4.5 
Population 4 
Clinical Decision Support (CDS)Coaches who participate in qualitative interviews. In Phase 4 we will 
conduct semi-structured interviews with CDS Coaches. Maximum anticipated enrollment: 10 
3.4.5.1 
Eligible CDS Coaches will meet the following inclusion criteria: 
• 
They were a CDS Coach for one or more sites during Phase 2 and/or Phase 3.  
• 
Fluent in English 
3.4.5.2 
Eligible CDS Coaches will be excluded if they meet any of the following exclusion 
criteria: 
• 
No exclusion criteria 
 
4 
STUDY PROCEDURES 
4.1 
Description of interventions (deimplementation strategies) 
4.1.1 
Unlearning (both study arms) 
4.1.1.1 
Educational outreach  
Educational outreach will focus on communicating core messages to staff including the national 
guidelines, the evidence and rationale underlying the guidelines, and talking points to use if parents 
ask about monitoring, using language adapted from a successful parent-focused intervention.68 The 
educational materials used in our pilot trial will be reviewed and updated with input from the Advisory 
Committee and Family Advisors, to ensure the parent perspective is represented and to navigate any 
issues related to family-facing communications. Outreach to clinicians will include several formats: 
(1) in-person educational outreach sessions delivered in site-specific forums (e.g., staff meetings, 
safety huddles69,70) and led by the Site PI or designee and/or a nurse partner (depending on 

 
 
 
10 
 
 audience). Sessions will occur weekly during day and night shifts in the month prior to the start of 
active deimplementation, with refresher sessions monthly for the remainder of P2. (2) Handouts and 
posters locally adapted with each hospital’s institutional logos and design standards, posted in key 
locations (e.g. unit break room). (3) Short educational videos and messaging distributed using an 
email marketing platform (e.g., Constant Contact71) that enables measurement of open rates, clicks, 
and message engagement. All educational messaging will be designed to minimize the likelihood of 
psychological reactance by minimizing choice-restricting language, using a collaborative tone, and 
emphasizing justification for recommendations.60 All messaging will be field tested with clinicians in 
virtual focus groups prior to launch.  
4.1.1.2 
Audit & Feedback 
Audit & feedback will match our successful pilot study methods55 and will include two levels: (1) 
weekly unit-level feedback and (2) real-time 1:1 feedback to clinicians. Weekly unit-level feedback 
will be based on findings from direct observation of bronchiolitis patients in room air, as described in 
Aim 1 methods. Each week, the Data Coordinating Center (DCC) will compute the prior week’s 
continuous SpO2 overuse rate and distribute these data to sites in the form of a pdf visual dashboard 
that includes comparisons over time and between hospitals. The dashboard will then be shared 
locally with clinicians in person (through educational outreach channels) and via email. In contrast, 
real-time 1:1 feedback will occur at the clinician level. When collecting data on an individual patient, 
data collectors encountering guideline-discordant continuous monitoring are empowered to briefly 
ask any available clinician responsible for that patient’s care, in a nonjudgmental way, about 
indications for monitoring that patient. The clinician is ultimately responsible for deciding if any 
changes are indicated. In our 6-hospital pilot,55 real-time feedback was provided for 73% of eligible 
patients, demonstrating feasibility.  
4.1.2 
Substitution (experimental arm only) 
The substitution strategy includes a clinical pathway integrated into the EHR.  
4.1.2.1 
Clinical pathway 
Clinical pathways guide clinicians step-by-step through evidence-based care.72 Guidelines state that 
continuous SpO2 monitoring should be used in bronchiolitis only if the patient is actively receiving 
supplemental oxygen or previously experienced apnea or cyanosis.14 For patients continuously 
monitored due to a supplemental oxygen requirement, guidelines recommend transitioning to 
intermittent SpO2 measurement within 1 hour of achieving saturations ≥90% in room air.14 During 
Year 1 of the trial, clinician stakeholders, Family Advisors, and our Advisory Committee will 
participate in a virtual guideline-to-pathway translation exercise led by Co-I Schondelmeyer, who led 
the Delphi process resulting in the BEEP guidelines14 and has prior experience translating guidelines 
to pathways.49,73–75 Based on the existing guidelines, the new Eliminating Monitor Overuse (EMO) 
pathway will clearly specify (a) situations when it is appropriate to initiate intermittent SpO2 
measurement (the alternative practice) instead of continuous SpO2 monitoring, and (b) when it is 
appropriate to discontinue continuous SpO2 monitoring and transition to intermittent SpO2 
measurement. We will make the pathway available via web link exclusively to hospitals randomized 
to the unlearning + substitution arm.   
4.1.2.2 
EHR integration 
Table 2. Guiding principles for pathway EHR integration. 
Right information  
When to initiate intermittent measurement instead of continuous monitoring 
When to transition from continuous monitoring to intermittent measurement 
after supplemental oxygen is discontinued 
Right people 
Physicians who order monitoring 
Nurses who monitor patients for clinical changes and place or remove 
monitoring equipment 
Right channels 
EHR and Clinical Pathways Program webpage (via web link exclusively to 

 
 
 
11 
 
 
unlearning + substitution arm hospitals) 
Right points in 
workflow 
Hospital admission, supplemental oxygen management, rounds 
Right format(s) 
may include, but 
not be limited to, 
any combination 
of: 
• EHR-embedded link to pathway website presented to clinicians at 
appropriate points in workflow (minimum standard) 
• Order set for bronchiolitis monitoring that guides physicians to appropriately 
order (a) guideline-concordant monitoring initiation and (b) transition to 
intermittent measurement, and clearly communicates instructions to staff 
• Clinical reminder / alert that notifies nurses that continuous monitoring may 
no longer be indicated based on a documented discontinuation of 
supplemental oxygen 
We have developed guiding principles that will facilitate a standard approach to EHR integration 
while also allowing flexibility in format (Table 2). This pragmatic approach is directly aligned with 
recommendations for research involving flexible EHR-based strategies.76 Flexibility is essential to the 
success of this strategy because EHR software, configuration, resources, and context vary between 
hospitals; it would be unrealistic to expect all hospitals to integrate pathway recommendations in 
exactly the same way.76 Instead of standardizing the software itself, we will standardize the process 
and function of the intervention (an approach promoted by the UK Medical Research Council).77,78  
Since integrating pathways into the EHR is a form of clinical decision support (CDS), we have 
standardized the process of EHR integration around the “Five Rights of CDS,” aiming to ensure 
delivery of (1) the right information, (2) to the right people, (3) in the right intervention format, (4) 
through the right channels, (5) at the right point in workflow.79,80 We have set a very feasible 
minimum standard for pathway EHR integration that requires minimal effort to ensure that all 
hospitals have at least the most basic form of CDS in place at the start of the active 
deimplementation period (P2): an EHR-embedded link to the clinical pathway website inserted into 
the EHR at appropriate points in workflow (e.g. within monitoring order forms and nursing vital sign 
flowsheets). For sites that have some form of existing CDS providing the “right information,” the goal 
of the EHR-integration will be to optimize the EHR CDS to better fit the local context and maximize 
the capability of the CDS based on the EHR and local infrastructure.  
In order to match local capabilities to EHR CDS formats, we have partnered with the national 
Pediatric CDS Collaborative (PCC). Following randomization, each site assigned to the unlearning + 
Table 3. Measures and timing of measurement throughout the trial. Details of each measure in 
Section 5. 
Measure 
Aim 
P1 
P2 
P3 
P4 
Use of continuous pulse oximetry 
(to calculate penetration, sustainability, and underuse) 
1,3 
X 
X 
X 
X 
Acceptability 
1 
  
X 
  
  
Routinization & institutionalization 
2 
 
X 
X 
 
Implementation climate 
2 
 
X 
 
 
Implementation leadership 
2 
 
X 
 
 
Psychological reactance 
2 
 
X 
 
 
Fidelity  
1 
  
X 
  
  
Cost 
1 
  
X 
X 
  
Qualitative inquiry (staff & parents) 
2 
  
  
X 
X 
Length of hospital stay 
3 
X 
X 
X 
  
Oxygen supplementation duration  
3 
X 
X 
X 
  
Safety monitoring 
N/A 
X 
X 
X 
X 

 
 
 
12 
 
 substitution arm will be matched with an EHR integration “coach” from PCC who has experience 
implementing CDS across institutions. Each coach will facilitate integration of the clinical pathway 
into the local EHR by liaising directly with the Site PI, local clinicians, and informatics staff to ensure 
that CDS meeting the criteria in Table 3 is built on time and in a way that matches local capabilities. 
Because EHR interventions should be optimized over time to meet the evolving needs of clinicians, 
coaches will continue meeting with the site team members through the end of P3 to assist with 
optimizations and software updates, or to align with new bronchiolitis guidelines (should they 
emerge). The flexibility and ongoing optimization of the EHR-integrated pathway is a pragmatic 
strength of this proposal entirely consistent with the foundations of implementation science and 
clinical informatics, and one that will result in locally-developed EHR integration that matches local 
workflows.76 
 
4.2 
Procedures within each study phase (see also Table 3) 
4.2.1 
Baseline Phase (P1) 
• 
Measure baseline use of continuous SpO2 monitoring in patients with bronchiolitis who are in 
room air using direct observation 
• 
Medical record review 
• 
Measure length of hospital stay in bronchiolitis patients 
• 
Measure oxygen supplementation duration in bronchiolitis patients 
• 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are receiving ≥2L/min 
supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory support 
device (are inappropriately unmonitored).  
• 
Measure baseline EHR CDS alignment with the 5 rights using same approach used to assess 
fidelity in P2 
• 
Cluster-randomize hospitals with the lowest baseline penetration to either: the unlearning only 
arm (educational outreach with A&F) or the unlearning + substitution arm (adding an EHR-
integrated clinical pathway to facilitate the transition from continuous SpO2 monitoring to 
intermittent SpO2 measurement when clinically appropriate).  
4.2.2 
Active Deimplementation Phase (P2) 
• 
Deimplementation strategies (interventions) are deployed 
o 
unlearning only arm: educational outreach with A&F 
o 
unlearning + substitution arm: educational outreach with A&F + an EHR-integrated 
clinical pathway to facilitate the transition from continuous SpO2 monitoring to 
intermittent SpO2 measurement when clinically appropriate 
• 
Measure use of continuous SpO2 monitoring in patients with bronchiolitis who are in room air 
using direct observation 
• 
Medical record review 
• 
Measure acceptability using questionnaire 
• 
Measure routinization & institutionalization using questionnaire 
• 
Measure implementation climate and leadership using questionnaire 
• 
Measure psychological reactance using questionnaire 
• 
Measure fidelity to each deimplementation strategy 
• 
Assess costs associated with delivering each deimplementation strategy 
• 
Measure length of hospital stay in bronchiolitis patients 
• 
Measure oxygen supplementation duration in bronchiolitis patients 
• 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are receiving 
≥2L/min supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory 
support device (are inappropriately unmonitored).  

 
 
 
13 
 
 
• 
At the end of P2, unlearning (educational outreach with A&F) interventions are withdrawn 
from both arms.  
4.2.3 
Sustainability Phase (P3) 
• 
Substitution (EHR-integrated pathway) is maintained exclusively in the unlearning + 
substitution arm 
• 
Measure use of continuous SpO2 monitoring in patients with bronchiolitis who are in room air 
using direct observation 
• 
Medical record review 
• 
Measure routinization & institutionalization using questionnaire 
• 
Assess costs associated with delivering each deimplementation strategy 
• 
Qualitative interviews aiming to better understand mechanisms of practice change in 
response to each deimplementation strategy  
• 
Qualitative interviews of parents/guardians of children hospitalized with bronchiolitis to 
explore their perceptions of, and reactions to, deimplementation activities 
• 
Measure length of hospital stay in bronchiolitis patients 
• 
Measure oxygen supplementation duration in bronchiolitis patients 
• 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are receiving 
≥2L/min supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory 
support device (are inappropriately unmonitored).  
• 
Calculate primary outcome of sustainability of guideline-concordant deimplementation 
(calculated based on use of continuous SpO2 monitoring in patients with bronchiolitis who 
are in room air over time)  
• 
Validate suctioning data by communicating with a clinical staff member to confirm data 
entries in the EHR are correct 
4.2.4 
Exploratory Phase (P4) 
• 
Identify the hospitals that meet criteria for participation in the exploratory data collection 
phase 
o 
We will create four groups consisting of four hospitals each based on the following 
characteristics: (1) Freestanding/Non-freestanding Children’s Hospitals and (2) pre-
existing CDS presence/no pre-existing CDS presence  
§ 
If there are ≤4 hospitals available in that group, invite all of them, 
§ 
If there are more than 4, we would first filter out hospitals that did not meet 
the criteria of statistical significant drop in overuse between P1 and P2 
(Experienced a significant increase in penetration of guideline-concordant 
care between P1 and P2) 
§ 
If still have >4 remaining, we would filter out hospitals that did not meet the 
90% penetration sustainment margin (Maintained the increased penetration 
at ≥90% of the P2 percentage [an established margin37] through P3) 
§ 
If still have >4 remaining, we would sort the hospitals by the numeric gap 
between the hospital's measured P3 penetration and the P2 90% margin 
and select the 4 with the smallest gap (best sustainment) 
• 
Measure use of continuous SpO2 monitoring in patients with bronchiolitis who are in room air 
using direct observation 
• 
Medical record review 
• 
Qualitative interviews aiming to better understand mechanisms of practice change in 
response to each deimplementation strategy  
• 
Qualitative interviews of parents/guardians of children hospitalized with bronchiolitis to 
explore their perceptions of, and reactions to, deimplementation activities 
• 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are receiving 
≥2L/min supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory 
support device (are inappropriately unmonitored).  
 

 
 
 
14 
 
 5 
STUDY EVALUATIONS AND MEASUREMENTS 
5.1 
Measure use of continuous SpO2 monitoring in patients with bronchiolitis who are in 
room air using direct observation  
We will perform direct observation of continuous SpO2 monitoring in order to measure its use, as we 
have done successfully in prior studies.24,55 As in the prior studies, we will simultaneously assess 
cardiorespiratory (electrocardiographic) monitoring. We must directly observe because our prior 
research has shown that analyzing orders for monitoring does not accurately capture actual 
monitoring status.81 Research staff at each hospital will perform cross-sectional observational data 
collection rounds during the study periods. During these data collection rounds, trained research staff 
walk to each hospital unit with eligible children. On the unit, staff determine the continuous 
monitoring status of each patient based on either (a) going to the bedside and visually examining the 
waveforms displayed on the monitor, or (b) going to a central monitoring station that displays the 
waveforms of multiple patients on the unit and visually examining the waveforms displayed on the 
monitor for the eligible child. Repeated over time, this method captures repeated cross-sections of 
penetration of guideline-concordant care, which is used to subsequently calculate the primary 
outcome of sustainability (see Statistical Considerations). Repeated measures in the same patient 
are permitted as long as they are separated by at least 6 hours. 
We have previously shown that, in hospitals that use the Epic EHR and have physiologic monitor 
integration with the EHR, examining the EHR for the presence of monitor data is a valid method for 
assessing the bedside monitoring status. The following approaches are permissible alternatives to 
direct in-person observation to determine continuous monitoring status: 
• 
In hospitals with physiologic monitor device integration with the EHR, visualizing device-
integrated monitor data in EHR flowsheets, reports, or real-time waveform displays. 
• 
In hospitals with clinical remote monitor viewing applications approved for clinical use at 
their institution (e.g. Airstrip, Philips, GE, Sickbay, Etiometry), visualizing remote monitor 
data in flowsheets, reports, or real-time waveform displays. 
 
During the observational rounds, as in our prior studies, we will also assess/confirm the 
supplemental oxygen / flow status (current device, flowrate, %FiO2) since the hospital chart may not 
always be perfectly accurate. This assessment may be performed using any of the following 
methods: 
• 
Visual assessment at the bedside 
• 
Asking a clinician caring for or overseeing the care of the patient (e.g. nurse, charge nurse, 
physician, respiratory therapist, medical assistant) either in-person, by telephone, or through 
a hospital-approved clinical communication / messaging platform (e.g. Epic SecureChat, 
TigerText, Voalte, Ascom). 
5.1.1 
Medical record review 
After direct observational data collection rounds, staff perform chart review for demographics and 
other covariates used in previous research: 
• 
Local hospital medical record number (required for tracking across admissions) 
• 
Local hospital patient admission /account number (required for tracking within admissions) 
• 
Date of birth (required for accurately tracking and validating age in days over time) 

 
 
 
15 
 
 
• 
Patient initials (as a secondary means of verifying the patient’s identity in the databases) 
• 
Hospital 
• 
Admission date and time 
• 
Discharge date and time 
• 
Unit name 
• 
Bed number 
• 
Patient age 
• 
Patient sex 
• 
Patient race and ethnicity 
• 
Patient primary diagnosis and chronic conditions 
• 
Forms of chronic neurologic impairment (e.g. static encephalopathy, cerebral palsy, 
hydrocephalus, spina bifida, epilepsy/seizure disorder, hypotonia, other form of neurologic 
impairment) 
• 
Technology dependence (e.g. short-term feeding tube, long-term feeding tube, central 
venous line, renal dialysis) 
• 
History of ICU stay earlier in this admission, with dates and times of ICU stays 
• 
Patient gestational age 
• 
Characteristics of supplemental oxygen administration (flow rate, fraction of inspired oxygen, 
oxygen delivery device, maximum support during hospitalization), also including dates of 
initiation, adjustment, discontinuation, last documented time on supplemental oxygen. 
• 
Admission weight in Kilograms 
• 
High Flow Nasal Canula (HFNC) use: 
• 
Characteristics of patient at time of initiation: time/date, initial flow rate, initial FiO2, 
SpO2, Respiratory Rate (RR)  
• 
Characteristics of patient prior to initiation: Mode of oxygen delivery, time/date SpO2 
last documented and SpO2 rate, time/date RR last documented and RR 
• 
Respiratory or general exam flowsheet: time/date of flowsheet, characteristics 
observed (head bobbing, grunting, altered mental status), and any free text notes 
• 
Physical exam clinical note (e.g., event note, progress note, RRT documentation or 
nurse/RT flowsheet assessment): time/date of note, general exam and respiratory 
exam findings, and any free text notes 

 
 
 
16 
 
 
• 
Blood gas: time/date of sample, pH, CO2, source 
(venous/arterial/capillary/unknown) 
• 
Highest documented HFNC flow rate and highest documented FiO2 
• 
Last documented time/date of HFNC and/or time/date of transition to LFNC or room 
air 
• 
Final flow rate, last documented FiO2 21%, time/date of change to FiO2 21% 
• 
Laboratory testing and imaging studies that may indicate other forms of medical overuse 
(e.g. respiratory viral testing, chest x-rays), including results of those studies. 
• 
Medication administration that may indicate other forms of medical overuse (e.g. 
bronchodilators). 
• 
Characteristics of enteral, intravenous, subcutaneous, and other forms of fluid administration 
• 
Limited English Proficiency status of family, as evidenced by chart review suggesting that 
there is a language barrier, limited English proficiency, interpreter needed, interpreter 
previously used to communicate with this family, that the preferred language of patient/family 
is something other than English, or other similar documentation. 
• 
Role composition / staffing model of the team caring for the patient, for example, the team 
was comprised of attending physicians supervising nurse practitioners, vs. the team was 
comprised of an attending physician supervising residents and medical students; the 
nurse:patient ratio was 1:3, the respiratory therapist:patient ratio was 1:8. 
• 
Unit characteristics including unit census, cohorting of bronchiolitis patients in specific 
respiratory care areas, and the presence of oxygen weaning protocols. 
• 
How long have they been off respiratory support (e.g. supplemental oxygen or room air 
flow), according to the chart? 
• 
Are any pulse oximetry measurements documented in the chart since the patient has been 
off respiratory support (e.g. supplemental oxygen or room air flow) as an indication that 
intermittent SpO2 measurement is taking place? 
• 
Primary site where patient receives primary care  
• 
Post-hospitalization Primary Care follow-up recommendations and subsequent appointment 
information  
• 
Characteristics of instances of respiratory suctioning, including the date/time of suctioning, 
the route of suctioning (e.g. nasal), the role of the person performing the suctioning (e.g. 
respiratory therapist), and any comments about clinical sequelae of the suctioning 
documented in the EHR (e.g. "improved" or "coughing") 
 

 
 
 
17 
 
 5.2 
Questionnaires 
Site PIs will prepare lists of the nurses and physicians who provide care for bronchiolitis patients on 
the units participating in the study, as well as administrators who oversee the medical care and 
safety for bronchiolitis patients at institutional and unit levels. Questionnaires will be distributed 
electronically and managed centrally from the DCC using REDCap. 
For hospitals that elect not to provide the DCC with staff contact information for questionnaire 
distribution, the site PIs or their designees will distribute a survey link to the relevant stakeholder 
groups or administer the questionnaire (without any PHI) on paper for subsequent entry into 
REDCap. 
Questionnaire items are included in the eIRB appendix. We will submit amendments to the IRB for 
any changes to the questionnaire prior to distribution. No questionnaire items will involve PHI. 
5.2.1 
Measure acceptability using questionnaire 
Acceptability among nurses and physicians will be measured using items adapted from the 4-item 
Acceptability of Intervention Measure (AIM)56 previously used in our pilot study. We have also 
included items from our pilot study questionnaire covering feasibility, appropriateness, and perceived 
safety. See appendix in eIRB. 
5.2.2 
Measure routinization & institutionalization using questionnaire 
We will use an adapted version of the valid and reliable Slaghuis measurement instrument for 
sustainability of work practices to assess the potential mediators: routinization & institutionalization.57 
The instrument assesses two closely related but conceptually distinct processes: routinization, in 
which clinicians develop new routines such that the practice change becomes part of their everyday 
work, and institutionalization, in which the organization embeds the practice change into its existing 
systems and structures. See appendix file for draft questionnaire items. 
5.2.3 
Measure implementation climate and leadership using questionnaire 
We will administer the Implementation Climate Scale to understand whether the environment 
expects, supports, and rewards deimplementation of continuous SpO2 monitoring.58 We will measure 
implementation leadership using the Implementation Leadership Scale to understand leader 
behaviors (e.g., proactive, knowledgeable, and supportive of deimplementation).59  
5.2.4 
Measure psychological reactance using questionnaire 
We will measure psychological reactance in clinicians using the same multiple choice instruments 
used in seminal reactance work60 adapted as necessary to this deimplementation trial to assess (1) 
perceptions of threats to freedom in response to deimplementation messaging, (2) emotional 
responses, and (3) cognitive responses.60,82,83  
5.3 
Measure fidelity to each deimplementation strategy 
Deimplementation fidelity is the extent to which educational outreach, A&F, and the EHR-integrated 
clinical pathways are performed per protocol. Fidelity data for educational outreach and A&F will be 
extracted from intervention logs maintained by Site PIs and entered into a data collection form. 
Fidelity data for the EHR integration will be assessed (at baseline as a benchmark of pre-existing 
EHR CDS and then later in P2 as a fidelity measure) using local EHR screenshots and descriptions 
uploaded by each site to assess alignment of the actual EHR interface with each informatics guiding 
principle in Table 2. Each principle will be scored independently by 2 clinical informaticists as being 
fully met, partially met, or not met for each hospital. 

 
 
 
18 
 
 5.4 
Assess costs associated with delivering each deimplementation strategy 
Cost of delivering each of the strategies will be assessed using a Time-Driven Activity Based Costing 
method,84 based on who (personnel completing the task) does what (specific activities performed), 
when (timing), and how often (the frequency, intensity, and/or duration of the activity). First, we will 
create a step-by-step process map that carefully specifies activities within the unlearning and 
substitution strategies. We will specify the names of specific strategies (e.g., conducting A&F 
sessions), actions (specific procedures that need to be performed, e.g., holding weekly unit-based in-
person group discussion) and actors (who is involved in the procedures, e.g., EHR coach, clinician) 
in each strategy. Second, we will determine frequency and duration of each of these actions. Once 
we determine how many times an actor participated in a given action and the duration of the action, 
multiplying frequency and duration will yield total time spent by the actor on a given action. 
Multiplying this by the actor’s hourly wage rate will yield total cost incurred by that actor to perform 
that specific action. The total cost of an action is the sum of costs incurred by each actor who 
participated in that action. In this way, we will determine cost of each action in each implementation 
strategy. Adding up the cost of each action of a specific implementation strategy will yield the total 
cost of personnel resources for that strategy. For non-personnel resources, we will itemize 
consumable equipment, supplies (training materials, assessment and evaluation materials, office 
supplies, etc.). Cost of each item will be determined from administrative records. Personnel and non-
personnel costs will be added together to determine overall unlearning and substitution strategy 
costs. In this way, we will discretely outline the entirety of the unlearning and substitution process 
and assign resource use and costs to these processes to a high degree of specificity. We expect 
there to be communication-related actions that are not protocolized, such as staff responding to 
emails, making phone calls and holding in-person meetings to provide assistance as needs arise. To 
estimate frequency and duration of these type of actions, we will choose a defined period during 
active implementation and ask staff to save their emails and record their scheduled phone calls and 
in-person meetings on their calendar so we can determine an estimate of the frequency of 
communication. We will also ask them to provide an estimate of the average duration of these 
communications.  
5.5 
Clinical outcomes 
We will measure the primary clinical outcome of LOS and the secondary clinical outcome of oxygen 
supplementation duration in the cohort of patients whose data were collected in Aim 1, using chart 
review performed at each site after hospital discharge. 
5.5.1 
Measure length of hospital stay (LOS) in bronchiolitis patients 
LOS is defined as the time interval in minutes from admission to an inpatient or observation hospital 
unit until discharge from an inpatient or observation hospital unit. 
5.5.2 
Measure oxygen supplementation duration in bronchiolitis patients 
Total oxygen supplementation duration is defined as the total minutes during the entire admission 
that the patient received any supplemental oxygen or supplemental flow of room air via any 
respiratory support device (e.g. high flow nasal cannula, ventilator). 
5.6 
Qualitative interviews aiming to better understand mechanisms of practice change in 
response to each deimplementation strategy  
Our qualitative inquiry aims to better understand mechanisms of practice change in response to each 
deimplementation strategy. Using a deviance approach,85,86 we will conduct semi-structured 
interviews with nurses and physicians who provide care to bronchiolitis patients at the end of P4 (the 
exploratory study phase that examines sustainability 2 years after withdrawal of educational outreach 
and A&F) as well as with CDS Coaches. In collaboration with site PIs, eligible clinicians will be 
identified purposively from staff rosters and invited to participate in interviews by telephone call, 
texting, and/or emailing to discuss the process of deimplementation as experienced by stakeholders. 
Purposive sampling will seek to obtain a wide range of nurse and physician experience levels, nurse 

 
 
 
19 
 
 and physician supervisory vs front line care roles, nurse shift (day vs night shift), and physician care 
model (teaching service vs attending-only hospitalist model). Interviews will explore mechanistic 
relationships between (a) the strategies, (b) the findings of the quantitative measures above, and (c) 
sustainability. Interviews will be conducted by telephone or using teleconferencing software (e.g. 
Zoom, BlueJeans, Teams). Interviews will be recorded. The participant will be permitted to choose 
the format (telephone vs teleconferencing software with audio-only vs teleconferencing software with 
audio and video). The informed consent process will include discussion that if the participant elects 
to conduct the interview with video on, the video will also be recorded. 
The staff member’s email address will be collected in order to send a recruitment email. They will be 
provided a copy of their consent form by email or mail. In order to provide the staff member with a 
copy of their consent form, we will also collect their mailing address if mail is their preferred means of 
receiving that document. 
A sample interview guide is attached. As is standard practice in qualitative research, the interview 
guide will be a fluid document that will be modified and updated as we proceed with interviews. We 
will not submit amendments to the IRB for changes to the interview guide unless there are major 
changes in the scope. No interview questions will involve PHI. 
5.7 
Qualitative interviews of parents/guardians of children hospitalized with bronchiolitis to 
explore their perceptions of, and reactions to, deimplementation activities 
We will conduct semi-structured interviews with parents or guardians of children hospitalized with 
bronchiolitis who were found to be in room air during Aim 1 data collection. Eligible parents or 
guardians will be identified purposively from bronchiolitis patients’ trial records during P3 and invited 
to interview by telephone call, texting, and/or emailing during the 8 weeks following discharge in 
order to explore their perceptions of, and reactions to, SpO2 deimplementation activities. Purposive 
sampling will seek to obtain a wide range of patient ages since parents’ perspectives likely differ for 
very young infants (e.g. 2-3 months old) compared to older children (e.g. 18-23 months old). 
Purposive sampling will also be used to optimize diversity among the participants. In order to 
optimize diversity across racial and ethnic groups, we will first recruit 5 patients, then assess the 
racial makeup of the sample, and target for the distribution of all 15 patients to be 1/3 Hispanic Black, 
1/3 Non-Hispanic Black, and 1/3 Other. Interviews will be conducted by telephone or using 
teleconferencing software (e.g. Zoom, BlueJeans, Teams). Interviews will be recorded. The 
participant will be permitted to choose the format (telephone vs teleconferencing software with audio-
only vs teleconferencing software with audio and video). The informed consent process will include 
discussion that if the participant elects to conduct the interview with video on, the video will also be 
recorded. 
Parent or guardian names, email addresses, and telephone numbers will be collected for recruitment 
purposes. They will be provided a copy of their consent form by email or mail. In order to provide the 
parent or guardian with a copy of their consent form, we will collect the parent/guardian’s email 
address and/or mailing address, depending on their preferred means of receiving that document. 
A sample interview guide is attached. As is standard practice in qualitative research, the interview 
guide will be a fluid document that will be modified and updated as we proceed with interviews. We 
will not submit amendments to the IRB for changes to the interview guide unless there are major 
changes in the scope. No interview questions will involve PHI. 
5.8 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are 
receiving ≥2L/min oxygen (are inappropriately unmonitored).  
We will also collect additional data to capture any underuse of monitoring in higher risk patients that 
could plausibly occur in response to deimplementation. We define underuse as failing to 
continuously monitor bronchiolitis patients receiving ≥2L/min supplemental oxygen or ≥2L/min of 
supplemental room air flow via any respiratory support device (a marker of more severe disease)5 
and will measure it using the same in-person bedside observational data collection methods used to 
observe patients in room air during Aim 1 (and in practice, sites will likely perform these 

 
 
 
20 
 
 observational rounds for this distinct population simultaneously with observational rounds for Aim 1). 
See Section 5.1 above for data collection methods. 
5.9 
Monitoring for Equitable Deimplementation 
The DCC will perform monthly surveillance for signals in the data that may suggest hospital- or 
study-level inequities in deimplementation stratified by patient race and ethnicity and generate 
descriptive reports throughout the trial. These reports will be periodically reviewed by the 
investigative team and DSMB.  
 
6 
STATISTICAL CONSIDERATIONS 
6.1 
Primary Endpoint 
The primary endpoint is the sustainability of guideline-concordant deimplementation of continuous 
SpO2 monitoring.  
6.2 
Secondary Endpoints 
Secondary endpoints will include the following: 
• 
The changes in penetration of guideline-concordant care within each study arm 
• 
The acceptability, feasibility, appropriateness, and perceived safety of deimplementation among 
staff, compared between nurses and physicians 
• 
The mediation effects of routinization and institutionalization  
• 
The moderation effects of implementation climate, implementation leadership, and psychological 
reactance  
• 
Fidelity to each deimplementation strategy 
• 
Costs associated with delivering each deimplementation strategy 
• 
The relationship between penetration of guideline-concordant care and length of hospital stay 
(LOS) in bronchiolitis patients  
• 
The relationship between penetration of guideline-concordant care and oxygen supplementation 
duration in bronchiolitis patients  
• 
Themes from qualitative interviews  
• 
Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are receiving ≥2L/min 
supplemental oxygen or ≥2L/min of supplemental room air flow via any respiratory support 
device. 
• 
Exploration of deimplementation inequities 
 
6.3 
Statistical Methods 
6.3.1 
PRIMARY ENDPOINT: Sustainability of guideline-concordant deimplementation of 
continuous SpO2 monitoring 
We will analyze sustainability as a longitudinal difference in differences comparison between study 
arms of the change in penetration between baseline (P1) and the phase after withdrawal of 
educational outreach and A&F (P3), expressed as (P3-P1 | Arm 2) - (P3-P1 | Arm 1). Penetration is 
defined as the percentage of bronchiolitis patients in room air who are receiving guideline-
concordant care: appropriately not being continuously SpO2-monitored. In order to make this 

 
 
 
21 
 
 comparison, we will use the patient-level direct observation data that includes the binary outcome of 
monitored vs. unmonitored. We will use generalized hierarchical mixed-effects models with logit link 
for longitudinal binary outcome data. We will include hospital- and patient-specific random intercepts 
and slopes to account for clustering. We will apply spline-based piecewise regression in the mixed-
effects models to allow for estimation of trajectories over time, with the start of each phase being the 
pre-specified knots. The model will include two time variables, one indicating the week number and 
one indicating the phase (i.e. P1, P2, P3). The interaction between spline-specific slope and 
intervention will be included in the model to assess the difference in differences between study arms. 
To account for differences in patient-level factors that cannot be accounted for by the cluster-
randomization, we will adjust for the same covariates used in previous research, including age, 
gestational age, time since weaning from supplemental oxygen, presence of an enteral feeding tube, 
neurologic impairment, and overnight observation. Statistical contrasts at the hospital level will be 
made using postestimation marginal effects. These methods facilitate estimation of standardized (for 
patient-level characteristics) sustainability and the effect of adding the substitution strategy to the 
unlearning strategy. We will perform a sensitivity analysis considering the hospital unit as an 
additional level of clustering in the hierarchical model.   
6.3.2 
The changes in penetration of guideline-concordant care within each study arm 
Penetration during active deimplementation will be analyzed using a similar model to that described 
above for sustainability, with contrasts of interest within each study arm (not differences in 
differences): P2-P1 | Arm 1; P2-P1 | Arm 2.  
6.3.2.1 
Exploratory analysis of penetration including prior years of EMO 
In winter 2018-19 we conducted the original EMO observational study in 56 sites,24 and in winter 
2019-20 we conducted a pilot study in 6 sites. In this exploratory analysis we will include deidentified 
penetration data available from the prior studies with data from the trial and analyze penetration 
within-hospital over time using longitudinal regression in hospitals participating in the trial to gain 
further insight into longitudinal penetration with and without intercurrent interventions. 
6.3.3 
The acceptability of deimplementation among staff, compared between nurses and 
physicians 
Acceptability (ordinal measure) will be analyzed using ordered logit regression, with comparisons 
between professions (nurse vs. physician). The same approach will be applied to other questionnaire 
items assessing feasibility, appropriateness, and perceived safety (see appendix questions in eIRB). 
6.3.4 
The mediation effects of routinization and institutionalization of the practice change 
on the relationship between the deimplementation strategies and sustainability, 
penetration 
Mediation analysis allows us to separate the direct effects of an exposure from effects that occur via 
an intermediate variable (indirect effects). For each outcome (penetration and sustainability) we will 
perform separate mediation analyses for routinization and institutionalization, each based on 
dimension-specific scores on the adapted Slaghuis questionnaire. Mediation will be tested using the 
product of coefficients approach for multilevel mediation analysis,87–89 which we have used in 
previous studies.90–92 In this approach, the total effect of the deimplementation strategy is parsed into 
direct and indirect effects.  
 

 
 
 
22 
 
 
As shown in Figure 2, “Path a” 
represents the effect of the 
deimplementation strategy on the 
hospital-level mediators. “Path b” 
represents the effect of the hospital-
level mediators on the outcomes. An 
unbiased estimate of the mediated 
effect is derived via the product of 
the “a” and “b” paths (i.e., a*b) from 
these analyses.88,89,92 We will test 
the statistical significance of the 
mediated effects using the joint 
significance test.93 We will use Monte Carlo simulation methods to derive confidence intervals.93–96 
Given recent developments in the causal mediation analysis literature,97 we will estimate the indirect 
effect using both the conventional method described here and using the counterfactually-defined 
causal effects approach for cluster-randomized studies.97,98 Models will be estimated in Mplus which 
permits estimation of both types of effects and includes Monte Carlo simulation facilities for 
conducting sensitivity analyses useful for evaluating the tenability of embedded assumptions.99 
Moderators of the strategies’ effects will be tested separately by adding terms for each moderator 
and its interaction with the deimplementation strategy to the Aim 1 models only during the active 
implementation period.  
6.3.5 
The moderation effects of implementation climate, implementation leadership, and 
psychological reactance among staff on the relationship between the 
deimplementation strategies and sustainability, penetration 
Moderators will be tested by adding terms for each moderator and its interaction with the strategy to 
the Aim 1 regression models for penetration and sustainability (Path c′). 
6.3.6 
Fidelity to each deimplementation strategy 
Deimplementation fidelity (descriptive) to the unlearning strategy will be calculated as the number of 
educational outreach and A&F sessions delivered in each hospital divided by the number expected 
in the protocol during each week and averaged over all weeks of the active deimplementation period. 
Fidelity to the substitution strategy will be assessed as the extent to which the clinical pathway is 
integrated according to the “5 Rights” guiding principles; each principle will be scored as being fully 
met, partially met, or not met and summarized for each hospital.  
6.3.7 
Costs associated with delivering each deimplementation strategy 
Analysis for costs is descriptive and the data resulting from the Time-Driven Activity Based Costing 
method84 described will be summarized for each of the strategies using descriptive statistics. 
6.3.8 
The relationship between penetration of guideline-concordant care and length of 
hospital stay (LOS) in bronchiolitis patients in each study phase 
The hospital-level penetration for each phase (P1, P2, P3) will be the primary exposure variable. 
Penetration will be combined with patient-level data that includes LOS. We will use generalized 
mixed-effects regression to model the LOS (assuming a gamma distribution with a log link function to 
account for the expected skewed nature of LOS data) and use hospital-specific random intercepts to 
account for within-hospital clustering.  We will estimate the change in LOS for each 5% change in 
penetration using postestimation marginal effects.  
6.3.9 
The relationship between penetration of guideline-concordant care and oxygen 
supplementation duration in bronchiolitis patients in each study phase 
The hospital-level penetration for each phase (P1, P2, P3) will be the primary exposure variable. 
Penetration will be combined with patient-level data that includes duration of oxygen 
supplementation. We will use linear mixed-effects regression to model the duration of oxygen 
Figure 2. Mediators and moderators. 

 
 
 
23 
 
 supplementation (assuming a normal distribution) and use hospital-specific random intercepts to 
account for within-hospital clustering.  We will estimate the change in oxygen supplementation for 
each 5% change in penetration using postestimation marginal effects.  
6.3.10 Themes from qualitative interviews  
Interview transcripts will be loaded into NVivo software. Analysis will follow an integrated approach 
using the CFIR as a starting framework while also allowing new concepts to emerge and become 
part of the coding scheme. This approach uses an inductive process of iterative coding to identify 
recurrent themes, categories, and relationships. After initial exploration of the data, a comprehensive 
coding scheme is developed and applied to all data in order to produce a highly granular descriptive 
analysis. Two trained team members will separately double-code a sample of the transcripts to 
assess the reliability of the coding scheme. Disagreements in coding will be resolved through team 
discussion.  
6.3.11 Underuse of continuous SpO2 monitoring in patients with bronchiolitis who are 
receiving ≥2L/min oxygen (are inappropriately unmonitored) within each study arm, 
across all study phases  
We will examine underuse of continuous SpO2 monitoring during P1, P2, and P3 as the percentage 
of patients with bronchiolitis observed receiving ≥2L/min supplemental oxygen or ≥2L/min of 
supplemental room air flow via any respiratory support device who are inappropriately unmonitored. 
We will analyze underuse longitudinally and by study arm using similar patient-level mixed-effects 
logistic regression models as in the primary analysis for Aim 1. 
6.3.12 Exploration of deimplementation inequities 
Deimplementation inequities will be explored by including patient sex, race, and ethnicity in all 
regression models (including potential interactions with other exposure variables).  
6.4 
Sample Size and Power 
Winter 2020-2021 saw very low bronchiolitis volumes due to public health measures to contain the 
SARS-CoV-2 pandemic. If bronchiolitis patient volumes are lower than a typical bronchiolitis season 
during Phase 1 of this trial, we will re-run power calculations after Phase 1 based on that data and 
may amend the protocol based on those revised, updated calculations. 
6.4.1 
Aim 1 Sample Size and Power 
The trial’s overall power analysis is based upon the primary deimplementation outcome 
(sustainability), consistent with hybrid type III. We expect that the pilot study’s pre-post effect size of 
30 percentage points likely overestimates the effect size expected in this trial. Based on NIH 
guidance, we have used data from our pilot trial primarily to assess feasibility and understand 
important design effects (e.g. clustering). Therefore, we have placed greater emphasis on 
incorporating clinically meaningful differences of 15-20 percentage points (%pts) that, based on 
consensus of our research team and experts within PRIS, could reasonably impact future guidelines 
and policy. In this trial, power is primarily driven by the number of hospitals, the variation across 
hospitals, and the within-hospital correlation over time. The degree of correlation can be expressed 
as either the intra-cluster correlation coefficient or the between-cluster coefficient of variation. While 
the two approaches are equally valid, we have used the between-cluster coefficient of variation 
method in our calculations because it is more flexible and is more readily understood. Table 4 below 
outlines detectable effect sizes at 80% power based on 12 hospitals/arm remaining at the end of P3 
(16 randomized/arm minus 4/arm assuming 25% dropout).  
Table 4. Aim 1 primary outcome power calculations. Calculated using PASS 2020, v20.0.1. 
Hypothesis 
Effect size at 80% power for n=12 
hospitals/arm 
Primary (sustainability). When comparing the difference 
in penetration between P3 and P1, Arm 2 will have a 
Power is 80% to detect a difference in 
differences of 16 %pts between P1 and P3 

 
 
 
24 
 
 greater difference than Arm 1, expressed as: (P3-P1 | 
Arm 2) - (P3-P1 | Arm 1)  
among study arms. Using footnote parameter 
ranges, effect size range is 12-20 %pts.  
Secondary (penetration). Each arm will experience 
greater penetration during P2 compared to P1, expressed 
as: P2-P1 | Arm 1, and 
P2-P1 | Arm 2 
Within each arm, power is 81% to detect a 
difference of 12 %pts between P1 and P2. Using 
footnote parameter ranges, effect size range is 
9-16 %pts.  
Arm 1 = unlearning only. Arm 2 = unlearning + substitution. 
Parameters informed by preliminary data: approximately 50% monitored in P1, average 90 
patients/hospital/phase, 2-sided alpha=.05, moderate within-hospital correlation across phases of 0.6 (range 
0.5-0.7), moderate-high between-hospital standard deviation of 15 %pts (range 13-17). 
Feasibility note: In the pilot trial, each hospital collected data on an average of 118 unique patients during 1 
season. 
 
6.4.2 
Aim 2 Sample Size and Power 
6.4.2.1 
Quantitative  
We have performed a conservative power calculation for mediation effects in logistic regression 
(since our primary dataset is patient-level with the binary outcome of monitored vs. unmonitored for 
each patient) using PASS. We assume 12 hospitals/arm, 30 clinician responses/hospital to the 
adapted Slaghuis questionnaire, correlation of 0.3 between the deimplementation strategy arm 
indicator variable and the mediator, and penetration in the unlearning only arm of 55% during P3. 
Using these parameters, we will have 80% power to detect an indirect mediation effect of 1.25, which 
is interpreted as a 1.25-fold increased odds of guideline-concordant care penetration with each 1 SD 
increase in the mediator (the dimension-specific score on the adapted Slaghuis questionnaire). This 
is a conservative estimate because the calculation does not account for the additional power that will 
be gained due to the two measurement time points in each hospital used for the sustainability 
outcome.  
6.4.2.2 
Qualitative 
We estimate that we will need to conduct 48 clinician interviews to achieve thematic saturation with 
12 interviews across each of the 4 possible profession and sustainability subgroups 
(nurse/physician, high/low sustainability). We expect thematic saturation after performing 15 parent 
interviews since there are no subgroup analyses planned. 
6.4.3 
Aim 3 Sample Size and Power 
For the primary clinical outcome of LOS, we performed a power analysis informed by estimates of 
LOS from prior clinical trials and observational studies, based on a median LOS of 36 hours, 
interquartile range 20-52 hours, and a clinically important difference in LOS of 6 hours between 
lowest and highest quartiles of penetration observed in the study. With an estimated 90 patients per 
phase per hospital across 3 phases and a minimum of 24 hospitals, we will have 80% power to 
detect a difference in LOS of 2.4 hours between the lowest and highest quartiles of penetration 
(alpha=.05). The underuse outcome requires in-person data collection in a new population distinct 
from Aim 1. With surveillance of 35 patients/hospital receiving ≥2L/min supplemental oxygen or 
≥2L/min of supplemental room air flow via any respiratory support device observed within each 
phase, we will have 80% power to detect a clinically significant 23 %pt difference in underuse of 
monitoring from 5% at baseline in P1 to 28% in P2 or P3, assuming a within-hospital correlation of 
0.4. Other Aim 3 outcomes are descriptive. 
6.5 
Interim Analysis 
An analysis is required and planned immediately after P1 (prior to any interventions) in order to 
obtain baseline penetration of guideline-concordant care for each participating hospital and confirm 
sites with adequate bronchiolitis volumes, infrastructure, and capability to conduct subsequent 
randomization based on these results. 

 
 
 
25 
 
 We also propose interim analyses following active deimplementation (Phase 2) and sustainability 
(Phase 3) phases in order to provide the DSMB with up-to-date data for these meetings in order to 
assess safety and feasibility. 
In addition, the DSMB will, at their discretion, be able to request additional interim analyses.  
There are no pre-specified stopping rules for efficacy or safety for this project focused on aligning 
practice with the standard of care.  
 
7 
SAFETY MANAGEMENT 
7.1 
Clinical Adverse Events 
Clinical adverse events (AEs), any untoward or unfavorable medical occurrence in a human subject; 
including any abnormal sign, symptom or disease that is temporally related to the research, whether 
or not it is related to the subject's participation in the research, will be monitored throughout the 
study. See details in Section 8 describing the proactive adverse event surveillance plan. 
7.2 
Adverse Event Reporting 
Since the study procedures are not greater than minimal risk and the purpose of the interventions is 
to align practice with existing guidelines and the standard of care, SAEs are not expected. If any 
unanticipated problems related to the research involving risks to subjects or others happen during 
the course of this study (including SAEs) they will be reported to the IRB in accordance with CHOP 
IRB SOP 408: Unanticipated Problems Involving Risks to Subjects. Unanticipated Problems are any 
incidents, experiences, or outcomes that are unexpected (in terms of the nature, severity or 
frequency), related to a subject's participation in the research, and suggest that the research places 
subjects or others at greater risk of harm than was previously known or recognized. Serious Adverse 
Events (SAEs) in the context of this study are AEs that meet any of the following conditions: (1) 
results in death; (2) is life-threatening (places the subject at immediate risk of death from the event 
as it occurred); (3) requires prolongation of existing hospitalization; (4) results in persistent or 
significant disability/incapacity; (5) based upon appropriate medical judgment, may jeopardize the 
subject’s health and may require medical or surgical intervention to prevent one of the other 
outcomes listed in this definition above.  
AEs that do not meet prompt reporting requirements will be summarized in narrative or other format 
and submitted to the IRB at the time of continuing review (if continuing reviews are required), or will 
be tracked and documented internally by the study team but not submitted to the IRB (if continuing 
reviews are not required).  
 
8 
STUDY ADMINISTRATION 
8.1 
Treatment Assignment Methods 
8.1.1 
Randomization 
This cluster-randomized trial uses covariate constrained randomization to randomize at the hospital 
level, accounting for important covariates. See details in Section 3.2. 
8.1.2 
Blinding 
There is no blinding in the trial. 
8.2 
Data Collection and Management 
The Clinical Research Computing Unit (CRCU) at the University of Pennsylvania will serve as the 
DCC.  

 
 
 
26 
 
 8.2.1 
Database 
All research data for this trial will be stored in an electronic database that is managed by the 
Research Technologies Department (RTD) of the CRCU. The database will be hosted on secure 
computing servers and will be restricted to only those individuals who are authorized to work on the 
trial. Individual user accounts with passwords will be used to restrict access to the database. Specific 
privilege assignments within the database will also be employed to limit the types of functions that 
authorized users can perform to those functions that are appropriate for their role in the trial. 
Additional measures to prevent unauthorized external access to the database environment will be 
employed using network firewall technologies. The CRCU team will work closely with the 
investigators and to design, develop, and test a database to support the requirements of the study 
and to promote data security and integrity. Electronic audit trails of changes to database contents will 
be incorporated into the design and will capture and record those changes automatically. In addition 
to the trial database where actual results will be maintained, a development database will be created. 
The development database is a working environment that facilitates the development, testing, 
troubleshooting, enhancement, and training for the data management system (DMS) without 
adversely affecting the integrity of the collected project data. 
8.2.2 
Data Entry and Data Quality Module 
The CRCU will configure a database to allow remote data entry (RDC) from the participating trial 
sites. The RDC module will be available to any computer with a persistent internet connection and 
will be run using standard web browser software. Data entry checks will be included in the entry 
screen designs where appropriate to limit the opportunity for erroneous entries due to mistyping. 
Such data entry checks would include value range comparisons, valid data type checks, required 
value checks, and skip pattern enforcement. This data entry module will be configured for single data 
entry.  Validation rules will be defined by CRCU clinical data management personnel, working closely 
with the investigators, to identify data items that may have been collected incorrectly or entered into 
the database inaccurately. Clinical site personnel will review the results of the data validation queries 
and take any required corrective action for invalid data. Queries will be recorded and tracked in the 
data quality module. Corrections identified for individual data items will be managed by the clinical 
sites. All changes made will be recorded in an electronic audit trail and documented using change 
control procedures. 
8.2.3 
Reports Dashboard Module 
Reports Dashboard Module: Using RStudio Connect software, the CRCU will create a dashboard 
specific to the functional needs of the trial. The dashboard will make it possible to connect to a 
variety of data sources, and explore the objects and data inside the connection. CRCU will design 
and program standard report templates that can be scheduled to be executed/distributed on 
proscribed timetables. Assigned users will be able to share and collaborate on study data for 
purposes of conducting site audits, monitoring the trial progress and safety, providing individual and 
overall feedback to sites and committees, and assessing data entry metrics and data quality 
assessments.  
8.2.4 
Testing 
Prior to deployment and use by the clinical sites and CRCU personnel, the database and DMS will 
be subjected to extensive functional testing. This testing is conducted according to a written test plan 
and is intended to verify the proper functioning of all components of the DMS. Any components that 
do not function as they were intended will be identified and evaluated by the development team to 
determine appropriate corrective action. Testing will also include an evaluation by user 
representatives for adherence to the requirements established by the intended users for the DMS. 
Successful completion of these user acceptance tests will mark the end of development and 
predicate the deployment of the DMS for use in storing and managing active trial data. 

 
 
 
27 
 
 8.2.5 
Operations and Administration 
RTD personnel will provide database administration services for the life of the project and will 
maintain the database in an appropriate manner for the retention period required by regulation and 
institutional policies. Database administration includes user account maintenance, database security 
and performance monitoring and tuning. RTD staff will also perform operational tasks as necessary. 
Such tasks may include electronic data loading, database software management, and process 
documentation. 
8.2.6 
Maintenance 
RTD staff will maintain the DMS over the life of the study. Any modifications made to the DMS will be 
conducted in accordance with change control procedures. RTD will work closely with study 
investigators and CDM personnel to identify desired modifications and to assess the impact of any 
proposed modifications. 
8.2.7 
User Training 
CRCU staff will provide training annually, prior to the start of each phase, for trial personnel 
associated with data collection. This session will cover correct handling of all data collection 
instruments and use of the data entry system.  Training webinars will be conducted during study 
implementation at the sites. Site users will be instructed to perform mock data entry for the data 
management system training certification process.  
8.2.8 
User Support 
CRCU will interface with the participating sites (vary by year) to provide assistance with problem 
solving related to use of the remote data entry system, as needed.  
8.2.9 
Reports 
CRCU will generate periodic reports on subject accrual, study progress, adverse events and data 
quality will be generated and provided to investigators, reviewers, and any study quality and safety 
monitoring boards as may be specified by the protocol and applicable regulations. CRCU staff will 
collaborate with the biostatistics team on reports for the DSMB, Interim Analysis and Annual 
Continuing Reviews and other NHLBI mandated reports. 
8.2.10 Sharing Data with Analytic Core 
The CRCU will work closely with biostatistics faculty and staff to ensure that data analysis needs are 
addressed during the design phase of the data collection instruments and the trial DMS. CRCU staff 
will grant read-only access to the study data for biostatisticians to perform interim and final analyses. 
Data dictionaries and any other relevant information will be provided to the biostatisticians to 
facilitate their analyses.  
8.2.11 Database Lock 
CRCU staff will lock the database once all data have been collected.  Study data will be packaged for 
data sharing purposes. A snapshot of the database will be taken and stored in a secure folder. 
8.3 
Confidentiality 
All data and records generated during this study will be kept confidential in accordance with 
Institutional policies and HIPAA on subject privacy. The Investigator and other site personnel will not 
use such data and records for any purpose other than conducting the study.  
No identifiable data will be used for future study without first obtaining IRB approval. The investigator 
will obtain a data use agreement between provider and any recipient researchers before sharing a 
limited dataset.  

 
 
 
28 
 
 8.3.1 
Record Retention Policy Reconciliation 
In accordance with their local retention of records policies, the DCC will retain all information from 
this study for 6 years after the study is completed, after which PHI will be removed from the data set. 
We recognize that other relying sites may have different record retention requirements. For this 
reason, the DCC will provide all data submitted from relying sites to the DCC back to each Site PI in 
the form of “study close out packages” for each site at final data lock using a Penn IT approved 
secure method, allowing each site to meet their local record retention requirements. We will not 
submit Site-Specific IRB Amendments describing each site’s record retention requirements. 
8.4 
Regulatory and Ethical Considerations 
8.4.1 
Data and Safety Monitoring Plan 
8.4.1.1 
Data and Safety Monitoring Board 
A locally-appointed Data and Safety Monitoring Board (DSMB) will oversee this clinical trial. The 
purpose of the DSMB is to monitor safety and ensure participant welfare throughout the trial. The 
DSMB will advise the Federal funding agency (NHLBI) and the Principal Investigators regarding the 
continuing safety of study subjects and the continuing validity and scientific merit of the study. The 
DSMB will be responsible for monitoring accrual of study subjects, adherence to the protocol, 
assessments of data quality, performance of individual clinical sites, and review of SAEs should they 
occur.  
Proposed DSMB members will undergo conflict of interest review by the Compliance Operations and 
Conflict of Interest (COI) Department at Children’s Hospital of Philadelphia. The investigative team 
will provide the Department with the names and contact information of proposed DSMB members. 
Upon receiving this information, the Department will distribute COI disclosure forms to the proposed 
members for completion. Upon receipt of the forms, the Department will review any potential conflicts 
and seek clarification from each individual as needed. In accordance with FDA guidance on the 
structure and operation of clinical trial data monitoring committees, DSMB members should be 
independent of all entities sponsoring, organizing, conducting, or regulating the trial. They should not 
have any significant financial interest in the study’s conduct or outcome, nor be involved in the study 
design, nor work at the MPIs’ home institutions, nor serve as a Site PI for this trial during their period 
of DSMB membership. The Department will review COI forms to determine whether any 
consultancies, relationships with medical device companies, research support, financial interests, or 
any other relationships exist that could be construed as introducing potential bias to their role as a 
DSMB member. If any consultancies or financial interests of a proposed member may be viewed as 
potentially materially impacting their objectivity, they will not be invited to serve on the DSMB. Each 
DSMB member will be responsible for informing the sponsor and DSMB Chair if any relevant 
changes in financial interest or other developments affecting potential or perceived conflict of interest 
develop during the duration of DSMB membership. 
The investigators will draft a DSMB charter to guide its function for the trial and the charter will be 
reviewed, revised as necessary, and approved by the DSMB. The Data Coordinating Center (DCC) 
will send reports to DSMB members prior to each DSMB meeting. The DSMB will have the final say 
in determining meeting intervals. 
In its meetings the DSMB will review outcome data, safety data, equitable deimplementation data, 
and subject accrual. We propose convening one DSMB meeting with interim analysis planned 
following each of the 4 phases included in the trial: baseline measurement, active deimplementation, 
sustainability, and exploratory (when all enrollment will be complete), along with meetings scheduled 
ad hoc as necessary.  An interim analysis will be provided by the Analytic Core to the DSMB after 
each of the phases listed above. In addition, the DSMB will, at their discretion, be able to request 
additional interim analyses. The DSMB can recommend whether or not to terminate enrollment in the 
trial because of potential safety concerns or study feasibility issues. At the final meeting following 
completion of subject enrollment, the DSMB will review final subject accrual and safety data to 
include in a written report. 

 
 
 
29 
 
 8.4.1.2 
Adverse Event Surveillance, Reporting, and Management 
The risks of this trial are minimal as it involves deimplementation strategy interventions intended to 
align clinical practice with established evidence, national guidelines, and recommendations for high 
quality bronchiolitis care. This is primarily an implementation trial, rather than a trial focused on 
efficacy or effectiveness. The interventions are assigned at the cluster (hospital) level to staff who 
are caring for patients with bronchiolitis on participating units. In this section we describe potential 
adverse events in patients that may result from changes in staff physiologic monitoring practices, 
which may or may not be attributable to the deimplementation strategy interventions. 
Since the interventions are assigned at the hospital level to staff who are caring for patients with 
bronchiolitis, we will perform surveillance for adverse events in all bronchiolitis patients hospitalized 
on units participating in the study. This also includes (a) patients who were not subjects of data 
collection because the intervention is applied to staff working on participating units and could impact 
the care of patients with bronchiolitis even if they are not reviewed during data collection rounds, and 
(b) patients who may not have had their care impacted in any way by the trial interventions because 
they are applied to staff members, not directly to patients.  
On a monthly basis during each post-randomization phase (P2, P3, and P4), site investigators will 
perform surveillance for (a) readmission of bronchiolitis patients within 7 days of discharge from units 
participating in the trial and (b) code blue and rapid response team activations in bronchiolitis 
patients hospitalized on units participating in the study that have the potential to meet “unanticipated 
problems involving risk to subjects” criteria. In order to establish a baseline for these events prior to 
any deimplementation interventions, we will also ask each site that will be continuing on to P2 to 
perform retrospective surveillance using data from their highest data collection volume month in P1. 
Using existing local patient safety databases and reports (e.g., local code blue and rapid response 
team activation logs, readmission reports), site investigators will review the charts of each 
bronchiolitis patient who was readmitted or was the subject of a code blue or rapid response team 
call. Readmission reviews will initially determine if the patient was hypoxemic to <85% at the time of 
re-presentation to the emergency department. Code blue and rapid response team activation 
reviews will initially determine if the patient was unmonitored and subsequently found to be 
hypoxemic to <85% at the time of the event when SpO2 monitors were applied. Readmissions and 
code blue/rapid response team activations meeting those initial criteria will be considered study AEs, 
whether or not they are related to the research intervention. Site investigators will complete a report 
form assessing the items in the CHOP IRB “Unanticipated Problems Decision Tree” to analyze 
relatedness, unexpectedness, and seriousness. Relatedness will be assessed through chart review 
and speaking with staff involved in the patient’s care if necessary to determine if the event was at 
least possibly related to the study interventions, summarized as “there is a reasonable possibility that 
the event may have been caused by the trial interventions.” See additional details below in the 
Section “Risk of deimplementation procedures” for context to inform assessment of relatedness, 
unexpectedness, and seriousness. Site investigators will report events that meet “unanticipated 
problems involving risk to subjects” criteria to the DCC and their local IRBs in accordance with the 
regulatory timelines indicated below and local IRB policies.  
SAE and Unanticipated Problem reporting will adhere to the NHLBI SAE and Unanticipated Problem 
Event Reporting Timelines illustrated in the table below and available at 
https://www.nhlbi.nih.gov/grants-and-training/policies-and-guidelines/nhlbi-adverse-event-and-
unanticipated-problem-reporting-policy. Investigators must also take into account local IRB reporting 
timelines if they are shorter than NHLBI Policy. Relying sites’ local IRB policies with shorter reporting 
timelines for SAEs and Unanticipated Problems supersede longer timelines in the NHLBI guidance. 
SERIOUS ADVERSE EVENT AND UNANTICIPATED PROBLEM REPORTING TIMELINES  

 
 
 
30 
 
 
Event Type 
When Event is Reported 
Shown in italics: Reconciliation between local IRB Policy and 
NHLBI Policy based on applying the policy with the shorter 
reporting timeframe using CHOP as an example. Other relying 
sites’ local IRB reporting requirements may differ. Relying sites’ 
IRB policies with shorter reporting timelines for SAEs and 
Unanticipated Problems supersede longer timelines in the NHLBI 
guidance. 
By Whom 
Event is 
Reported 
To Whom Event is 
Reported 
Fatal or life-
threatening 
unexpected, 
suspected 
serious 
adverse 
reactions 
NHLBI Policy: Within 7 calendar days of initial receipt of 
information. 
CHOP IRB Policy: SAEs that involve a CHOP subject and involve 
the death of the subject or are considered life-threatening need to 
be reported to the IRB within 1 business day of discovery 
(telephone, fax, email, eIRB) with a full report submitted in eIRB 
within 48 hours of the initial notification. All unanticipated problems 
involving subjects external to CHOP must be reported to the CHOP 
IRB within 7 business days of receipt of the report from the study 
sponsor, data coordinating center or overall study PI. 
Policy reconciliation: Refer to CHOP Policy for reporting events 
involving CHOP subjects to CHOP IRB. Refer to NHLBI policy for 
all other reporting and reports to CHOP IRB regarding subjects 
external to CHOP. Relying sites’ IRB policies with shorter reporting 
timelines for SAEs and Unanticipated Problems supersede longer 
timelines in the NHLBI guidance. 
Investigator 
 
 
 
Sponsor or 
designee1 
• 
Local/internal 
IRBs 
• 
NHLBI and/or 
Data 
Coordinating 
Center (DCC) 
 
• 
FDA (if IND 
study) 
Non-fatal, non-
life-
threatening 
unexpected, 
suspected 
serious 
adverse 
reactions 
NHLBI Policy: Within 15 calendar days of initial receipt of 
information. 
CHOP IRB Policy: All other unanticipated problems involving 
CHOP subjects must be reported within 7 business days of 
discovery. All unanticipated problems involving subjects external to 
CHOP must be reported to the CHOP IRB within 7 business days 
of receipt of the report from the study sponsor, data coordinating 
center or overall study PI. 
Policy reconciliation: Refer to CHOP Policy for reporting to CHOP 
IRB. Refer to NHLBI policy for all other reporting. Relying sites’ IRB 
policies with shorter reporting timelines for SAEs and 
Unanticipated Problems supersede longer timelines in the NHLBI 
guidance. 
Investigator 
 
 
 
Sponsor or 
designee 
• 
Local/internal 
IRBs/Institutional 
Officials 
• 
NHLBI and/or 
DCC 
 
• 
FDA 
(IND/Marketed 
Products) 
• 
All participating 
investigators 
Unanticipated 
adverse 
device effects 
NHLBI Policy: Within 10 working days of investigator first learning 
of effect. 
CHOP IRB Policy: All other unanticipated problems involving 
CHOP subjects must be reported within 7 business days of 
Investigator 
 
• 
Local/internal 
IRBs 

 
 
 
31 
 
 
discovery. All unanticipated problems involving subjects external to 
CHOP must be reported to the CHOP IRB within 7 business days 
of receipt of the report from the study sponsor, data coordinating 
center or overall study PI. 
Policy reconciliation: Refer to CHOP Policy for reporting to CHOP 
IRB. Refer to NHLBI policy for all other reporting. Relying sites’ IRB 
policies with shorter reporting timelines for SAEs and 
Unanticipated Problems supersede longer timelines in the NHLBI 
guidance. 
 
 
Sponsor or 
designee 
• 
NHLBI and/or 
DCC 
 
• 
FDA (if IDE 
study) 
Unanticipated 
Problem that 
is not an SAE 
NHLBI Policy: Within 14 days of the investigator becoming aware 
of the problem. 
CHOP IRB Policy: All other unanticipated problems involving 
CHOP subjects must be reported within 7 business days of 
discovery. All unanticipated problems involving subjects external to 
CHOP must be reported to the CHOP IRB within 7 business days 
of receipt of the report from the study sponsor, data coordinating 
center or overall study PI. 
Policy reconciliation: Refer to CHOP Policy for reporting to CHOP 
IRB. Refer to NHLBI policy for all other reporting. Relying sites’ IRB 
policies with shorter reporting timelines for SAEs and 
Unanticipated Problems supersede longer timelines in the NHLBI 
guidance. 
Investigator 
• 
Local/internal 
IRBs/Institutional 
Officials 
• 
NHLBI and/or 
DCC 
All 
Unanticipated 
Problems2 
NHLBI Policy: Within 30 days of the IRB’s receipt of the report of 
the unanticipated problem from the investigator. 
IRB 
 
 
Investigator3 
• 
OHRP 
 
 
• 
External IRBs 
NHLBI Policy Footnotes: 
1. 
Designee is appointed by the sponsor; for example, DCC, CRO. 
2. 
Per OHRP guidance: only when a particular AE or series of AEs is determined to meet the criteria for an UP should a report of 
the AE(s) be submitted to the IRB at each institution under the HHS regulations at 45 CFR part 46. Typically, such reports to 
the IRBs are submitted by investigators. 
3. 
Investigators should also take into account local IRB guidance if reporting timelines for UPs are shorter than OHRP guidance 
 

 
 
 
32 
 
  
One or more medical monitor(s) will be appointed by the PIs in consultation with the DSMB. The 
medical monitor(s) will assess all Adverse Event and Unanticipated Problem reports sent to the DCC 
by site investigators, reviewing and electronically signing each report. The monitor will then follow the 
CHOP IRB “Unanticipated Problems Decision Tree” to determine the appropriate course of action, 
keeping in mind the expected natural history and outcomes outlined below in the Section “Risk of 
deimplementation procedures.” Reports that suggest that subjects or others are placed at a greater 
risk of harm then initially anticipated will be accompanied by an Action Plan that outlines the steps 
that will be taken to mitigate the newly identified risk(s). 
8.4.2 
Risk Assessment 
Risks are not greater than minimal in this study.  
8.4.2.1 
Risks of direct observation and medical record review 
The primary risk of direct observation and medical record review procedures is breach of 
confidentiality of the data collected. These risks are mitigated by using the measures described 
above. In addition, each participant will be assigned a study identification number. This number will 
be used instead of names and other private information in any data exported from the primary 
database. 
8.4.2.2 
Risks of questionnaires 
The primary risk of questionnaire procedures is breach of confidentiality of the data collected. These 
risks are mitigated by using the measures described above. In addition, each participant will be 
assigned a study identification number. This number will be used instead of names and other private 
information in any data exported from the questionnaire database. 
8.4.2.3 
Risks of qualitative interviews 
The risks of qualitative interview procedures include: 
• 
Breach of confidentiality (from data or audio recording)  
• 
Momentary embarrassment or discomfort 
These risks are mitigated by using the measures described above. In addition, each participant will 
be assigned a study identification number. This number will be used on data collection forms and in 
the database instead of names and other private information. A separate list will be maintained that 
will links each participant's name to the study identification number for future reference and 
communication. To mitigate risk of embarrassment or discomfort, subjects will be informed that they 
do not have to answer any questions that make them too uncomfortable. For the recording of the 
interview, no one other than the research team, the person who takes notes during the interview, and 
the transcription company will hear the recordings. If someone’s name is mentioned, it will not be 
included on any notes or transcription. And if there are confidentiality concerns related to specific 
questions, the subject may elect to skip any question or stop the interview at any time. 
8.4.2.4 
Risks of deimplementation procedures 
The risks of deimplementation procedures include: 
• 
Clinicians extending their practice beyond the recommendations delivered in this trial and 
deimplementing monitoring in patient populations for whom deimplementation is not 
recommended (e.g. patients requiring supplemental oxygen) 
• 
Missed episodes of desaturation in patients not requiring supplemental oxygen, which in 
some situations could represent patient deterioration 

 
 
 
33 
 
 Brief episodes of desaturation are common and expected in bronchiolitis patients during recovery 
from acute illness, and is rarely associated with adverse outcomes. A landmark 2016 study by 
Principi and colleagues enrolled 118 infants with bronchiolitis at the time of emergency department 
discharge.101  
Given that there are 3 sets of national recommendations that promote using less continuous 
monitoring in stable bronchiolitis, this study is not introducing any additional risk than would be 
introduced if clinical leaders initiated efforts into routine clinical care to align their hospitals’ practice 
with the existing national guidelines. The trial simply aims to determine the best way to integrate and 
sustain this practice by showing clinicians’ their own data, providing information/education on the 
existing recommendations, and providing decision support in the EHR. All decisions about changes 
to monitoring are made by the primary medical team, not the researchers. 
In addition, we have outlined above a plan to perform surveillance for (a) readmission of bronchiolitis 
patients within 7 days of discharge from units participating in the trial and (b) code blue and rapid 
response team activations in bronchiolitis patients hospitalized on units participating in the study. In 
our pilot study, no episodes of deterioration were attributable to the deimplementation interventions. 
8.4.3 
Potential Benefits of Trial Participation 
8.4.3.1 
Direct 
There are no direct benefits to participation in this study. 
8.4.3.2 
Indirect 
The study outlined in this proposal will generate important knowledge that will advance the fields of 
implementation science and hospital medicine, specifically advancing methods to promote 
sustainable deimplementation of ineffective medical practices in the hospital setting. 
Patients, their families, and the health professionals caring for them have the potential to indirectly 
benefit from higher quality, guideline-concordant health care. This may produce fewer monitor 
alarms, and thereby fewer interruptions from alarms that may break the concentration of health 
professionals and contribute to errors, wake patients from sleep, or cause unnecessary anxiety in 
patients and their loved ones. The patients also may have a lower risk of harm due to a reduction in 
the probability of the clinicians caring for them experiencing the alarm fatigue that can result from 
high alarm rates. Patients may also benefit by not having their hospital stays prolonged 
unnecessarily due to the overdiagnosis of inconsequential mild hypoxemia. 
8.4.4 
Risk-Benefit Assessment 
Based on the discussion above and the lack of deviation from standard clinical care, the benefits of 
this study far outweigh the minimal risks. Given that the risks of the study are minimal, it is 
reasonable to proceed with the project.  
8.5 
Recruitment Strategy 
8.5.1 
Recruitment of hospital sites 
Recruitment of US and Canadian hospital sites for this trial commenced in fall 2020 while the PIs 
were preparing the grant proposal. At that time a message was sent to all PRIS Network hospitals 
inviting them to express preliminary interest in participating in the trial. A total of 67 sites expressed 
interest and 52 sites signed a letter of support included in the grant proposal. Post-award, we will 
reach out to all sites that initially expressed interest in 2020 as well as additional PRIS sites that 
previously participated in EMO projects but did not express initial interest in 2020. Sites from outside 
the PRIS Network referred to our team will also be considered for inclusion.  
Among sites that previously participated in the original 56-hospital EMO observational study,24 we 
will exclude sites that: 
• 
Failed to collect sufficient data in original EMO observational study24 (<20 observations) 

 
 
 
34 
 
 
• 
Had baseline adjusted overuse <30% in original EMO observational study.24 We chose this 
cut point because, of the prior EMO sites that collected sufficient data in the original EMO 
observational study24 and did not partake in the pilot study (n=43), if we took all, the average 
baseline among prior EMO hospitals would be 45% overuse. The power calculation 
assumes a baseline of 50% overuse. If we exclude hospitals with <30% baseline overuse 
from the trial, the average baseline among prior EMO participants would be 52% overuse 
(48% penetration), aligned with the power calculation. 
All sites that participated in the 6-hospital pilot study will be invited to participate regardless of their 
baseline overuse percentage. 
If we fail to engage at least 50 sites in commencing the IRB reliance and data use agreement 
process, we will perform additional outreach to PRIS Network sites and may consider expanding to 
direct recruitment outside of the PRIS Network (e.g. in the American Academy of Pediatrics Value in 
Inpatient Pediatrics Network). 
8.5.2 
Recruitment of individuals 
8.5.2.1 
Patients for in-person observational rounds 
For the data collection in-person observational rounds, site investigators will identify all patients 
currently admitted to their hospital’s non-ICU, non-emergency department, non-step down units with 
bronchiolitis by reviewing the census of each unit that cares for children with bronchiolitis and 
examining the charts of patients meeting age criteria to determine if they are eligible.  
8.5.2.2 
Staff for questionnaires 
The DCC will electronically distribute the staff questionnaire measures to the following health care 
professional types who worked on an intervention unit in a role involving the care of patients with 
bronchiolitis during the intervention period: 
• 
Attending physicians 
• 
Resident physicians 
• 
Nurses 
 
Site PIs will prepare lists of the nurses and physicians who provide care for bronchiolitis patients on 
the units participating in the study, as well as administrators who oversee the medical care and 
safety for bronchiolitis patients at institutional and unit levels. Site PIs will submit these lists of staff 
email addresses of these potential subjects to the DCC, who will import these into the survey 
distribution system.  
8.5.2.3 
Staff for qualitative interviews 
For the qualitative interviews, using a deviance approach,85,86 we will conduct semi-structured 
interviews with nurses and physicians who provide care to bronchiolitis patients (the exploratory 
study phase that examines sustainability 2 years after withdrawal of educational outreach and A&F) 
as well as with CDS Coaches. In collaboration with site PIs, eligible clinicians will be identified 
purposively from staff rosters and invited to participate in interviews by telephone call, texting, and/or 
emailing to discuss the process of deimplementation as experienced by stakeholders. Purposive 
sampling will seek to obtain a wide range of nurse and physician experience levels, nurse and 
physician supervisory vs front line care roles, nurse shift (day vs night shift), and physician care 
model (teaching service vs attending-only hospitalist model). Interviews will explore mechanistic 
relationships between (a) the strategies, (b) the findings of the quantitative measures above, and (c) 
sustainability.  
8.5.2.4 
Parents or guardians for qualitative interviews 
For the qualitative interviews of families, eligible parents or guardians will be identified purposively 
from bronchiolitis patients’ trial records during P3 and invited to interview by telephone call, texting, 
and/or emailing during the 8 weeks following discharge in order to explore their perceptions of, and 

 
 
 
35 
 
 reactions to, SpO2 deimplementation activities. Purposive sampling will seek to obtain a wide range 
of patient ages since parents’ perspectives likely differ for very young infants (e.g. 2-3 months old) 
compared to older children (e.g. 18-23 months old). 
8.6 
Informed Consent/Assent and HIPAA Authorization 
8.6.1 
Population 1: Hospital Staff 
Population 1 includes all hospital staff who are exposed to the interventions by being present on 
participating hospital units or overseeing bronchiolitis care during active deimplementation, 
sustainability, and exploratory phases. These hospital staff members are considered third parties to 
the research. They are not considered research subjects and informed consent is not required 
unless they participate in study questionnaires or qualitative interviews (and thereby fall into hospital 
staff research subject subpopulations 1a or 1b, outlined below). 
8.6.1.1 
Population 1a 
Hospital staff who complete study questionnaires. Informed consent will be obtained for the 
questionnaires administered to health care professionals. We request a waiver of documentation of 
consent. The first section of the electronic questionnaire webpage will include the required elements 
of consent and a statement that completion of the questionnaire indicates willingness to participate. 
Specific language is included in the eIRB application attachments. HIPAA Authorization is not 
necessary for the questionnaire portion of the study; names and email addresses of health care 
professionals are being collected in order to distribute and manage the questionnaire but are not 
considered PHI because they do not relate to the health of an individual. The potential subjects will 
have ample time to review the consent language, minimizing the likelihood of coercion. 
We seek to enroll subjects under IRB SOP 501 8(a) to allow recruitment of employees and trainees 
who are under the supervision of the investigator(s), including those who report to Site PIs as 
participants in study questionnaires. This waiver is appropriate because employees and trainees 
(e.g. residents and fellows) working in Pediatric Hospital Medicine are by design the subjects of this 
aspect of the research and are the same specialty as the Site PIs in most cases. Investigators will 
not have access to data or reports that would allow them to know which employees or trainees chose 
to participate or not participate. The research could not practicably be carried out without the waiver 
or alteration. The intent is to obtain a large, unbiased sample of clinicians to obtain insights on care 
related to pulse oximetry monitoring in bronchiolitis that will then help us understand the mechanisms 
(mediators and moderators) of the interventions. The research could not practicably be carried out 
without the waiver or alteration, as approximately 25% of sites have Site PIs with some form of 
leadership role that would prohibit us from recruiting subject cohorts at their sites, which would 
threaten the above research goal. The study could not practicably be carried out without the Waiver 
of Documentation of Consent as subjects will not be seen in person. 
8.6.1.2 
Population 1b 
Hospital staff who participate in qualitative interviews. Informed consent will be obtained for the 
qualitative interviews of staff. We request waiver of documentation under 45 CFR 46.117©(1)(ii). 
Interviews will be conducted by telephone or using teleconferencing software (e.g. Zoom, BlueJeans, 
Teams). The investigator will obtain consent following the same requirements as written consent but 
the subject will not sign a consent form. The form employed uses the CHOP consent template but 
substitutes the usual signature page with the IRB's documentation of verbal consent page. The study 
will be explained as outlined in the consent form and the subject will be given ample opportunity to 
ask questions, and will be provided with extra time to think about it if requested (to minimize the 
likelihood of coercion). If interested in participating, the documentation of verbal consent page of the 
form will be completed and the interview will proceed. HIPAA Authorization is not necessary for the 
staff interviews; names and email addresses of health care professionals are being collected in order 
to manage the interview process but are not considered PHI because they do not relate to the health 
of an individual. 

 
 
 
36 
 
 We seek to enroll subjects under IRB SOP 501 8(a) to allow recruitment of employees and trainees 
who are under the supervision of the investigator(s), including those who report to Site PIs as 
participants in study interviews. This waiver is appropriate because employees and trainees (e.g. 
residents and fellows) working in Pediatric Hospital Medicine are by design the subjects of this 
aspect of the research and are the same specialty as the Site PIs in most cases. Investigators will 
not have access to data or reports that would allow them to know which employees or trainees chose 
to participate or not participate. The research could not practicably be carried out without the waiver 
or alteration. The intent is to conduct semi-structured interviews aiming to better understand 
mechanisms of practice change in response to each deimplementation strategy . The research could 
not practicably be carried out without the waiver or alteration, as approximately 25% of sites have 
Site PIs with some form of leadership role that would prohibit us from recruiting subject cohorts at 
their sites, which would threaten the above research goal. The study could not practicably be carried 
out without the Waiver of Documentation of Consent as subjects will not be seen in person. 
 
8.6.2 
Populations 2a and 2b 
(2a) Bronchiolitis patients directly observed while not receiving supplemental oxygen (“in room air,” 
for primary trial outcome) and (2b) Bronchiolitis patients directly observed while receiving 
supplemental oxygen (for underuse evaluation). 
Waivers of consent/ assent and of HIPAA Authorization are requested.  
The research meets the criteria of 45 CFR 46.116(d), as illustrated below:  
• 
Risks are not greater than minimal in this study. The primary risk is breach of privacy and 
confidentiality. The risk will be minimized by keeping all data and records confidential in 
accordance with institutional policies on subject privacy. The investigators will not use data 
or records for any purpose other than conducting the study.  
• 
The waiver or alteration will not adversely affect the rights and welfare of the subjects 
because PHI will only be accessed by IRB approved study staff performing data collection, 
and we have taken extensive precautions by using a professional DCC to minimize risk of 
PHI exposure. 
• 
The research could not practicably be carried out without the waiver or alteration. The intent 
is to obtain a large, unbiased sample of patients with bronchiolitis to obtain insights on 
hospital-level practices and systems of care related to pulse oximetry monitoring in 
bronchiolitis. The research could not practicably be carried out without the waiver or 
alteration, as it will not be feasible to locate and contact the families of all of the patients who 
are being monitored in room air with bronchiolitis. All of the patients admitted to study sites 
with bronchiolitis who are monitored in room air are required in order for this to be an 
unbiased sample representative of the population with adequate sample size. 
• 
There will not be any pertinent patient-level information that would benefit the patient 
directly after participation so none of the subjects will be provided with additional 
information after participation. 
We also request a waiver of HIPAA Authorization for the patient subjects. The study presents no 
more than minimal risk to the privacy of individuals. The research meets the criteria of 45 CFR 
46.116(d), as illustrated below: 
 (A) The use or disclosure of protected health information involves no more than a 
minimal risk to the privacy of individuals, based on, at least, the presence of the following 
elements:  
(1) An adequate plan to protect the identifiers from improper use and disclosure;  
• 
We have provided an adequate plan to protect identifiers from improper 
use and disclosure above. 

 
 
 
37 
 
 
(2) An adequate plan to destroy the identifiers at the earliest opportunity consistent 
with conduct of the research, unless there is a health or research justification for 
retaining the identifiers or such retention is otherwise required by law; and  
• 
We have provided a plan to destroy the identifiers at the earliest 
opportunity consistent with the conduct of the research that is also 
consistent with institutional record retention policies. 
(3) Adequate written assurances that the protected health information will not be 
reused or disclosed to any other person or entity, except as required by law, for 
authorized oversight of the research project, or for other research for which the use 
or disclosure of protected health information would be permitted by this subpart;  
• 
Protected health information will not be reused or disclosed to any other 
entity except as required by law, for authorized oversight of the research 
project, or for other research for which the use or disclosure of protected 
health information would be permitted by this subpart. 
(B) The research could not practicably be conducted without the waiver or alteration; and  
• 
The research could not practicably be conducted without the waiver as 
described above in waiver of consent request. 
(C) The research could not practicably be conducted without access to and use of the 
protected health information. 
• 
The research could not practicably be conducted without access to and 
use of protected health information needed to track patients over time 
and to track the dates of admission and data collection at each site 
during the trial. 
8.6.3 
Population 3 
Parents or guardians of bronchiolitis patients who participate in qualitative interviews. Informed 
consent will be obtained for the qualitative interviews of parents and guardians. We request a Waiver 
of HIPAA Authorization for Recruitment Purposes for Population 3, the parents/guardians of 
bronchiolitis patients whose contact information we have proposed collecting so that we may contact 
them and invite them to participate in the qualitative interviews. The study presents no more than 
minimal risk to the privacy of individuals. The research meets the criteria of 45 CFR 46.116(d), as 
illustrated below: 
 (A) The use or disclosure of protected health information involves no more than a 
minimal risk to the privacy of individuals, based on, at least, the presence of the following 
elements:  
(1) An adequate plan to protect the identifiers from improper use and disclosure;  
• 
We have provided an adequate plan to protect identifiers from improper 
use and disclosure above. 
(2) An adequate plan to destroy the identifiers at the earliest opportunity consistent 
with conduct of the research, unless there is a health or research justification for 
retaining the identifiers or such retention is otherwise required by law; and  
• 
We have provided a plan to destroy the identifiers at the earliest 
opportunity consistent with the conduct of the research that is also 
consistent with institutional record retention policies. 
(3) Adequate written assurances that the protected health information will not be 
reused or disclosed to any other person or entity, except as required by law, for 
authorized oversight of the research project, or for other research for which the use 
or disclosure of protected health information would be permitted by this subpart;  

 
 
 
38 
 
 
• 
Protected health information will not be reused or disclosed to any other 
entity except as required by law, for authorized oversight of the research 
project, or for other research for which the use or disclosure of protected 
health information would be permitted by this subpart. 
(B) The research could not practicably be conducted without the waiver or alteration; and  
• 
The research could not practicably be conducted without the waiver 
because subjects will not be met in person during the hospitalization due 
to the design of the data collection procedures for the patients 
(populations 2a and 2b).  
(C) The research could not practicably be conducted without access to and use of the 
protected health information. 
• 
The research could not practicably be conducted without access to and 
use of protected health information needed to contact the 
parents/guardians to recruit them to participate. 
We also request waiver of documentation under 45 CFR 46.117(c)(1)(ii). Interviews will be 
conducted by telephone or using teleconferencing software (e.g. Zoom, BlueJeans, Teams). The 
investigator must obtain consent following the same requirements as written consent but the subject 
does not sign a consent form. The form substitutes the usual signature page with the IRB's 
documentation of verbal consent page. The study will be explained as outlined in the consent form 
and the subject will be given ample opportunity to ask questions, and will be provided with extra time 
to think about it if requested (to minimize the likelihood of coercion). If interested in participating, the 
documentation of verbal consent page of the form will be completed and the interview will proceed. 
HIPAA Authorization will be obtained using the same process with documentation on the same form. 
8.6.4 
Population 4 
 
Clinical Decision Support (CDS)Coaches who participate in qualitative interviews. In 
Phase 4 we will conduct semi-structured interviews with CDS Coaches. Informed consent will be 
obtained for the qualitative interviews. We request waiver of documentation under 45 CFR 
46.117©(1)(ii). Interviews will be conducted by telephone or using teleconferencing software (e.g. 
Zoom, BlueJeans, Teams). The investigator will obtain consent following the same requirements as 
written consent but the subject will not sign a consent form. The form employed uses the CHOP 
consent template but substitutes the usual signature page with the IRB's documentation of verbal 
consent page. The study will be explained as outlined in the consent form and the subject will be 
given ample opportunity to ask questions, and will be provided with extra time to think about it if 
requested (to minimize the likelihood of coercion). If interested in participating, the documentation of 
verbal consent page of the form will be completed and the interview will proceed. HIPAA 
Authorization is not necessary for the interviews; names and email addresses are being collected in 
order to manage the interview process but are not considered PHI because they do not relate to the 
health of an individual. 
We seek to enroll subjects under IRB SOP 501 8(a) to allow recruitment of employees and trainees 
who are under the supervision of the investigator(s), including those who report to Site PIs as 
participants in study interviews. This waiver is appropriate because CDS coaches, employees, and 
trainees (e.g. residents and fellows) working in Pediatric Hospital Medicine are by design the 
subjects of this aspect of the research and are the same specialty as the Site PIs in most cases. 
Investigators will not have access to data or reports that would allow them to know which employees 
or trainees chose to participate or not participate. The research could not practicably be carried out 
without the waiver or alteration. The intent is to conduct semi-structured interviews with CDS 
Coaches who participated in the design and development of decision support for bronchiolitis 
patients aiming to better understand mechanisms of practice change in response to each 
deimplementation strategy. The research could not practicably be carried out without the waiver or 
alteration, as approximately 25% of sites have Site PIs with some form of leadership role that would 
prohibit us from recruiting subject cohorts at their sites, which would threaten the above research 

 
 
 
39 
 
 goal. The study could not practicably be carried out without the Waiver of Documentation of Consent 
as subjects will not be seen in person. 
 
 
8.7 
Payment to Subjects/Families 
Staff and parent/guardian participants in qualitative interviews will each be paid $25 for their time and 
effort. No other subjects will be paid for their participation.  
9 
PUBLICATION 
Peer-reviewed publication is planned. No identifiable information will be used in publication. 
 
 
10 REFERENCES 
1.  
Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. 
Implementation Science. 2020;15(1):2. doi:10.1186/s13012-019-0960-9 
2.  
Hasegawa K, Tsugawa Y, Brown DFM, Mansbach JM, Camargo CA. Trends in bronchiolitis hospitalizations in the 
United States, 2000-2009. Pediatrics. 2013;132(1):28-36. 
3.  
Keren R, Luan X, Localio R, et al. Prioritization of comparative effectiveness research topics in hospital pediatrics. Arch 
Pediatr Adolesc Med. 2012;166(12):1155-1164. 
4.  
McCulloh R, Koster M, Ralston S, et al. Use of intermittent vs. continuous pulse oximetry for nonhypoxemic infants and 
young children hospitalized for bronchiolitis: a randomized clinical trial. JAMA Pediatr. 2015;169(10):898-904. 
5.  
Mahant S, Wahi G, Bayliss A, et al. Intermittent vs. continuous pulse oximetry in hospitalized infants with bronchiolitis: 
a randomized clinical trial. JAMA Pediatrics. Published online 2021:Published online March 1, 2021. 
doi:doi:10.1001/jamapediatrics.2020.6141 
6.  
Cunningham S, Rodriguez A, Adams T, et al. Oxygen saturation targets in infants with bronchiolitis (BIDS): a double-
blind, randomised, equivalence trial. Lancet. 2015;386(9998):1041. 
7.  
Schroeder AR, Marmor AK, Pantell RH, Newman TB. Impact of pulse oximetry and oxygen therapy on length of stay in 
bronchiolitis hospitalizations. Archives of Pediatrics & Adolescent Medicine. 2004;158(6):527. 
8.  
Unger S, Cunningham S. Effect of oxygen supplementation on length of stay for infants hospitalized with acute viral 
bronchiolitis. Pediatrics. 2008;121(3):470-475. 
9.  
McBride SC, Chiang VW, Goldmann DA, Landrigan CP. Preventable adverse events in infants hospitalized with 
bronchiolitis. Pediatrics. 2005;116(3):603-608. 
10.  
Bonafide CP, Lin R, Zander M, et al. Association between exposure to nonactionable physiologic monitor alarms and 
response time in a children’s hospital. J Hosp Med. 2015;10(6):345-351. 
11.  
Bonafide CP, Localio AR, Holmes JH, et al. Video analysis of factors associated with response time to physiologic 
monitor alarms in a children’s hospital. JAMA Pediatr. 2017;171(6):524-531. 
12.  
Ralston SL, Lieberthal AS, Meissner HC, et al. Clinical practice guideline: the diagnosis, management, and prevention 
of bronchiolitis. Pediatrics. 2014;134(5):e1474-e1502. 
13.  
Quinonez RA, Garber MD, Schroeder AR, et al. Choosing wisely in pediatric hospital medicine: five opportunities for 
improved healthcare value. Journal of Hospital Medicine. 2013;8(9):479-485. 
14.  
Schondelmeyer AC, Dewan ML, Brady PW, et al. Cardiorespiratory and pulse oximetry monitoring in hospitalized 
children: a Delphi process. Pediatrics. 2020;146(2). doi:10.1542/peds.2019-3336 

 
 
 
40 
 
 15.  
Cheston CC, Vinci RJ. Overuse of continuous pulse oximetry for bronchiolitis: the need for deimplementation science. 
JAMA. 2020;323(15):1449-1450. doi:10.1001/jama.2020.4359 
16.  
Nilsen P, Ingvarsson S, Hasson H, von Thiele Schwarz U, Augustsson H. Theories, models, and frameworks for de-
implementation of low-value care: a scoping review of the literature. Implementation Research and Practice. 
2020;1:2633489520953762. doi:10.1177/2633489520953762 
17.  
Morgan DJ, Brownlee S, Leppin AL, et al. Setting a research agenda for medical overuse. BMJ. 2015;351:h4534. 
18.  
Brownlee S, Chalkidou K, Doust J, et al. Evidence for overuse of medical services around the world. The Lancet. 
2017;390(10090):156-168. 
19.  
Prasad V, Ioannidis JP. Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare 
practices. Implement Sci. 2014;9:1. 
20.  
Parsons Leigh J, Niven DJ, Boyd JM, Stelfox HT. Developing a framework to guide the de-adoption of low-value 
clinical practices in acute care medicine: a study protocol. BMC Health Serv Res. 2017;17(1):54. 
21.  
Wolf ER, Krist AH, Schroeder AR. Deimplementation in pediatrics: past, present, and future. JAMA Pediatr. 
2020;[online ahead of print]. doi:10.1001/jamapediatrics.2020.4681 
22.  
Fujiogi M, Goto T, Yasunaga H, et al. Trends in bronchiolitis hospitalizations in the United States: 2000–2016. 
Pediatrics. 2019;144(6):e20192614. doi:10.1542/peds.2019-2614 
23.  
Haynes AK, Prill MM, Iwane MK, Gerber SI, Centers for Disease Control and Prevention (CDC). Respiratory syncytial 
virus: United States, July 2012-June 2014. Morbidity and Mortality Weekly Report. 2014;63(48):1133-1136. 
24.  
Bonafide CP, Xiao R, Brady PW, et al. Prevalence of continuous pulse oximetry monitoring in hospitalized children with 
bronchiolitis not requiring supplemental oxygen. JAMA. 2020;323(15):1467-1477. doi:10.1001/jama.2020.2998 
25.  
Quinonez RA, Coon ER, Schroeder AR, Moyer VA. When technology creates uncertainty: pulse oximetry and 
overdiagnosis of hypoxaemia in bronchiolitis. BMJ. 2017;358:j3850. 
26.  
Cunningham S, Rodriguez A, Boyd KA, McIntosh E, Lewis SC, on behalf of the BIDS Collaborators Group. 
Bronchiolitis of Infancy Discharge Study (BIDS): a multicentre, parallel-group, double-blind, randomised controlled, 
equivalence trial with economic evaluation. Health Technology Assessment. 2015;19(71):1-172. 
27.  
Leidy NK, Margolis MK, Marcin JP, et al. The impact of severe respiratory syncytial virus on the child, caregiver, and 
family during hospitalization and recovery. Pediatrics. 2005;115(6):1536-1546. doi:10.1542/peds.2004-1149 
28.  
Chang LV, Shah AN, Hoefgen ER, et al. Lost earnings and nonmedical expenses of pediatric hospitalizations. 
Pediatrics. 2018;142(3):e20180195. doi:10.1542/peds.2018-0195 
29.  
Mansbach JM, Clark S, Piedra PA, et al. Hospital course and discharge criteria for children hospitalized with 
bronchiolitis. Journal of Hospital Medicine. 2015;10(4):205-211. doi:10.1002/jhm.2318 
30.  
Ralston S, Parikh K, Goodman D. Benchmarking overuse of medical interventions for bronchiolitis. JAMA Pediatr. 
2015;169(9):805. doi:10.1001/jamapediatrics.2015.1372 
31.  
Paine CW, Goel VV, Ely E, et al. Systematic review of physiologic monitor alarm characteristics and pragmatic 
interventions to reduce alarm frequency. Journal of Hospital Medicine. 2016;11(2):136-144. 
32.  
Winters BD, Cvach MM, Bonafide CP, et al. Technological distractions (part 2): a summary of approaches to manage 
clinical alarms with intent to reduce alarm fatigue. Critical Care Medicine. 2018;46(1):130-137. 
33.  
Schondelmeyer AC, Brady PW, Goel VV, et al. Physiologic monitor alarm rates at 5 children’s hospitals. Journal of 
Hospital Medicine. 2018;13(6):396-398. doi:10.12788/jhm.2918 
34.  
Lawless ST. Crying wolf: False alarms in a pediatric intensive care unit. Crit Care Med. 1994;22(6):981-985. 
35.  
Karnik A, Bonafide CP. A framework for reducing alarm fatigue on pediatric inpatient units. Hospital Pediatrics. 
2015;5(3):160-163. 

 
 
 
41 
 
 36.  
Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and 
innovations: a review of the empirical literature and recommendations for future research. Implementation Science. 
2012;7(1):17. doi:10.1186/1748-5908-7-17 
37.  
Ament SMC, de Groot JJA, Maessen JMC, Dirksen CD, van der Weijden T, Kleijnen J. Sustainability of professionals’ 
adherence to clinical practice guidelines in medical care: a systematic review. BMJ Open. 2015;5(12):e008073. 
doi:10.1136/bmjopen-2015-008073 
38.  
Gerber JS, Prasad PA, Fiks AG, et al. Effect of an outpatient antimicrobial stewardship intervention on broad-spectrum 
antibiotic prescribing by primary care pediatricians: a randomized trial. JAMA. 2013;309(22):2345-2352. 
doi:10.1001/jama.2013.6287 
39.  
Biondi EA, McCulloh R, Staggs VS, et al. Reducing Variability in the Infant Sepsis Evaluation (REVISE): a national 
quality initiative. Pediatrics. 2019;144(3):e20182201. doi:10.1542/peds.2018-2201 
40.  
Kaiser SV, Jennings B, Rodean J, et al. Pathways for Improving Inpatient Pediatric Asthma Care (PIPA): a multicenter, 
national study. Pediatrics. Published online May 6, 2020:e20193026. doi:10.1542/peds.2019-3026 
41.  
Ralston SL, Garber MD, Rice-Conboy E, et al. A multicenter collaborative to reduce unnecessary care in inpatient 
bronchiolitis. Pediatrics. 2016;137(1):e20150851. 
42.  
Chan WV, Pearson TA, Bennett GC, et al. ACC/AHA special report: Clinical practice guideline implementation 
strategies: a summary of systematic reviews by the NHLBI Implementation Science Work Group: A report of the 
American College of Cardiology/American Heart Association Task Force on clinical practice guidelines. Circulation. 
2017;135(9):e122-e137. 
43.  
Ivers NM, Grimshaw JM. Reducing research waste with implementation laboratories. Lancet. 2016;388(10044):547-
548. doi:10.1016/S0140-6736(16)31256-9 
44.  
Grimshaw JM, Ivers N, Linklater S, et al. Reinvigorating stagnant science: implementation laboratories and a meta-
laboratory to efficiently advance the science of audit and feedback. BMJ Quality & Safety. 2019;28(5):416-423. 
doi:10.1136/bmjqs-2018-008355 
45.  
Helfrich CD, Rose AJ, Hartmann CW, et al. How the Dual Process Model of human cognition can inform efforts to de-
implement ineffective and harmful clinical practices: a preliminary model of unlearning and substitution. J Eval Clin 
Pract. 2018;24(1):198-205. doi:10.1111/jep.12855 
46.  
Soong C, Shojania KG. Education as a low-value improvement intervention: often necessary but rarely sufficient. BMJ 
Quality & Safety. 2020;29(5):353-357. doi:10.1136/bmjqs-2019-010411 
47.  
Colla CH, Mainor AJ, Hargreaves C, Sequist T, Morden N. Interventions aimed at reducing use of low-value health 
services: a systematic review. Medical Care Research and Review. 2017;74(5):507-550. 
doi:10.1177/1077558716656970 
48.  
Kwan JL, Lo L, Ferguson J, et al. Computerised clinical decision support systems and absolute improvements in care: 
meta-analysis of controlled clinical trials. BMJ. 2020;370. doi:10.1136/bmj.m3216 
49.  
Schondelmeyer AC, Simmons JM, Statile AM, et al. Using quality improvement to reduce continuous pulse oximetry 
use in children with wheezing. Pediatrics. 2015;135(4):e1044-1051. doi:10.1542/peds.2014-2295 
50.  
Mittal S, Marlowe L, Blakeslee S, et al. Successful use of quality improvement methodology to reduce inpatient length 
of stay in bronchiolitis through judicious use of intermittent pulse oximetry. Hospital Pediatrics. 2019;9(2):73-78. 
51.  
Kaiser SV, Shadman KA, Biondi EA, McCulloh RJ. Feasible strategies for sustaining guideline adherence: cross-
sectional analysis of a national collaborative. Hospital Pediatrics. 2019;9(11):903-908. doi:10.1542/hpeds.2019-0152 
52.  
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services 
research findings into practice: a consolidated framework for advancing implementation science. Implementation 
Science. 2009;4(1):50. 
53.  
Wolk CB, Schondelmeyer AC, Barg FK, et al. Barriers and facilitators to guideline-adherent pulse oximetry use in 
bronchiolitis. Journal of Hospital Medicine. Published online 2020:In press. 

 
 
 
42 
 
 54.  
Fernandez ME, ten Hoor GA, van Lieshout S, et al. Implementation Mapping: using Intervention Mapping to develop 
implementation strategies. Frontiers in Public Health. 2019;7:158. doi:10.3389/fpubh.2019.00158 
55.  
ClinicalTrials.gov: Aligning Pulse Oximetry With Guidelines (EMO Pilot). ClinicalTrials.gov. Accessed November 13, 
2020. https://clinicaltrials.gov/ct2/show/NCT04178941 
56.  
Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome 
measures. Implementation Science. 2017;12(1):108. doi:10.1186/s13012-017-0635-3 
57.  
Slaghuis SS, Strating MM, Bal RA, Nieboer AP. A framework and a measurement instrument for sustainability of work 
practices in long-term care. BMC Health Services Research. 2011;11(1):314. doi:10.1186/1472-6963-11-314 
58.  
Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development 
and validity testing of the Implementation Climate Scale (ICS). Implementation Sci. 2014;9(1):157. 
doi:10.1186/s13012-014-0157-1 
59.  
Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): development of a brief measure of 
unit level implementation leadership. Implementation Sci. 2014;9(1):45. doi:10.1186/1748-5908-9-45 
60.  
Dillard JP, Shen L. On the nature of reactance and its role in persuasive health communication. Communication 
Monographs. 2005;72(2):144-168. doi:10.1080/03637750500111815 
61.  
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining 
elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 
2012;50(3):217-226. 
62.  
Simon TD, Starmer AJ, Conway PH, et al. Quality improvement research in pediatric hospital medicine and the role of 
the Pediatric Research in Inpatient Settings (PRIS) network. Acad Pediatr. 2013;13(6 Suppl):S54-60. 
doi:10.1016/j.acap.2013.04.006 
63.  
Moulton LH. Covariate-based constrained randomization of group-randomized trials. Clinical Trials. 2004;1(3):297-305. 
doi:10.1191/1740774504cn024oa 
64.  
Dickinson LM, Beaty B, Fox C, et al. Pragmatic cluster randomized trials using covariate constrained randomization: a 
method for practice-based research networks (PBRNs). The Journal of the American Board of Family Medicine. 
2015;28(5):663-672. doi:10.3122/jabfm.2015.05.150001 
65.  
Lorenz E, Gabrysch S. Covariate-constrained randomization routine for achieving baseline balance in cluster-
randomized trials. The Stata Journal. 2017;17(2):503-510. doi:10.1177/1536867X1701700214 
66.  
Hayes RJ, Moulton LH. Choosing Whether to Randomise by Cluster. In: Cluster Randomised Trials. 2nd Edition. CRC 
Press; 2017:25-29. 
67.  
Aron DC, Tseng C-L, Soroka O, Pogach LM. Balancing measures: identifying unintended consequences of diabetes 
quality performance measures in patients at high risk for hypoglycemia. Int J Qual Health Care. 2019;31(4):246-251. 
doi:10.1093/intqhc/mzy151 
68.  
Chi KW, Coon ER, Destino L, Schroeder AR. Parental perspectives on continuous pulse oximetry use in bronchiolitis 
hospitalizations. Pediatrics. 2020;146(2):e20200130. doi:10.1542/peds.2020-0130 
69.  
Bonafide CP, Localio AR, Stemler S, et al. Safety huddle intervention for reducing physiologic monitor alarms: a hybrid 
effectiveness-implementation cluster randomized trial. Journal of Hospital Medicine. 2018;13(9):609-615. 
70.  
Dewan M, Wolfe H, Lin R, et al. Impact of a safety huddle–based intervention on monitor alarm rates in low-acuity 
pediatric intensive care unit patients. Journal of Hospital Medicine. 2017;12(8):652-657. 
71.  
Email Marketing Software for Small Businesses. Constant Contact. Accessed November 4, 2020. 
https://www.constantcontact.com/email-marketing 
72.  
Kinsman L, Rotter T, James E, Snow P, Willis J. What is a clinical pathway? development of a definition to inform the 
debate. BMC Med. 2010;8:31. doi:10.1186/1741-7015-8-31 

 
 
 
43 
 
 73.  
Brower L, Schondelmeyer A, Wilson P, Shah SS. Testing and empiric treatment for neonatal herpes simplex virus: 
challenges and opportunities for improving the value of care. Hosp Pediatr. 2016;6(2):108-111. 
doi:10.1542/hpeds.2015-0166 
74.  
Brower LH, Wilson PM, Murtagh Kurowski E, et al. Using quality improvement to implement a standardized approach 
to neonatal herpes simplex virus. Pediatrics. 2019;144(2). doi:10.1542/peds.2018-0262 
75.  
Pater CM, Sosa TK, Boyer J, et al. Time series evaluation of improvement interventions to reduce alarm notifications in 
a paediatric hospital. BMJ Qual Saf. 2020;29(9):717-726. doi:10.1136/bmjqs-2019-010368 
76.  
Wright A, McCoy AB, Choudhry NK. Recommendations for the conduct and reporting of research involving flexible 
electronic health record–based interventions. Ann Intern Med. 2020;172(Supplement 11):S110-S115. 
doi:10.7326/M19-0880 
77.  
Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomised controlled trial be? BMJ. 
2004;328(7455):1561-1563. doi:10.1136/bmj.328.7455.1561 
78.  
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: 
the new Medical Research Council guidance. BMJ. 2008;337. doi:10.1136/bmj.a1655 
79.  
Osherhoff JA. Improving Medication Use and Outcomes with Clinical Decision Support: A Step-by-Step Guide. 
Healthcare Information and Management Systems Society; 2009. 
80.  
Campbell R. The five “rights” of clinical decision support. J AHIMA. 2013;84(10):42-47. 
81.  
Brady PW, Schondelmeyer AC, Landrigan CP, et al. Validity of continuous pulse oximetry orders for identification of 
actual monitoring status in bronchiolitis. Journal of Hospital Medicine. 2020;15(11):665-668. doi:10.12788/jhm.3443 
82.  
Silvia PJ. Reactance and the dynamics of disagreement: multiple paths from threatened freedom to resistance to 
persuasion. European Journal of Social Psychology. 2006;36(5):673-685. doi:10.1002/ejsp.309 
83.  
Reynolds-Tylus T, Bigsby E, Quick BL. A comparison of three approaches for measuring negative cognitions for 
psychological reactance. Communication Methods and Measures. Published online September 2, 2020:Published 
online ahead of print. doi:10.1080/19312458.2020.1810647 
84.  
Cidav Z, Mandell D, Pyne J, Beidas R, Curran G, Marcus S. A pragmatic method for costing implementation strategies 
using time-driven activity-based costing. Implementation Science. 2020;15(1):28. doi:10.1186/s13012-020-00993-1 
85.  
Marsh DR, Schroeder DG, Dearden KA, Sternin J, Sternin M. The power of positive deviance. BMJ. 
2004;329(7475):1177-1179. doi:10.1136/bmj.329.7475.1177 
86.  
Lawton R, Taylor N, Clay-Williams R, Braithwaite J. Positive deviance: a different approach to achieving patient safety. 
BMJ Qual Saf. 2014;23(11):880-883. doi:10.1136/bmjqs-2014-003115 
87.  
Krull JL, MacKinnon DP. Multilevel modeling of individual and group level mediated effects. Multivariate Behavioral 
Research. 2001;36(2):249-277. doi:10.1207/S15327906MBR3602_06 
88.  
Pituch KA, Murphy DL, Tate RL. Three-level models for indirect effects in school- and class-randomized experiments in 
education. The Journal of Experimental Education. 2009;78(1):60-95. doi:10.1080/00220970903224685 
89.  
Zhang Z, Zyphur MJ, Preacher KJ. Testing multilevel mediation using hierarchical linear models: problems and 
solutions. Organizational Research Methods. 2009;12(4):695-719. doi:10.1177/1094428108327450 
90.  
Williams NJ, Glisson C, Hemmelgarn A, Green P. Mechanisms of change in the ARC organizational strategy: 
increasing mental health clinicians’ EBP adoption through improved organizational culture and capacity. Adm Policy 
Ment Health. 2017;44(2):269-283. doi:10.1007/s10488-016-0742-5 
91.  
Williams NJ, Beidas RS. Navigating the storm: how proficient organizational culture promotes clinician retention in the 
shift to evidence-based practice. PLoS One. 2018;13(12):e0209745. doi:10.1371/journal.pone.0209745 
92.  
Glisson C, Williams NJ, Hemmelgarn A, Proctor E, Green P. Aligning organizational priorities with ARC to improve 
youth mental health service outcomes. J Consult Clin Psychol. 2016;84(8):713-725. doi:10.1037/ccp0000107 

 
 
 
44 
 
 93.  
MacKinnon DP, Lockwood CM, Hoffman JM, West SG, Sheets V. A comparison of methods to test mediation and 
other intervening variable effects. Psychological Methods. 2002;7(1):83-104. doi:10.1037/1082-989X.7.1.83 
94.  
MacKinnon DP, Lockwood CM, Williams J. Confidence limits for the indirect effect: distribution of the product and 
resampling methods. Multivariate Behav Res. 2004;39(1):99. doi:10.1207/s15327906mbr3901_4 
95.  
Biesanz JC, Falk CF, Savalei V. Assessing mediational models: testing and interval estimation for indirect effects. 
Multivariate Behavioral Research. 2010;45(4):661-701. doi:10.1080/00273171.2010.498292 
96.  
Hayes AF, Scharkow M. The relative trustworthiness of inferential tests of the indirect effect in statistical mediation 
analysis: does method really matter? Psychological Science. 2013;24(10):1918-1927. doi:10.1177/0956797613480187 
97.  
VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. Oxford University Press; 
2015. 
98.  
Qin, X., Hong, G. Causal mediation analysis in multi-site trials: An application of ratio-of-mediator-probability weighting 
to the Head Start Impact Study. JSM Proceedings, Social Statistics Section. Published online 2014:912-926. 
99.  
Muthén, L. K., Muthén, B. Mplus User’s Guide. 8th edition. Muthén & Muthén; 1998. 
100.  Hayes RJ, Moulton LH. Regression Analysis Based on Individual-Level Data. In: Cluster Randomised Trials. 2nd 
Edition. CRC Press; 2017:245-279. 
101.  Principi T, Coates AL, Parkin PC, Stephens D, DaSilva Z, Schuh S. Effect of oxygen desaturations on subsequent 
medical visits in infants discharged from the emergency department with bronchiolitis. JAMA Pediatr. 2016;170(6):602-
608.