Improving CarE for Community Acquired Pneumonia 1 (ICE-CAP1): Prognostic Decision Support
Phase: NA
Conditions: Pneumonia Childhood
Sponsor: Vanderbilt University Medical Center

Protocol Version #: 2.1 
 
1 
Protocol Date: October 26, 2022 
 
 
 
 
 
 
Improving CarE for Community Acquired Pneumonia (ICE-CAP) 
 
Phase 1: Antibiotic Decision Support 
Phase 2: Prognostic Decision Support 
 
 
 
Principal Investigator 
Derek J. Williams, MD, MPH 
Vanderbilt University Medical Center 
 
Co-Investigators 
Vanderbilt University Medical Center 
Kathryn M. Edwards, MD 
Carlos G. Grijalva, MD, MPH 
Yuwei Zhu, MD, MS 
Frank Harrell, Jr., PhD 
Asli Weitkamp, PhD 
Matthew Weinger, MS, MD 
Jason Slagle, PhD 
Don Arnold, MD, MPH 
 
University of Pittsburgh/Children’s Hospital of Pittsburgh 
Judith M. Martin, MD 
 Srinivasan Suresh, MD 
 Robert Hickey, MD  
Rafael Ceschin, MS 
 
 
 
 
 

Protocol Version #: 2.1 
 
2 
Protocol Date: October 26, 2022 
 
 
 
 
Table of Contents: 
 
1.0 
Background 
2.0 
Rationale and Study Phases 
3.0 
Previous Human Studies 
4.0 
Study Design and Randomization 
5.0 
Inclusion/Exclusion Criteria 
6.0 
Screening/Enrollment 
7.0 
Study Procedures 
8.0 
Statistical Considerations 
9.0 
Risks to Participants 
10.0 
Privacy/Confidentiality Issues 
11.0 
Reporting of Adverse Events or Unanticipated Problems involving Risk to Participants 
12.0 
Study Withdrawal/Discontinuation 
13.0 
Follow-Up and Record Retention 
14.0 
References 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Protocol Version #: 2.1 
 
3 
Protocol Date: October 26, 2022 
Protocol Summary: 
 
Children with pneumonia presenting to the ED at Monroe Carell Jr. Children’s Hospital at Vanderbilt or 
Children’s Hospital Pittsburgh at the University of Pittsburgh will be potentially eligible for study. During 
intervention periods, providers caring for enrolled children will be presented with a detailed decision 
support strategy that emphasizes management in accordance with national guideline recommendations. 
The anticipated study duration is 18 months for Phase 1 (P1) and 24 months for Phase 2 (P2). As this 
study does not include direct contact with enrolled subjects, there is no anticipated follow up. 
 
Protocol Versions 
Version 
Date of Approval 
Major Changes from Prior Version 
1.0 
Jul 15, 2018 
- 
2.0 
Feb 8, 2019 
Update study site; minor edits for clarity  
2.1 
Oct 26, 2022 
Update study design designation  
  
 
 
1.0 
Background 
 
Pneumonia is the most common serious infection in childhood. In the United States (US), pneumonia 
accounts for 1-4% of all emergency department (ED) visits in children (3-28 per 1,000 US children per 
year)1 and ranks among the top 3 reasons for pediatric hospitalization with >100,000 hospitalizations per 
year (15-22 per 100,000 US children per year)2-5. Pneumonia also accounts for more days of antibiotic use 
in US children’s hospitals than any other condition.6   
 
Safely reducing inappropriate antibiotic use is critical to slow the progression of antimicrobial resistance, 
and childhood pneumonia is a key area where substantial improvements can be made. In the 2011 
Pediatric Infectious Diseases Society/Infectious Diseases Society of America (PIDS/IDSA) pneumonia 
guideline, appropriate use of antibiotics was an important area of emphasis.7 For presumed bacterial 
pneumonia, recommendations emphasize the use of a single, narrow-spectrum antibiotic (i.e. amoxicillin 
or ampicillin). In children <6 years of age treated in the outpatient setting, the guideline recommended 
considering withholding antibiotics, recognizing that pneumonia in this population is most often caused 
by viruses. Both of these recommendations were graded as strong and supported by high-quality 
evidence.8-10 Nonetheless, in a large database study we conducted just prior to release of the 2011 
guideline, use of broad-spectrum antibiotics was very common among children hospitalized with 
pneumonia, with substantial differences in antibiotic selection patterns among the various hospitals.11 In 
that study, use of narrow-spectrum ampicillin was rare (<5%). High rates of broad-spectrum antibiotic use 
were also noted in a study of children with pneumonia treated and released from US EDs, with <30% of 
children receiving narrow-spectrum therapy.12 
 
Emergency care for childhood pneumonia, including hospitalization rates, varies widely across the nation. 
A study examining hospital admission rates at 35 US children’s hospitals from 2009-12 showed marked 
differences in severity-adjusted pneumonia hospital admission rates (median 31%; range 19-69%)13. 
Extending these findings, Florin et al. detected great hospital-level variation in the use of diagnostic tests 

Protocol Version #: 2.1 
 
4 
Protocol Date: October 26, 2022 
for children presenting to the ED with pneumonia.14 They reported that high test-utilizing hospitals had a 
nearly 2-fold greater odds of hospitalization compared with low test-utilizing hospitals.14 Using the same 
database as Florin et al., we previously reported similar institutional variation in diagnostic test utilization 
in children hospitalized with pneumonia. The range across hospitals for even the most common diagnostic 
tests was very wide with some hospitals performing specific tests in <30% of children and others 
performing those same tests in >75% of children. We also demonstrated an association between high test-
utilizing hospitals and increased average hospital length of stay.11  
 
Provider preferences and inaccurate risk perceptions contribute to differences in hospitalization rates. 
Within the Intermountain Healthcare System in Utah, Dean et al. exposed large differences in admission 
rates (range 38-79%) among 18 individual ED providers providing care for >2,000 adults with 
pneumonia.15 Differences were not explained by patient characteristics or illness severity and higher rates 
of hospitalization did not reduce hospital readmissions or mortality. In another multicenter study of 472 
adults with pneumonia at <4% risk of 30-day mortality estimated using objective severity scores, 
providers overestimated the risk of mortality in 5% of outpatients (range across institutions 0-12%) and 
41% of inpatients (range across institutions 36-48%).16 These studies suggest that risk perceptions are 
often inaccurate, and potentially lead to unnecessary or prolonged hospitalizations and intensive therapies. 
Similar studies have not been performed in children because no valid prognostic tools exist to reliably 
predict pediatric pneumonia severity.  
 
2.0 
Rationale and Study Phases 
 
Extensive variation in both antibiotic use and hospitalization decisions is evident among clinicians caring 
for children with pneumonia, with high potential for avoidable harm. In our prior study, one-third of 
children were hospitalized for less than 48 hours and nearly 10% less than 24 hours. Some of these 
hospitalizations were likely unnecessary. Most of these children also received broad-spectrum antibiotic 
therapy instead of the narrow-spectrum agents recommended by the national guideline. Conversely, one-
third of children admitted to intensive care were initially managed on a general ward—earlier transfer for 
more intensive therapy may have improved outcomes for some of these children. New strategies to inform 
decision making are needed, and the combination of risk stratification using objective tools and clinical 
decision support in the ED setting are innovative and promising approaches to achieve this goal. 
 
Phase 1: To test the hypothesis that electronic antibiotic decision support increases guideline-concordant 
antibiotic use compared with usual care in the ED. 
 
The primary outcome is the proportion of children exclusively receiving guideline-concordant 1st line 
antibiotic therapy during the first 24 hours of care (Table 1). Secondary outcomes include exclusive use 
of concordant antibiotic therapy for the entire episode, any use of concordant antibiotic therapy during the 
first 24 hours of care and for the entire episode, and ED revisits and hospitalizations within 72 hours and 
7 days of the index discharge.   
 
Phase 2: To test the hypothesis that the delivery of severity information generated by our prognostic tool 
leads to more appropriate site of care disposition compared to usual care. 
 

Protocol Version #: 2.1 
 
5 
Protocol Date: October 26, 2022 
The primary outcome is appropriate site of care disposition (Table 2). Surveillance for subsequent ED 
visits and hospitalizations to our institutions as well as escalation to higher levels of care will be captured 
within the EHR data extracted by the decision support application. Secondary efficacy and safety 
outcomes include the overall site of care disposition (ED discharge, ward, ICU), and ED revisits and 
hospitalizations within 72 hours and 7 days of the index discharge. 
 
In these studies, conducted at two experienced academic centers, we will implement and evaluate an 
EHR-based clinical decision support application to promote antibiotic use in concordance with the 2011 
PIDS/IDSA guideline in a pragmatic, cluster-randomized trial (P1). Next, we will incorporate risk 
stratification using a previously developed prognostic tool into a second EHR-based decision support 
application, testing its impact on site of care disposition compared to usual care in a pragmatic 
randomized trial (P2). In both studies, all subjects will receive standard of care. Decisions regarding 
management, including antibiotic selection and site of care, will be at the discretion of the treating 
provider and will not be restricted or altered in any way. Thus, this study poses no greater than minimal 
risks to participants. Due to the nature of the research, we will seek a waiver of informed consent as has 
been done in similar pragmatic studies at our institutions.17-21 
 
3.0 
Previous Studies 
 
From 2010 to 2012, our research team at Vanderbilt collaborated with two other sites to conduct the 
Centers for Disease Control and Prevention (CDC) funded Etiology of Pneumonia in the Community 
(EPIC) Study, a prospective, population-based surveillance study of community-acquired pneumonia 
hospitalizations in children and adults.2,22 We enrolled >2600 children, making EPIC the largest 
prospective study of pneumonia hospitalizations ever conducted in US children. Over 70% of children 
enrolled in EPIC had a viral pathogen detected whereas bacterial pathogens were identified in only 15% 
of children. Nevertheless, nearly 90% of children received antibiotics. In most cases, antibiotic treatment 
was discordant with the national guideline and broad-spectrum antibiotic use was common.23 We have 
also evaluated the impact of the 2011 PIDS/IDSA guideline on antibiotic selection for children with 
pneumonia enrolled in EPIC, demonstrating only a modest reduction in the use of broad-spectrum 
antibiotics following publication of the guideline. Guideline-concordant use of narrow-spectrum 
aminopenicillins remained <20% by the end of the study, indicating a clear opportunity for further 
improvement.23  
 
We also used data from the EPIC study to develop a prognostic tool for assessing pneumonia disease 
severity in children. Prognostic tools use statistical modeling to combine inputs, such as demographics, 
clinical variables, and results of diagnostic tests, to produce reliable and unbiased risk estimates for a 
specified outcome. For our prognostic tool, we focused on acute in-hospital outcomes, organized into an 
ordinal severity scale consisting of 3 levels: severe, moderate, and mild. Our modeling strategy, ordinal 
logistic regression, offers increased statistical power and precision compared to binary/multinomial 
logistic regression, and the resulting output allows one to simultaneously estimate probabilities for each 
level on the outcome scale or in combination (e.g., probability of moderate to severe and probability of 
severe outcomes). Predictor variables for our prognostic tool were selected a priori and also informed by 
clinical expertise and literature review. We focused on objectively defined variables with strong 
hypothesized associations with the outcome that could be rapidly ascertained at presentation in the 

Protocol Version #: 2.1 
 
6 
Protocol Date: October 26, 2022 
clinical setting. This process resulted in a pool of 20 predictors. We also created a separate, more 
parsimonious model restricted to 9 predictors commonly available within electronic health records 
(EHR).  
 
Both models accurately identified risk for moderate or severe pneumonia, indicating strong potential to 
improve risk stratification. Model discrimination, which measures the degree of concordance between 
observed and predicted outcomes, was excellent with a c-index ranging from 0.78-0.81. Differences in 
model fit and quality were considered negligible, and results from the internal bootstrap validation were 
comparable to the primary models. In the full model, PaO2:FiO2 ratio (PF ratio, estimated from the 
SaO2FiO2 ratio),24 age, heart and respiratory rates, altered mental status, temperature, chest indrawing, 
infiltrate pattern, and systolic blood pressure were the strongest predictors, contributing 96% of the 
explainable outcome variation. The EHR model retained most of these predictors (omitting altered mental 
status, chest indrawing, and infiltrate pattern). Nonetheless, predictive performance of the EHR model 
equaled that of the full model. For the full model, the median predicted risk for moderate or severe 
pneumonia was 18% (IQR 9-32%) and for severe pneumonia 5% (3-11%). 
 
Prognostic tools used in adults with pneumonia improve risk stratification and clinical decision making, 
but these tools are underutilized, indicating a need for improved implementation strategies. A number of 
prognostic tools have been developed to assist providers caring for adults with pneumonia,25-31 and most 
have undergone external validation.26,32-42 The two best studied tools are the PSI and CURB-65,30,31 which 
were designed to predict 30-day mortality. Newer tools (e.g., SMART-COP and SCAP)26,27 have also 
been developed to predict acute in-hospital outcomes, including mechanical ventilation and shock. These 
tools have been used to safely increase the number of low-risk patients treated in the outpatient setting 
and improve guideline-concordant antibiotic management.43-47 As a result, their use in the management of 
adults with pneumonia in outpatient and inpatient settings is strongly recommended.25 Nonetheless, 
several studies indicate these tools are often applied incorrectly or not used at all.15,48-50 Major barriers to 
clinical use include tool complexity (inability to recall parameters and inaccurate scoring or 
interpretation) and disruptions to provider workflow (suspension of patient care to calculate risk by hand 
or use of an electronic application outside of routine care).51 Failure to validate new tools and assess their 
impact in new populations are also major impediments to clinical use. Each of these barriers must be 
carefully addressed to ensure effective implementation of new prognostic tools. 
 
Effective decision support facilitates the adoption of new evidence into practice and helps standardize 
management to improve care and optimize outcomes. Clinical decision support is intended to directly 
impact provider decision-making to improve the quality, safety, effectiveness, and efficiency of care by 
steering behavior toward evidence-based care and away from unnecessary or unproven practices. The 
availability of comprehensive EHRs has allowed for the development of increasingly complex decision 
support applications, including risk stratification using real-time electronic capture of patient 
demographics, physiologic data, and diagnostic testing results.17,52 In a randomized trial conducted in 
primary care settings comparing decision support integrated with 2 prognostic tools vs. usual care without 
decision support, McGinn and colleagues showed a 25% reduction in antibiotic use overall and a 50% 
reduction in broad-spectrum antibiotic use in adults with possible bacterial pharyngitis or pneumonia 
when using the decision support application.52 The application was triggered on chief complaint and 
incorporated a risk calculator, which used automated and user-supplied inputs to identify risk for bacterial 

Protocol Version #: 2.1 
 
7 
Protocol Date: October 26, 2022 
pharyngitis or pneumonia, along with evidence-based treatment guidelines and order sets. In another 
study of adults with pneumonia presenting for emergency care within the Intermountain Healthcare 
System, Dean et al. showed increased guideline concordant care and reduced mortality in four different 
EDs using clinical decision support that included CURB-65 severity assessment and tailored 
recommendations for site of care, diagnostic testing, and antibiotic use compared to 3 usual care EDs.17 
These successful examples highlight the potential for EHR-based decision support to improve care by 
providing useful, evidence-based guidance at the point of care as management decisions are being made. 
 
Both Vanderbilt and the University of Pittsburgh have extensive experience leveraging EHR data for 
decision support and research purposes. As an example of such capabilities, in the EPIC study, a decision 
support tool used in adults with pneumonia was adapted for use as a screening tool.17 The tool used 
natural language processing to identify concepts suggestive of pneumonia in radiology “wet” reads in 
real-time and required a provider-confirmation of a pneumonia diagnosis. For EPIC, we adapted this tool 
to deliver electronic alerts to our enrollment teams. Importantly, a similar tool was used to identify 
pneumonia in the study by Dean et al. described in the preceding paragraph. A second example highly 
relevant to this proposal was developed by our co-investigator, Dr. Weitkamp, and involves antibiotic 
decision support to reduce inappropriate use of antibiotics in critically ill neonates. Importantly, the 
algorithm was followed in >70% of infants and led to significant reductions in antibiotic exposure.53  
Additionally, our co-investigators, Drs. Weinger and Slagle, experts in human factors engineering and 
user-interface design of health information technology, possess extensive experience designing and 
evaluating decision support applications that maximize utility and minimize potential for human error.54-63 
These examples highlight that the investigators at Vanderbilt and the University of Pittsburgh possess the 
necessary skills, expertise, and institutional resources to develop and implement effective decision 
support applications.  
 
4.0 
Study Design and Randomization 
 
4.1 
Study Design and Randomization (P1, EHR-based antibiotic decision support) 
Effectiveness of the EHR-based antibiotic decision support application for promoting guideline-
concordant antibiotic prescribing in children presenting for emergency care will be evaluated in a 
pragmatic, cluster-randomized study conducted over a period of 18 months that includes two respiratory 
seasons. Our institutions possess demonstrated experience in the execution of these pragmatic designs. 17-
21 Assignments to control or intervention arms will occur in 4-week periods using a randomly determined 
sequence within each hospital. To ensure balanced representation of each arm in periods of both low and 
high pneumonia prevalence, randomization will occur in 3 permuted blocks (size=6).  The antibiotic 
decision support application will be provided to those randomized to the intervention arm, whereas the 
control arm will receive usual care. Due to the nature of the intervention, blinding of treating providers 
will not be possible. Importantly, since this is a trial focused on provider behavior, all children will 
receive standard of care management. All treatment decisions will be made by the clinical providers and 
will not be restricted or altered in any way. 
 
4.2 
Study Design and Randomization (P2, EHR-based risk stratification)  
We will conduct a randomized controlled trial comparing our prognostic tool (intervention arm) to usual 
care (control arm) over a period of 24 months. Randomization will occur at the patient level. Allocation to 

Protocol Version #: 2.1 
 
8 
Protocol Date: October 26, 2022 
intervention or control will be based on medical record number (even vs. odd) or similar strategy and will 
be assigned automatically once a provider confirms the diagnosis of pneumonia. Importantly, all standard 
of care treatment options will be available and decision-making will not be restricted in any way in either 
group. A similar randomized trial that predicts readmission risk among all hospitalized adults using EHR 
inputs is currently ongoing at Vanderbilt. In that study, every adult admitted to Vanderbilt University 
Hospital is enrolled (under waiver of consent). Readmission risk is calculated automatically for all 
patients, but only displayed for a random half to determine if providing this additional information 
improves outcomes compared to usual care. 
 
5.0 
Inclusion/Exclusion Criteria (Phases 1 and 2) 
 
Inclusion Criteria 
1. Six months to <18 years of age 
2. Radiographic evidence of pneumonia in ED  
3. Provider-confirmed diagnosis of pneumonia  
 
Exclusion Criteria (Provider query and/or automated data abstraction) 
1. Children with tracheostomy, cystic fibrosis, immunosuppression 
2. Inter-hospital transfers 
3. Hospitalization within preceding 7 days 
4. Previously enrolled within preceding 28 days 
5. Provider preference for any reason 
 
6.0 
Screening and Enrollment 
 
6.1 
Screening and Enrollment (P1) 
Children with pneumonia presenting to the ED at Monroe Carell Jr. Children’s Hospital at Vanderbilt or 
Children’s Hospital of Pittsburgh will be potentially eligible for study. The pneumonia radiology alert 
tool will be used to identify potentially eligible children in a systematic and efficient manner using natural 
language processing (NLP). Those who screen positive according to the radiology NLP algorithm will 
trigger a Best Practice Alert (BPA) or similar within the EHR which queries the provider to confirm the 
diagnosis of pneumonia and assesses for study inclusion/exclusion criteria. In instances where a chest 
radiograph is not captured via the NLP algorithm (e.g. a radiograph obtained at an outside facility), an 
alternative pathway is available to trigger the subsequent BPAs leading to enrollment. Due to the nature 
of the research, we will seek a waiver of informed consent as has been done in similar pragmatic studies 
at our institutions.17-21 Thus, at this point, eligible children will be considered enrolled. During 
intervention periods, providers caring for enrolled children will be presented with a detailed decision 
support strategy that emphasizes management in accordance with national guideline recommendations. 
During control periods, no additional decision support will be provided.   
 
6.2 
Screening and Enrollment (P2) 
Screening and enrollment procedures for P2 will be identical to P1. For children randomized to the 
intervention arm, providers will be presented with prognostic information regarding risk for severe 
disease outcomes via an EHR-based decision support strategy that emphasizes management in accordance 

Protocol Version #: 2.1 
 
9 
Protocol Date: October 26, 2022 
with national guideline recommendations. During control periods, no additional decision support will be 
provided.   
 
7.0 
Study Procedures 
 
7.1 
EHR-based Antibiotic Decision Support (P1) 
For enrolled subjects assigned to the decision support arm, providers will receive antibiotic 
recommendations in accordance with the 2011 PIDS/IDSA guideline, tailored to site of care and illness 
severity. For instance, in young children with mild illness being discharged from the ED, the support tool 
will offer guidance regarding when a strategy of watchful waiting might be favored over empiric 
amoxicillin (e.g., fully immunized toddler with upper respiratory tract symptoms). Similarly, in those 
being admitted to a general ward, the tool will advise consideration of oral therapy in children with good 
oral intake, or when to consider adjunctive therapy with a macrolide or anti-staphylococcal antibiotic. 
Importantly, the tool will offer treatment recommendations only and will not proscribe a specific 
treatment plan. If a provider chooses to prescribe therapy not in accordance with the national guideline, 
the tool will query providers regarding reasons for discrepant prescribing for later analysis.  
 
7.2 
Outcomes (P1) 
The primary outcome is the 
proportion of children exclusively 
receiving guideline-concordant 1st 
line antibiotic therapy during the 
first 24 hours of care (Table 1). For 
patients discharged from the ED, 
this will include antibiotics received 
in the ED as well as antibiotic 
prescriptions provided at discharge. For hospitalized patients, this will include therapy received during the 
first 24 hours of care beginning with ED triage. A patient following treatment according to table 1 will be 
considered as concordant, whereas deviations from these treatments will be considered discordant.  
 
Secondary outcomes include exclusive use of concordant antibiotic therapy for the entire episode, any use 
of concordant antibiotic therapy during the first 24 hours of care and for the entire episode, and ED 
revisits and hospitalizations within 72 hours and 7 days of the index discharge.  Index visit is defined as 
the enrollment ED visit or hospitalization through discharge.   
 
Exploratory outcomes include time to first antibiotic, lack of antibiotics during the first 24 hours of care 
and for the entire episode, and changes in antibiotics after 72 hours, and death, and for hospitalized 
children, delayed ICU transfer, defined as >24 hours following hospitalization.  
 
7.3 
Data Collection/Management (P1) 
Data collection will be triggered by the pneumonia BPA and will be acquired directly from the EHR and 
stored securely for later analysis. Prior to study initiation, support analysts will conduct detailed 
requirement explorations to identify necessary data elements and associated data flows, validate accuracy, 
and evaluate for missing data. Data to be collected includes decision support usage statistics, visit details 
(e.g., admission and discharge date and time, site of care, ED revisits or hospital readmissions), patient 
Table 1. Concordant Use Definitions for 1st Line Antibiotics1 
Disposition 
<6 Years of Age 
≥6 Years of Age 
Home care 
 
No antibiotics or amoxicillin Amoxicillin 
Inpatient, ward 
 
Ampicillin/amoxicillin 
Ampicillin/amoxicillin 
Inpatient, ICU or 
complicated disease2 
Ampicillin or ceftriaxone 
+/- anti-staphylococcal 
+/- macrolide 
Ampicillin or ceftriaxone 
+/- anti-staphylococcal 
+/- macrolide 
ICU, intensive care unit 1Unless specific pathogen identified or known drug 
allergy 2eg, empyema 

Protocol Version #: 2.1 
 
10 
Protocol Date: October 26, 2022 
demographics (e.g., age, sex, race, insurance), medication orders and prescriptions (dose, frequency, 
route, and duration), laboratory testing and results, radiologic studies, and data elements from the decision 
support applications.  
 
7.4 
EHR-based Prognostic Decision Support (P2) 
For enrolled subjects assigned to the decision support arm, providers will receive prognostic information 
derived using our previously validated and best performing model.  The decision support application will 
automatically calculate predicted risk for moderate (intensive care) and severe (respiratory failure or 
shock) outcomes using the parameters derived from the prognostic tool’s regression equation. Outcome 
probabilities will be integrated into the decision support application and displayed within the EHR. The 
support application will also report any objective criteria necessitating hospitalization (e.g., hypoxia) or 
potential intensive care (e.g., hypotension, SpO2<92% despite FiO2≥ 50%) to further inform site of care 
decisions. The application will also query providers regarding factors influencing site of care decisions to 
inform analyses describing discrepancies between risk assessments and decision-making (e.g., 
hospitalization of low risk children) and provider variation in risk tolerance. Importantly, the tool will 
offer prognostic 
information only and will 
not proscribe a specific 
site of care or 
management plan. All site 
of care decisions will 
remain at the discretion of 
the treating provider. 
 
7.5 
Outcomes (P2) 
The primary outcome is appropriate site of care disposition (Table 2). Surveillance for subsequent ED 
visits and hospitalizations to our institutions as well as escalation to higher levels of care will be captured 
within the EHR data extracted by the decision support application. To adjudicate the appropriateness of 
brief hospitalizations and ICU stays (<24 hours), an investigator in both Nashville and Pittsburgh will 
review all hospitalizations meeting these criteria, blinded to intervention assignment.  
 
Secondary efficacy and safety outcomes include the overall site of care disposition (ED discharge, ward, 
ICU), and ED revisits and hospitalizations within 72 hours and 7 days of the index discharge. 
 
Exploratory outcomes include death, and for those hospitalized at the index visit, hospital length of stay. 
 
7.6 
Data Collection/Management (P2) 
See Section 7.3 
 
8.0 
Statistical Considerations  
 
8.1 
Power and Sample Size (P1) 
Based on ED and hospitalization data from our hospitals during the conduct of the EPIC study, we 
anticipate nearly 2000 total children with pneumonia will be evaluated annually during the conduct of 
Table 2. Appropriate Site of Care Definitions 
Disposition 
Appropriate if: 
Home care 
No subsequent hospitalization within 24h 
Inpatient, ward Hospital length of stay ≥ 24h; or hospital length of stay < 24h but 
objective criteria for admission present (eg, need for supplemental 
oxygen); and no subsequent transfer to ICU within 24h 
Inpatient, ICU ICU length of stay ≥ 24h; or ICU length of stay < 24h but objective 
criteria for ICU admission present (eg, respiratory failure) 
h, hours; ICU, intensive care unit 

Protocol Version #: 2.1 
 
11 
Protocol Date: October 26, 2022 
these studies. Assuming conservatively that data for 750 children are enrolled in each arm, and 30% 
guideline-concordant antibiotic use in the usual care arm (based on our prior study of antibiotic use in the 
EPIC cohort)23, we will be able to detect an absolute difference of ≥6.8% in guideline-concordant 
prescribing at an alpha level of 0.05 with 80% power. Effective sample size may be reduced due to the 
cluster randomized nature of the study (a conservative estimate for effective sample size would be around 
500 children in each arm). 
 
8.2 
Analysis (P1) 
Analyzing data from cluster randomized trials requires accounting for correlations within clusters and 
time periods. As outlined by Turner et al.,21 we will use a hierarchical and mixed effects logistic 
regression model, a complex analytic approach which offers flexibility regarding model assumptions and 
incorporation of cluster-level and individual-level covariates. We will also use a weighted, cluster-level 
logistic regression to compare guideline-concordant antibiotic prescribing between groups. In this 
analysis, data for each cluster are collapsed into a single summary measure, and the summary measures 
are then analyzed as if they were raw data, facilitating simple data reporting and interpretation. 
21Important baseline covariates (e.g., age, sex, race, site of care, frequency of bacteremia and other 
pneumonia-related complications) will also be assessed and included as model covariates. Similar 
analyses will be conducted for the proposed secondary and exploratory outcomes using hierarchical and 
mixed effects logistic regression models for binary outcomes and frailty models for time to first antibiotic.  
 
An important consideration in our analysis is the possibility of contamination effects in the usual care arm 
related to changes in provider behavior due to past exposures and awareness of the intervention. Such 
contamination would be expected to bias toward the null hypothesis. We posit that such effects will be of 
minor concern since 1) guideline-concordant prescribing at our institutions remains low despite the 
presence of both national and local practice guidelines and order sets for several years; 2) the ED is a 
busy environment with many providers (e.g., 40+ ED faculty plus mid-level providers and 
fellows/residents at Vanderbilt alone), such that repeated exposure to the intervention by any one provider 
would be infrequent, particularly early during the study period and during times of low pneumonia 
prevalence; and 3) resident physicians, the providers responsible for the majority of order entry and thus 
directly exposed to the intervention, rotate on a monthly schedule and only work in the ED once each 
year. We have designed our monthly randomization periods around these important considerations. 
Nevertheless, we acknowledge the potential for contamination, and we will formally evaluate for such 
effects in a secondary analysis by modeling the monthly proportions of guideline-concordant prescribing 
in each arm over the duration of the study period. We will then compare prescribing trends in both arms 
with a historical trend estimated from monthly proportions of guideline-concordant prescribing in the year 
prior to study initiation. By comparing prescribing trends in each arm to this historical trend, we will 
estimate both the degree of potential contamination as well as the independent impact of the intervention. 
 
8.3 
Potential Limitations and Alternative Approaches (P1) 
We do not anticipate major difficulties designing or implementing our antibiotic decision support 
application. If challenges arise, our experienced team possesses the necessary expertise, past experience, 
and institutional resources to address and overcome them. Design challenges will be addressed by our 
usability and decision support experts to troubleshoot and develop effective solutions prior to and 
following implementation. Alternative analytic approaches were considered, including quasi-experimental 

Protocol Version #: 2.1 
 
12 
Protocol Date: October 26, 2022 
step wedge designs and randomization at the provider or patient level. The chosen design was selected 
since it is less subject to temporal bias (compared to the step wedge approach) and minimizes problems 
with contamination in the usual care arm (compared to provider or patient-level randomization) as 
outlined above. In addition to the analyses outlined above, if we fail to demonstrate meaningful outcome 
differences between the two arms, we will conduct exploratory analyses to elucidate reasons for lack of 
intervention effectiveness. 
 
8.4 
Power and Sample Size (P2) 
As outlined in P1, we anticipate nearly 2000 children with pneumonia will be evaluated annually during 
the conduct of these studies. Assuming conservatively that data for 2000 children (1000 in each arm) are 
captured during the two-year study period, and 90% appropriate site of care disposition in the usual care 
arm (based on data from the EPIC cohort)23, we will be able to detect an absolute difference of ≥3.4% in 
appropriate care disposition at an alpha level of 0.05 with 80% power. 
 
8.5 
Analysis (P2) 
Logistic regression models will be performed to compare appropriate site of care disposition between the 
two study arms, controlling for important baseline covariates (e.g., age, season, provider type) to 
maximize power. Robust standard errors will be calculated to account for hospital clustering effects. We 
will also perform analyses to assess for interactions between two study arms and predicted risk for severe 
outcomes, age, provider type, and initial site of care, to identify subgroups in whom decision support may 
be most useful. Similar analyses will be conducted for the proposed secondary and exploratory outcomes 
along with cox regression model for hospital length of stay outcome. 
 
8.6 
Potential Limitations and Alternative Approaches (P2) 
In contrast to our studies of antibiotic decision support, we chose to randomize at the individual level 
rather than monthly periods since potential for contamination effects is minimal (i.e. risk estimates are 
unique to each individual’s set of circumstances and linked to a complex algebraic equation). Although 
we hypothesize that supplying providers with objective prognostic information will improve care 
decisions, this information could lead to unintended outcomes, such as increased hospitalizations in low 
risk children. This would be an important finding in itself, as it contrasts with prior studies in adults 
highlighting the usefulness of adult pneumonia severity tools, and provides valuable data regarding how 
risk information is interpreted and applied. 
 
9.0 
Risks to Participants 
 
All study subjects will receive standard of care. All decisions regarding management, including antibiotic 
use and site of care disposition, will be at the discretion of the treating provider and will not be restricted 
or altered in any way by the study. This study does not involve an investigational intervention given or 
administered directly to patients (i.e. no treatment assignments will be made). Therefore, this study poses 
no greater than minimal risk to study participants. Vanderbilt is at the forefront of these pragmatic, 
minimal risk clinical trial designs, and our study team benefits greatly from the experiences and expertise 
of our colleagues who have completed similar studies under waiver of informed consent. The potential 
benefit of this study would come through more effective therapy.  
 

Protocol Version #: 2.1 
 
13 
Protocol Date: October 26, 2022 
The study coordinator, with PI oversight, is responsible for ensuring protocol compliance, data integrity, 
and participant safety. This protocol presents minimal risks to participants and adverse events or other 
problems are not anticipated. In the unlikely event that such events occur, unanticipated problems 
involving risks to subjects or others will be reported immediately to the PI and responsible IRBs, and in 
writing within 5 days to the IRB and any appropriate funding and regulatory agencies. The PI will apprise 
fellow investigators and study personnel of all adverse events that occur during the conduct of this 
research project through regular study meetings. Annual reports will be submitted to the responsible IRBs 
summarizing study progress, adverse events, complaints about the research or withdrawals, and any 
protocol violations.  
 
Although the study poses no greater than minimal risk to human subjects, we have appointed an 
independent data and safety monitoring board (DSMB) due to the inclusion of children and the multi-
center nature of the study. The DSMB is comprised of three individuals with expertise relevant to the 
study, including biostatistics, and prior experience serving in this capacity. The DSMB will be 
independent of the study team and will conduct formal data and safety evaluations at least every six 
months. During these meetings, the DSMB will assess study progress, including subject accrual and 
retention, data quality, and subject safety in a confidential manner. The DSMB will make 
recommendations regarding whether the study should continue unchanged or require 
modification/amendment. Based on the findings of the data and safety monitor’s reports, the PI, 
associated IRBs, and any other appropriate funding or regulatory agencies have the authority to stop or 
suspend the study or require modifications. 
 
10.0 
Privacy/Confidentiality Issues 
 
Privacy and confidentiality is of the highest priority, and all efforts will be made to keep personal 
information and research records private throughout these studies.  Records and documents pertaining to 
the conduct of this study will be kept in HIPPA-compliant, secure, password protected databases or in 
locked files accessible only by a limited number of study personnel. All patient data will be coded using a 
unique study ID. No PHI will be shared outside of the key study personnel at each institution and no PHI 
will be shared between institutions. The information from the research study may be published using 
aggregate results; however, no individual subject will be identified. 
 
11.0 
Reporting of Adverse Events or Unanticipated Problems involving Risk to Participants  
 
The PI is responsible for monitoring the data, assuring protocol compliance, and conducting safety 
reviews. Adverse events or other problems are not anticipated. In the unlikely event that such events 
occur, unanticipated problems involving risks to subjects or others will be reported immediately to the PI 
and responsible IRBs, and in writing within 5 days to the IRB and any appropriate funding and regulatory 
agencies. The PI will apprise fellow investigators and study personnel of all adverse events that occur 
during the conduct of this research project through regular study meetings. Annual reports will be 
submitted to the responsible IRBs summarizing study progress, adverse events, complaints about the 
research or withdrawals, and any protocol violations. 
 
12.0 
Study Withdrawal/Discontinuation 

Protocol Version #: 2.1 
 
14 
Protocol Date: October 26, 2022 
 
The investigators may choose to withdraw a subject if it is judged to be in the participant’s best interest, 
or if the subject no longer meets eligibility criteria (see Section 5.0). Due to the observational nature of 
the study, there are no formal interim analyses or planned halting criteria. 
 
13.0 
Follow-Up and Record Retention 
 
The anticipated study duration is 24 months for P1 and 24 months for P2. As this study does not include 
direct contact with enrolled subjects, there is no anticipated follow up. Records will be maintained in 
accordance with IRB, institutional, and sponsor policies. 
 
 

Protocol Version #: 2.1 
 
15 
Protocol Date: October 26, 2022 
14.0 
References 
 
1. 
Self WH, Grijalva CG, Zhu Y, et al. Rates of emergency department visits due to 
pneumonia in the United States, July 2006-June 2009. Acad Emerg Med. 2013;20(9):957-
960. 
2. 
Jain S, Williams DJ, Arnold SR, et al. Community-acquired pneumonia requiring 
hospitalization among U.S. children. N Engl J Med. 2015;372(9):835-845. 
3. 
Lee GE, Lorch SA, Sheffler-Collins S, Kronman MP, Shah SS. National hospitalization 
trends for pediatric pneumonia and associated complications. Pediatrics. 
2010;126(2):204-213. 
4. 
AHRQ. National Estimates on Use of Hospitals by Children from the HCUP Kids' 
Inpatient Database (KID). 2012; http://hcupnet.ahrq.gov/. Accessed Jan 12, 2014. 
5. 
Keren R, Luan X, Localio R, et al. Prioritization of comparative effectiveness research 
topics in hospital pediatrics. Archives of pediatrics & adolescent medicine. 
2012;166(12):1155-1164. 
6. 
Gerber JS, Kronman MP, Ross RK, et al. Identifying targets for antimicrobial 
stewardship in children's hospitals. Infect Control Hosp Epidemiol. 2013;34(12):1252-
1258. 
7. 
Bradley JS, Byington CL, Shah SS, et al. The management of community-acquired 
pneumonia in infants and children older than 3 months of age: clinical practice guidelines 
by the pediatric infectious diseases society and the infectious diseases society of america. 
Clin Infect Dis. 2011;53(7):e25-76. 
8. 
Michelow IC, Olsen K, Lozano J, et al. Epidemiology and clinical characteristics of 
community-acquired pneumonia in hospitalized children. Pediatrics. 2004;113(4):701-
707. 
9. 
Juven T, Mertsola J, Waris M, et al. Etiology of community-acquired pneumonia in 254 
hospitalized children. Pediatr Infect Dis J. 2000;19(4):293-298. 
10. 
Heiskanen-Kosma T, Korppi M, Jokinen C, et al. Etiology of childhood pneumonia: 
serologic results of a prospective, population-based study. Pediatr Infect Dis J. 
1998;17(11):986-991. 
11. 
Brogan TV, Hall M, Williams DJ, et al. Variability in processes of care and outcomes 
among children hospitalized with community-acquired pneumonia. Pediatr Infect Dis J. 
2012;31(10):1036-1041. 
12. 
Neuman MI, Shah SS, Shapiro DJ, Hersh AL. Emergency department management of 
childhood pneumonia in the United States prior to publication of national guidelines. 
Acad Emerg Med. 2013;20(3):240-246. 
13. 
Bourgeois FT, Monuteaux MC, Stack AM, Neuman MI. Variation in emergency 
department admission rates in US children's hospitals. Pediatrics. 2014;134(3):539-545. 
14. 
Florin TA, French B, Zorc JJ, Alpern ER, Shah SS. Variation in emergency department 
diagnostic testing and disposition outcomes in pneumonia. Pediatrics. 2013;132(2):237-
244. 
15. 
Dean NC, Jones JP, Aronsky D, et al. Hospital admission decision for patients with 
community-acquired pneumonia: variability among physicians in an emergency 
department. Ann Emerg Med. 2012;59(1):35-41. 

Protocol Version #: 2.1 
 
16 
Protocol Date: October 26, 2022 
16. 
Fine MJ, Hough LJ, Medsger AR, et al. The hospital admission decision for patients with 
community-acquired pneumonia. Results from the pneumonia Patient Outcomes 
Research Team cohort study. Arch Intern Med. 1997;157(1):36-44. 
17. 
Dean NC, Jones BE, Jones JP, et al. Impact of an Electronic Clinical Decision Support 
Tool for Emergency Department Patients With Pneumonia. Ann Emerg Med. 2015. 
18. 
Noto MJ, Domenico HJ, Byrne DW, et al. Chlorhexidine bathing and health care-
associated infections: a randomized clinical trial. JAMA. 2015;313(4):369-378. 
19. 
Semler MW, Weavind L, Hooper MH, et al. An Electronic Tool for the Evaluation and 
Treatment of Sepsis in the ICU: A Randomized Controlled Trial. Crit Care Med. 
2015;43(8):1595-1602. 
20. 
Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic 
monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care 
Med. 2012;40(7):2096-2101. 
21. 
Semler MW, Janz DR, Lentz RJ, et al. Randomized Trial of Apneic Oxygenation during 
Endotracheal Intubation of the Critically Ill. Am J Respir Crit Care Med. 
2016;193(3):273-280. 
22. 
Jain S, Self WH, Wunderink RG, et al. Community-Acquired Pneumonia Requiring 
Hospitalization among U.S. Adults. N Engl J Med. 2015;373(5):415-427. 
23. 
Williams DJ, Edwards KM, Self WH, et al. Antibiotic Choice for Children Hospitalized 
With Pneumonia and Adherence to National Guidelines. Pediatrics. 2015;136(1):44-52. 
24. 
Pandharipande PP, Shintani AK, Hagerman HE, et al. Derivation and validation of 
Spo2/Fio2 ratio to impute for Pao2/Fio2 ratio in the respiratory component of the 
Sequential Organ Failure Assessment score. Crit Care Med. 2009;37(4):1317-1321. 
25. 
Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of 
America/American Thoracic Society consensus guidelines on the management of 
community-acquired pneumonia in adults. Clin Infect Dis. 2007;44 Suppl 2:S27-72. 
26. 
Charles PG, Wolfe R, Whitby M, et al. SMART-COP: a tool for predicting the need for 
intensive respiratory or vasopressor support in community-acquired pneumonia. Clin 
Infect Dis. 2008;47(3):375-384. 
27. 
Espana PP, Capelastegui A, Gorordo I, et al. Development and validation of a clinical 
prediction rule for severe community-acquired pneumonia. Am J Respir Crit Care Med. 
2006;174(11):1249-1256. 
28. 
Rello J, Rodriguez A, Lisboa T, Gallego M, Lujan M, Wunderink R. PIRO score for 
community-acquired pneumonia: a new prediction rule for assessment of severity in 
intensive care unit patients with community-acquired pneumonia. Crit Care Med. 
2009;37(2):456-462. 
29. 
Renaud B, Labarere J, Coma E, et al. Risk stratification of early admission to the 
intensive care unit of patients with no major criteria of severe community-acquired 
pneumonia: development of an international prediction rule. Crit Care. 2009;13(2):R54. 
30. 
Fine MJ, Auble TE, Yealy DM, et al. A prediction rule to identify low-risk patients with 
community-acquired pneumonia. N Engl J Med. 1997;336(4):243-250. 
31. 
Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia 
severity on presentation to hospital: an international derivation and validation study. 
Thorax. 2003;58(5):377-382. 
32. 
Phua J, See KC, Chan YH, et al. Validation and clinical implications of the IDSA/ATS 
minor criteria for severe community-acquired pneumonia. Thorax. 2009;64(7):598-603. 

Protocol Version #: 2.1 
 
17 
Protocol Date: October 26, 2022 
33. 
Brown SM, Jones BE, Jephson AR, Dean NC. Validation of the Infectious Disease 
Society of America/American Thoracic Society 2007 guidelines for severe community-
acquired pneumonia. Crit Care Med. 2009;37(12):3010-3016. 
34. 
Yandiola PP, Capelastegui A, Quintana J, et al. Prospective comparison of severity scores 
for predicting clinically relevant outcomes for patients hospitalized with community-
acquired pneumonia. Chest. 2009;135(6):1572-1579. 
35. 
Marras TK, Gutierrez C, Chan CK. Applying a prediction rule to identify low-risk 
patients with community-acquired pneumonia. Chest. 2000;118(5):1339-1343. 
36. 
Liapikou A, Ferrer M, Polverino E, et al. Severe community-acquired pneumonia: 
validation of the Infectious Diseases Society of America/American Thoracic Society 
guidelines to predict an intensive care unit admission. Clin Infect Dis. 2009;48(4):377-
385. 
37. 
Kontou P, Kuti JL, Nicolau DP. Validation of the Infectious Diseases Society of 
America/American Thoracic Society criteria to predict severe community-acquired 
pneumonia caused by Streptococcus pneumoniae. Am J Emerg Med. 2009;27(8):968-974. 
38. 
Espana PP, Capelastegui A, Quintana JM, et al. Validation and comparison of SCAP as a 
predictive score for identifying low-risk patients in community-acquired pneumonia. J 
Infect. 2010;60(2):106-113. 
39. 
Ewig S, Ruiz M, Mensa J, et al. Severe community-acquired pneumonia. Assessment of 
severity criteria. Am J Respir Crit Care Med. 1998;158(4):1102-1108. 
40. 
Ewig S, de Roux A, Bauer T, et al. Validation of predictive rules and indices of severity 
for community acquired pneumonia. Thorax. 2004;59(5):421-427. 
41. 
Lim WS, Lewis S, Macfarlane JT. Severity prediction rules in community acquired 
pneumonia: a validation study. Thorax. 2000;55(3):219-223. 
42. 
Chalmers JD, Singanayagam A, Hill AT. Predicting the need for mechanical ventilation 
and/or inotropic support for young adults admitted to the hospital with community-
acquired pneumonia. Clin Infect Dis. 2008;47(12):1571-1574. 
43. 
Renaud B, Coma E, Labarere J, et al. Routine use of the Pneumonia Severity Index for 
guiding the site-of-treatment decision of patients with pneumonia in the emergency 
department: a multicenter, prospective, observational, controlled cohort study. Clin Infect 
Dis. 2007;44(1):41-49. 
44. 
Jo S, Kim K, Jung K, et al. The effects of incorporating a pneumonia severity index into 
the admission protocol for community-acquired pneumonia. J Emerg Med. 
2012;42(2):133-138. 
45. 
Atlas SJ, Benzer TI, Borowsky LH, et al. Safely increasing the proportion of patients 
with community-acquired pneumonia treated as outpatients: an interventional trial. Arch 
Intern Med. 1998;158(12):1350-1356. 
46. 
Marrie TJ, Lau CY, Wheeler SL, Wong CJ, Vandervoort MK, Feagan BG. A controlled 
trial of a critical pathway for treatment of community-acquired pneumonia. CAPITAL 
Study Investigators. Community-Acquired Pneumonia Intervention Trial Assessing 
Levofloxacin. JAMA. 2000;283(6):749-755. 
47. 
Chalmers JD, Singanayagam A, Akram AR, Choudhury G, Mandal P, Hill AT. Safety 
and efficacy of CURB65-guided antibiotic therapy in community-acquired pneumonia. J 
Antimicrob Chemother. 2010. 

Protocol Version #: 2.1 
 
18 
Protocol Date: October 26, 2022 
48. 
Barlow G, Nathwani D, Myers E, et al. Identifying barriers to the rapid administration of 
appropriate antibiotics in community-acquired pneumonia. J Antimicrob Chemother. 
2008;61(2):442-451. 
49. 
Lee RW, Lindstrom ST. A teaching hospital's experience applying the Pneumonia 
Severity Index and antibiotic guidelines in the management of community-acquired 
pneumonia. Respirology. 2007;12(5):754-758. 
50. 
Nadarajan P, Wilson L, Mohammed B, Connor M, Lane SJ. Compliance in the 
measurement of CURB-65 in patients with community acquired pneumonia and potential 
implications for early discharge. Ir Med J. 2008;101(5):144-146. 
51. 
Karsh BT, Weinger MB, Abbott PA, Wears RL. Health information technology: Fallacies 
and sober realities. J Am Med Inform Assoc. 2010;17(6):617-623. 
52. 
McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical 
decision support in primary care practices: a randomized clinical trial. JAMA Intern Med. 
2013;173(17):1584-1591. 
53. 
Coggins SA, Wynn JL, Hill ML, et al. Use of a computerized C-reactive protein (CRP) 
based sepsis evaluation in very low birth weight (VLBW) infants: a five-year experience. 
PLoS One. 2013;8(11):e78602. 
54. 
Anders S, Albert R, Miller A, et al. Evaluation of an integrated graphical display to 
promote acute change detection in ICU patients. Int J Med Inform. 2012;81(12):842-851. 
55. 
Anders S, Miller A, Joseph P, et al. Blood product positive patient identification: 
Comparative simulation-based usability test of two commercial products. Transfusion. 
2011;51(11):2311-2318. 
56. 
Slagle JM, Gordon JS, Harris C, et al. MyMediHealth – Designing a Next Generation 
System for Child-centered Medication Management. Journal of Biomedical Informatics. 
2010;43(5):S27-S31. 
57. 
Unertl K, Weinger M, Johnson K, Lorenzi N. Describing and modeling workflow and 
information flow in chronic disease care. J Am Med Inform Assoc. 2009;16(6):826-836. 
58. 
Weinger M, Wiklund M, Gardner-Bonneau D. Handbook of Human Factors in Medical 
Device Design. Boca Raton, FL: CRC Press/Taylor Francis; 2011. 
59. 
Weinger M, Pantiskas C, Wiklund M, Carstensen P. Incorporating human factors into the 
design of medical devices. JAMA. 1998;280(17):1484. 
60. 
Syroid ND, Agutter J, Drews FA, et al. Development and evaluation of a graphical 
anesthesia drug display. Anesthesiology. 2002;96(3):565-575. 
61. 
Wachter SB, Agutter J, Syroid N, Drews F, Weinger MB, Westenskow D. The 
employment of an iterative design process to develop a pulmonary graphical display. 
JAMIA. 2003;10:363-372. 
62. 
Xie M, Weinger M, Gregg W, Johnson K. Presenting multiple drug alerts in an 
ambulatory electronic prescribing system: a usability study of novel prototypes. Appl Clin 
Inform. 2014;5(2):334-348. 
63. 
Knab JH, Wallace MS, Wagner RL, Tsoukatos J, Weinger MB. The use of a computer-
based decision support system facilitates primary care physicians' management of chronic 
pain. Anesth Analg. 2001;93(3):712-720. 
64. 
Gesteland PH, Wagner MM, Chapman WW, et al. Rapid deployment of an electronic 
disease surveillance system in the state of Utah for the 2002 Olympic Winter Games. 
Proc AMIA Symp. 2002:285-289. 

Protocol Version #: 2.1 
 
19 
Protocol Date: October 26, 2022 
65. 
Wagner MM, Espino J, Tsui FC, et al. Syndrome and outbreak detection using chief-
complaint data--experience of the Real-Time Outbreak and Disease Surveillance project. 
MMWR Morb Mortal Wkly Rep. 2004;53 Suppl:28-31. 
66. 
Gesteland PH, Samore MH, Pavia AT, et al. Informing the front line about common 
respiratory viral epidemics. AMIA Annu Symp Proc. 2007:274-278. 
67. 
Reich NGM, A. M. Improving Efficiency in Cluster-Randomized Study Design and 
Implementation: Taking Advantage of a Crossover. Open Access Journal of Clinical 
Trials. 2013(6):11-15. 
68. 
Turner RM, White IR, Croudace T, Group PIPS. Analysis of cluster randomized cross-
over trial data: a comparison of methods. Stat Med. 2007;26(2):274-289.