Comparative Effectiveness of Direct Admission & Admission Through Emergency Departments for Children
Phase: NA
Conditions: Hospitalization, Child Health
Sponsor: Dartmouth-Hitchcock Medical Center

Grant Title: Comparative Effectiveness of Direct Admission & Admission 
Through Emergency Departments for Children 
Principal 
Investigator: 
JoAnna K. Leyenaar, MD, PhD, MPH 
ClinicalTrials.gov 
Number: 
NCT04192799 
Document: Final Approved Research Plan 
Date: 5/9/2022 (sub-heading corrected 7/26/24) 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
1 
PCORI RESEARCH PLAN TEMPLATE 
RESEARCH STRATEGY  
 
A. Specific Aims 
As our nation seeks a healthcare system that provides timely, equitable and patient-centered care, appropriate use of 
emergency departments (EDs) is a focus of national attention.1–3 In a landmark report, the Institute of Medicine 
describes our emergency medical system as overburdened, fragmented, and at the breaking point.1 Despite this, over 
the last two decades EDs have increasingly served as portals for hospital admission, contributing to care fragmentation 
and ED crowding.4 Of the 1.5 million non-elective pediatric hospitalizations that occur each year, 75% originate in EDs. 
The remainder occur via direct admission, defined as admission to hospital from the community without first receiving 
care in the hospital’s ED.5 Direct admission to hospital may originate from patients’ homes or from primary care or 
specialty clinics, typically facilitated by direct conversations between referring and accepting healthcare providers. 
While ED utilization patterns have been well studied, there is a paucity of research comparing the effectiveness of direct 
and ED admissions, particularly in children.4,6,7 Direct admission may offer benefits for both patients and healthcare 
systems, including reduced ED volumes, improved coordination between outpatient and hospital-based healthcare 
providers, and improved family experience of care. In addition, our systematic review identified several studies 
demonstrating that purposefully designed direct admission processes are associated with improved timeliness of clinical 
care for adults with acute myocardial infarction (AMI).8–21 However, unstructured direct admission processes may result 
in delays in clinical evaluation and treatment, which could adversely impact patient safety and quality of care.22,23  
The overall goal of this project is to compare the effectiveness of a standardized direct admission approach to admission 
beginning in the ED for hospitalized children. We will achieve this goal by addressing the following Specific Aims: 
Aim 1. Determine the effect of a pediatric direct admission intervention on timeliness of healthcare provision, family 
experience of care, and rates of clinical deterioration compared to pediatric hospital admission beginning in the ED.  
Hypothesis 1: Direct admission will be associated with more rapid initiation of clinical care (our primary outcome) and 
improved family experience of care with no significant differences in rates of clinical deterioration (as measured by rapid 
response calls and unexpected transfer to the intensive care unit (ICU)), compared with admission beginning in the ED. 
Aim 2. Identify the pediatric populations and conditions that experience the greatest benefits from direct admission 
with respect to timeliness of healthcare provision and family experiences of care. Hypothesis 2: Children with urinary 
tract infections (UTIs), skin and soft tissue infections (SSTIs), and co-morbid complex chronic illness will experience the 
greatest benefits from direct admission with respect to timeliness of care provision and family experience of care. 
Aim 3. Applying mixed qualitative and quantitative methods, identify barriers to and facilitators of implementing 
direct admission processes.  We will apply the RE-AIM implementation framework using mixed methods to carefully 
assess reach, effectiveness, adoption, implementation and maintenance of the intervention, as well as multi-level 
assessment of barriers of and facilitators to implementation.24 
We will accomplish these Aims by conducting a stepped-wedge cluster randomized controlled trial (RCT) to compare the 
effectiveness of direct and ED admission at 3 structurally and geographically diverse hospitals, randomizing 70 practice 
sites in the hospitals’ catchment area to crossover to the direct admission intervention at 4 time points, with a 
conservatively estimated sample size of 190 direct admissions and 1506 ED admissions from these practices. This 
research will be the first prospective study to evaluate the comparative effectiveness of direct and ED admissions for 
children, engaging diverse stakeholders to identify the populations, settings and processes most appropriate for this 
admission approach. In completing this research, we will generate knowledge of tremendous interest to both patients 
and families and their healthcare teams, enabling evidenced-informed decisions about how and when to admit children 
to hospital via direct admission. 
 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
2 
B. Background  
Transitions from outpatient settings into the hospital, whether via direct or ED admission, are experienced by almost 
every child and adult who requires hospital-based care. Approximately two million children are admitted to hospitals in 
the United States each year.25 These hospitalizations are costly, incurring healthcare costs that represent 40% of all 
national pediatric healthcare expenditures.26,27 In addition, hospital admissions have major implications for patients’ and 
families’ quality of life, disrupting normal routines, resulting in time lost from work and school, and contributing to both 
financial and emotional stress.28 These stressors are disproportionately experienced by children with chronic illnesses, 
who account for more than 50% of pediatric hospital admissions each year.25 Characterizing the effectiveness of direct 
and ED admissions will generate essential data to inform clinical decisions made by families and healthcare teams.29 
While hospital discharge processes have been the focus of tremendous research, policy, and quality improvement 
efforts over the last decade, research determining quality of care at the time of hospital admission is scant.30–32 
National programs have dedicated significant resources to improving hospital discharge processes: the Agency for 
Healthcare Research and Quality supported a major initiative to re-engineer discharge practices, the Centers for 
Medicaid and Medicare transformed payment structures with a focus on hospital readmissions, and national physicians 
organizations developed a Transitions of Care Consensus Policy Statement.30,33,34 Accordingly, the phrase “transition of 
care” is widely understood to describe the changes in setting, healthcare providers, and disease management strategies 
experienced by patients at the time of discharge. Like hospital discharge, hospital admission involves transitions in sites 
of care, handoffs between healthcare providers, and changes in medical therapies. Both are associated with significant 
stress to patients and their families.29,31 As a result, hospital admissions expose patients to many of same risks that have 
been the focus of hospital discharge reform: unstructured patient handoffs, poor communication between healthcare 
providers, and inefficient care. However, almost no research has aimed to improve transitions of care into the hospital.  
Direct admission accounts for 25% of all unplanned hospital admissions for children, with significant variation in rates 
of direct admission across hospitals and conditions.5 In our retrospective analysis using nationally representative data, 
we found that direct admission rates vary substantially across hospitals and conditions, with condition-specific direct 
admission rates for unplanned hospitalizations ranging from 8.9% for appendectomy to 38.0% for bipolar disorder.5 
Findings from this quantitative analysis are mirrored in our national survey of pediatric medical directors, with 97% of 
respondents reporting that they accept pediatric direct admissions; direct admission rates ranged from <10% to >50%.35 
Despite this, only one-third reported having formal direct admission policies in place, while 50% reported their belief 
that more children should be admitted directly. This significant variation across hospitals and conditions indicates clinical 
uncertainty regarding hospital admission best practices, emphasizing the value of our proposed research. 
Evidence Gaps 
Our systematic review of outcomes associated with direct admission to hospital reveals a paucity of research focused 
on children. In this recently completed review (manuscript currently being prepared), we identified 19 studies that 
reported outcomes associated with direct admission processes of care compared to admission through EDs.5,8–23,36,37 The 
vast majority of these evaluated purposefully designed direct admission systems targeting adult patients with signs and 
symptoms of AMI. Of the 14 studies focused on this population, 13 reported timeliness of care outcomes, and 12 of the 
13 reported more rapid definitive clinical management in adults who were directly admitted compared to those 
admitted through EDs.8–21 In contrast, two studies of adult populations conducted using administrative data have raised 
concerns about increased mortality associated with direct admission. The first, by Powell and colleagues, found that 
adults with sepsis experienced higher mortality when admitted directly than when admitted through EDs.23 The second, 
a study of unscheduled adult hospitalizations for a variety of common conditions, found that patients admitted directly 
had higher mortality for time-sensitive conditions such as AMI and sepsis; these differences were not observed among 
adults admitted less with emergent conditions including pneumonia, asthma and cellulitis.22                                                     
Only three studies compared the effectiveness of direct and ED admission in children, and all were retrospective cohort 
studies using health system data. The first study of children hospitalized with pneumonia found that children admitted 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
3 
directly received fewer diagnostic tests and incurred significantly lower healthcare costs than children admitted through 
the ED with no significant differences in rates of ICU transfer or hospital readmission.37 The second found that direct 
admission incurred healthcare costs 5-31% lower than ED admission.5 The third, focused on the relative risk of 
unplanned transfers to the ICU among direct- and ED-admitted children at a single children’s hospital, found no 
significant differences in rates of ICU transfer between the groups.38 Based on this systematic review, we conclude: (i) 
studies examining the comparative effectiveness of direct and ED admission are primarily focused on adult populations 
with heart disease; it is unknown if the results of these studies will translate to other populations; (ii) the vast majority 
of existing studies are cohort studies at high risk of bias based on application of a validated risk of bias assessment tool; 
and (iii) studies in children, a PCORI priority population, are under-represented in the existing literature.  
Completion of our proposed research has the potential to fill existing evidence gaps. Our proposed study represents 
the first RCT study to compare the effectiveness of direct and ED admission. Second, our rigorous design will examine 
admission processes and outcomes across the diversity of settings where children are admitted to hospital: at two 
children’s hospitals and one general hospital without pediatric subspecialty services.25 Third, we propose a focus on 
children, a PCORI priority population that is under-represented in research. Fourth, our direct admission intervention, 
developed based on a process of multistakeholder engagement with parents and both inpatient- and outpatient-based 
healthcare providers, will examine outcomes prioritized by diverse stakeholders.39 Finally, our proposal to augment our 
RCT with qualitative analyses of barriers and facilitators of project implementation will provide important contextual 
information about factors that support and hinder effective direct admission processes. 
C. Significance  
The rapid growth of hospital medicine in the United States has contributed to our need to evaluate direct and ED 
admission processes. Historically, many pediatric patients were admitted to hospitals by their primary care pediatricians 
(PCPs) or specialists who continued to provide their medical care during hospitalization and following hospital 
discharge.40 As a result, for physicians with hospital admission privileges, direct admission could be as straightforward as 
calling the pediatric ward to request a hospital bed and provide verbal orders. Pediatric Hospital Medicine (PHM), the 
pediatric specialty dedicated to the inpatient care of children, is the fastest growing specialty in pediatrics; an increasing 
number of hospitals have pediatric hospitalists in-house for up to 24 hours a day.41–43 While the consistent presence of 
hospitalists allows for around-the-clock pediatric-specific care, PHM also creates a discontinuity of care between 
outpatient and inpatient providers. As a result, direct admission requires a system to facilitate patient referrals and 
handoffs between ambulatory and inpatient settings. Without such direct admission systems, the majority of hospital 
admissions will begin in EDs. Given the rapid growth of PHM, research is needed to evaluate contemporary, 
standardized direct admission processes.  
Better understanding of the comparative effectiveness of direct and ED admission is important to parents of 
hospitalized children and to their healthcare teams.  In our formative work, we conducted interviews with parents of 
hospitalized children at four hospitals to characterize their experiences as 
they transitioned from outpatient to inpatient care and to identify 
hospital admission processes and outcomes most important to them.29 In 
addition, we conducted a national survey of pediatric medical directors 
regarding their experiences with and perspectives about direct admission 
to hospital.35 Both parents and physicians discussed potential benefits and challenges of direct and ED admission, and 
articulated a desire for standardized admission processes (Inset quote). Consistent with the Institute of Medicine’s 
domains of healthcare quality, parents’ described the importance of timely, effective, safe, and patient-centered clinical 
care, and expressed variation in their experiences with direct and ED admissions in achieving these aims.29   
We will evaluate outcomes of direct interest to patients, families, and healthcare teams.  To inform our proposed 
research we conducted a series of studies with rigorous multistakeholder engagement. As described above, we first 
conducted interviews with parents of hospitalized children to identify outcomes most important to them.29 We  
 “The process needs to be standardized. We currently 
have an open policy regarding direct admissions and 
utilize them according to our judgment. It would be 
useful to have a policy regarding suitable and unsuitable 
candidates for direct admission.” (Pediatric hospitalist) 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
4 
subsequently conducted deliberative discussions at three hospitals, discussing hospital admission processes and 
outcomes.39 Participants included parents, nurses, ED physicians, specialists, PCPs, pediatric hospitalists, and payers. 
Finally, we convened a panel of 9 experts representing these stakeholder groups, nominated via national organizations. 
Using a RAND/UCLA modified Delphi approach, stakeholders prioritized processes and outcomes to evaluate hospital 
admission systems at both children’s hospitals and general hospitals.44 Our proposed study is built directly upon this 
formative work, implementing both processes and outcomes prioritized by parents and other key stakeholders  (Table 
Key stakeholders in hospital admission decision-making processes are included in our study teams and advisory 
panels to ensure that our study processes and outcomes align with diverse stakeholder needs.  Key stakeholders in the 
hospital admission process include children and their families and referring healthcare providers who, together, often 
recognize the need for hospital admission. Pediatric hospitalists, ED physicians, resident physicians and nurses are key 
decision-makers influencing whether hospital admissions occur 
via direct or ED admission. At each of our three partner sites, we 
have therefore convened teams comprised of a senior health 
services researcher, a PHM clinical leader, and PCP and Parent 
Partners. If funded, these Direct Admission Leadership Teams 
(DALTs), will expand to include nurses, resident physicians (when 
applicable), complex care physicians, administrators, and practice 
managers. These site-based teams will be supported by a central 
research team and a national Direct Admission Advisory Panel 
(DAAP) that has contributed to our preliminary work.39 We 
provide examples of how these results will inform clinical 
decisions in Table 2.  
Table 1. Outcomes prioritized by key stakeholders to evaluate pediatric hospital admission processes, these will be evaluated in our proposed research. 
Outcome 
Representative Quotation (stakeholder) 
Total time from the time 
of arrival at the hospital to 
initiation of clinical care by 
the admitting physician or 
advanced practice 
provider 
“Sometimes the good thing is you get a lot of the tests a lot faster because it is in the ER and they need to get results pretty 
quickly to determine where they need to go from there. That is kind of a real benefit. That's why my doctor sent him to the ER 
too.” (parent) 
“I just feel like she gets better quicker because she doesn’t have to wait for so long…We go [for direct admission] and I know she 
will have an IV in her in an hour versus if we go to the emergency room it could be three to four hours before she has that IV.” 
(parent) 
“I think my biggest concern would be a child arriving on the floor and having no one see them for a period of time and having 
them get very sick before someone - either a nurse or a physician - to go in and see them. I mean I see that as the biggest 
potential risk. Making sure that if we do accept the direct admission, there is some person - some staff - free to go and assess 
that patient right away.” (Inpatient nurse) 
Patient and family 
experience of care 
 
“I had brought a copy of the x-ray with me, but they wanted their own copy.  So then they took another one…If I had my dream 
admission, the pediatrician…  would have called somebody here and said, ‘This is what I saw on the x-ray. I'm e-mailing you a 
copy of it. Pull it up on your screen. He or she would have pulled it up on the screen and said ‘Yes, I concur. Have them come to 
Floor 7 and check in at the desk and we will have the room ready.’ That would have been my perfect admission.” (parent) 
“… I’m making a generalization, but from our experience, when you’re in an ER situation people are just more stressed out...Like 
how could you not be?... And you might be on a stretcher with like curtains between each of you... Whereas, this is much more 
comfortable in terms of knowing like you’re coming right from a doctor’s office to a hospital room that’s ready for you, there’s a 
bed, there’s a nurse ready to come help you with needs.  She might have other patients but she’s not running around like an ER 
nurse.”  (parent)   
Unanticipated transfer for 
pediatric intensive care 
within 6 hours of hospital 
admission 
"Did the patient present as billed? …Were they discharged from the floor in a few hours because they were less acute than 
anticipated? Or were they transferred to the PICU?” (Inpatient nurse) 
 "That final outcome is really the patient recovery... How soon, how quick, how complete, is the recovery. And any readmission 
down the road. Any complications. Like escalated to ICU? All those are of interest in addition to patient and family experience..." 
(Primary care provider) 
Rapid response calls within 
6 hours of hospital 
admission  
“It's just that we, in our head, are thinking ‘Ok, what's the worst this patient could look like?’ Because we don't want the patient 
to come to the floor and then have a Rapid Response…” (Hospitalist) 
 
 
Table 2. Clinical decisions that will be informed by completion of 
this study. 
A child is seen by their primary care pediatrician with a skin 
infection that has not gotten better with oral antibiotics. The 
pediatrician thinks the child needs hospital admission. Should the 
pediatrician send the child to the emergency department for 
evaluation, or will the child receive more timely, safe and/or 
patient-centered care with a direct admission? 
A pediatric hospitalist receives a call from a primary care provider 
requesting a direct admission for a child with a presumed diagnosis 
of pneumonia.  Should the pediatric hospitalist accept the child for 
direct admission, or should s/he recommend initial evaluation and 
management in the ED instead?  

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
5 
Our local and central research team structure will enable dissemination of research findings beyond peer-reviewed 
publications and national conferences. We anticipate several scholarly papers and presentations resulting from this 
work. Beyond this, members of our central DAAP have committed to sharing summaries of our findings with the national 
organizations that they represent (Family Voices, the Health Care Delivery Committee of the Academic Pediatric 
Association, and the American Academy of Pediatrics Section on Hospital Medicine, Council on Pediatric Subspecialties,  
and Value in Inpatient Pediatrics Network; please see their Letters of Support (LOS)). In addition, we have the 
institutional support of senior administrators and clinical leaders at each of our performance sites; all have expressed a 
strong interest in receiving the results of this research to inform their clinical operations. Finally, Paige Stein, Director of 
Communications at The Dartmouth Institute for Health Policy & Clinical Practice (TDI), has committed to assist with 
purposeful dissemination of findings via social media, press releases, and media alerts (please see her LOS).  Our 
scientific approach, pairing a robust RCT design with a multi-level assessment of barriers to and facilitators of 
implementation, will generate valuable data about how positive findings can be reproduced at other healthcare 
systems nationally.  
D. Study Design or Approach 
1. Summary. We will conduct a stepped-wedge cluster randomized controlled trial to evaluate outcomes associated 
with a direct admission intervention at three health systems: Children’s Hospital of Pittsburgh (CHP) in Pittsburgh, 
Pennsylvania, Nationwide Children’s Hospital (NCH) in Columbus, Ohio, and Providence Regional Medical Center (PRMC-
E) in Everett, Washington. We have partnered with these health systems given their structural diversity, current low 
direct admission rates, and strong institutional support to implement and evaluate direct admission processes. A total of 
70 practices affiliated with these hospitals will be randomized to cross over from ED admission (usual care at these 
hospitals) to the direct admission intervention at 4 time points, stratified by practice group. Given 3 practice closures 
and mergers during the first year of the project, an additional 2 practices will be randomized to participate in the second 
half of the project. Our primary outcome will be timeliness of clinical care provision. Secondary outcomes include family-
reported experience of care and two measures of clinical deterioration:  unplanned transfer to the ICU and rapid 
response calls within 6 hours of hospital admission. We will compare outcomes in children and adolescents with five 
common diagnoses from practices randomized to the direct admission intervention to those admitted from practices 
not yet eligible for the intervention, adjusting for appropriate covariates.   
2. Conceptual Framework. Direct admission to hospital has several prerequisites, including access to timely ambulatory 
care for children with acute illnesses, and communication between outpatient and inpatient healthcare providers to 
initiate direct admission referrals. Our research approach reflects these prerequisites and is modified from Donabedian’s 
structure, process, outcome framework to also incorporate patient characteristics and family preferences (Figure 1).45 
This framework illustrates how systems factors and processes of care may influence where children access ambulatory 
care for acute illnesses (ED or primary care clinic) and whether admission occurs directly or through the ED. These 
factors, in turn, influence health outcomes. The System Factors summarized in this model, including clinic and ED 
factors, were described by parents of hospitalized children as influential to their decisions about when and where they 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
6 
seek 
  
     Figure 1. Conceptual framework informing research approach, modified from Donabedian’s structure, process, outcome framework.   
care for their child’s acute illnesses.29 The Processes of Care factors are derived from deliberative discussions with 
stakeholders in the hospital admission process,39 as well as our systematic review of the hospital admission literature. 
Finally, the health outcomes shown were prioritized by these same stakeholders.39 Accordingly this conceptual 
framework reflects a process of intensive stakeholder engagement in the development of our research question and in 
selecting our primary and secondary outcomes. From this Conceptual Framework, our Causal Model may be 
summarized as follows: We hypothesize that, by eliminating ED wait times, exposure to the often hectic ED 
environment, and reducing the number of healthcare teams involved in care provision, directly admitted children will 
have reduced time to clinical care and improved experiences. The relative benefits of direct admission may differ based 
on medical complexity, with complex patients, often well known to the hospital-based healthcare teams, more likely to 
receive timely, personalized care. Primary reason for hospitalization may also influence timeliness of care provision, with 
conditions requiring minimal diagnostic testing prior to treatment initiation (for example UTIs and SSTIs) having the 
greatest benefit from direct admission.29,35  
3. Comparators. This study will involve a comparison of outcomes in children who are admitted from eligible primary 
care practices where the direct admission intervention is available (including children admitted directly and through EDs) 
to those admitted from practices not yet eligible for the intervention (hence all ED admissions).  Both ED and direct 
admission are in current clinical use, with national statistics showing that 25% of children with unplanned 
hospitalizations are admitted via direct admission while 75% are admitted through EDs.5 Children who are transferred 
from another hospital (inter-hospital transfers) will be excluded, given our inability to account for types and duration of 
healthcare received at the transferring hospital. All hospitals in the US that receive reimbursement from the Centers for 
Medicare and Medicaid Services are required to document this variable on Universal Billing forms, and the validity of 
this variable has been demonstrated in two previous studies.46,47 
Intervention Core Elements. The direct admission intervention will involve five core elements, all of which were 
prioritized by our national multistakeholder panel in our preliminary work:39 (i) direct admission education and tools for 
both referring healthcare providers (PCP practices) and accepting healthcare providers (nurses, resident physicians and 
pediatric hospitalists), including diagnoses and populations eligible for direct admission, and patient referral methods; 
(ii) a system to facilitate direct communication between referring and accepting healthcare providers, enabling a single 
telephone call from a referring physician to an accepting physician (currently in place at all hospitals to facilitate 
acceptance of inter-hospital transfers), and use of a structured data collection tool to facilitate determination of the 
appropriateness of the patient for direct admission; (iii) instructions for families regarding when/where/how to proceed 
for direct admission; (iv) rapid evaluation of clinical stability of the patient upon hospital arrival by the inpatient 
healthcare team; and (v) timely initiation of appropriate clinical care. These five core elements will be augmented by 
supplemental intervention elements prioritized by our multi-stakeholder panel and detailed in Appendix A.39  
Local Adaptation and Fidelity Monitoring. All sites have agreed to implement the above-described core elements, but we 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
7 
recognize that resources available within each healthcare system will differ somewhat, as may the priorities, preferences 
and values expressed by key stakeholders at each project site. As a result, during the pre-rollout period each site’s DALT 
will meet with the central research team to adapt the core elements as needed while concurrently maintaining high 
implementation fidelity. This process will be informed by the Dynamic Adaptation Process developed by Aarons et al., 
engaging multiple stakeholders to take into account the local context and resources.48 During the pre-implementation 
period, each DALT will meet monthly via conference call with the central research team to discuss the core and 
supplemental intervention elements within the context of local system, organization, provider, and patient 
characteristics.48 These discussions will be supported by review of local administrative data and pilot data of study 
outcomes, with tailoring of educational materials and tools as appropriate. During the implementation phase, the DALT 
and central research team will meet quarterly to discuss implementation fidelity and the need for potential adaptation.  
Covariates. In addition to the above-described comparators, we will evaluate, at baseline, patient-level characteristics 
that may be associated with portal of hospital admission. These include: child age, gender, race/ethnicity, primary payer 
(Medicaid, commercial, other), primary reason for hospital admission (clinical condition), medical complexity, and level 
of care (observation or inpatient status). Medical complexity will be categorized as no concurrent chronic illness, chronic 
non-complex illness (i.e. diabetes, asthma), 
or complex chronic disease (i.e. 
neuromuscular disease, technology 
dependence) by applying the validated 
Pediatric Medical Complexity Algorithm 
(PMCA).49 All variables will be extracted 
from hospitals EHRs using standardized 
processes developed and piloted during 
the pre-rollout period. 
4. Study Design & Randomization. To 
achieve our research objectives, primary 
and urgent care practices at each hospital 
will be randomized to cross over from ED 
admission to the direct admission 
intervention using a cluster-randomized 
stepped wedge design (SWD). The direct 
admission intervention will be sequentially 
deployed to 4 groups of practices as shown 
in Figure 2. This crossover will be 
unidirectional and occur at 4 time points, 
with the first two blocks being six months 
in duration, and the second two blocks 
being nine months in duration given 
relatively slow enrollment during the 
COVID-19 pandemic. Additionally, to 
increase the number of children in the 
direct admission arm, we will continue to 
evaluate the direct admission program 
during a 9-month maintenance block 
(months 37-45) at NCH and PRMC-E 
following the current approach to 
recruitment and enrollment . Because of 
Figure 2. a) Original design: Characteristics of stepped wedge cluster randomized 
controlled trial, where shaded areas indicate direct admission intervention 
exposure and unshaded areas indicated control conditions  
 
TIME (0-36 month project period; each block represents 6 months) 
 
0 mo 
6 mo 
12 mo 
18 mo 
24 mo 
30 mo 
Group 1 (18 
practices) 
 
 
Pre-
rollout 
period 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
 
 
Post-
rollout 
period 
Group 2 (18 
practices) 
DA: 0 
ED: 106 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
Group 3 (17 
practices 
DA: 0 
ED: 106 
DA: 0 
ED: 106 
DA: 19 
ED: 87 
DA: 19 
ED: 87 
Group 4 (17 
practices) 
DA: 0 
ED: 106 
DA: 0 
ED: 106 
DA: 0 
ED: 106 
DA: 19 
ED: 87 
 
 
------------Rollout period--------------------------- 
(DA = direct admission; ED = ED admission; numbers indicate projected sample size) 
 
b) Modified design given reduced hospitalization volumes during the COVID-19 pandemic, 
with the 18 and 24 month steps extended to 9 months each, and addition of a 9-month 
maintenance block (months 37-45). 
 
 
 
TIME (0-50 month project period; each block represents 6 
months unless otherwise specified; the 27mo and 36mo blocks 
are projections estimated from enrollment to date) 
 
0 mo 
6 mo 
12 mo 
18 mo 
(9 mo 
step) 
27 mo 
(9 mo 
step) 
36 mo 
(9 mo 
step) 
45-50 
mo 
Group 1  
(18 
practices) 
 
 
Pre-
rollout 
period 
DA: 8 
ED: 129 
DA: 8 
ED: 92 
DA: 23 
ED: 216 
DA: 12 
ED: 232 
DA: 12 
ED: 148 
 
Post-
rollout 
period 
Group 2  
(18 
practices) 
DA: 6 
ED: 85 
DA: 12 
ED: 61 
DA: 12 
ED: 147 
DA: 5 
ED: 172 
DA: 5 
ED: 117 
Group 3  
(18 practices 
DA: 2 
ED: 132 
DA: 1 
ED: 66 
DA: 5 
ED: 153 
DA: 5 
ED: 145 
DA: 3 
ED: 85 
Group 4  
(18 
practices) 
DA: 6 
ED: 91 
DA: 5 
ED: 56 
DA: 10 
ED: 148 
DA: 22 
ED: 182 
DA: 22 
ED: 148 
 
 
 
------------Rollout period--------------Maintenance 
 
 

PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template 
8 
 
very minimal program updake at CHP, we will not e
 
valuate program maintenance at CHP.  Figure 2b summarizes our 
anticipated additional recruitment of patients admitted directly and via emergency department admission at NCH and 
PRMC-E, with numbers estimated from actual enrollment to date.  
We will randomize 70 primary and urgent care practices: 18 at NCH, 34 at CHP, and 18 at PRMC-E (Please see LOS from 
PCP practices affiliated with each hospital). Practices in each group will be randomized using a stratified approach, with 
17-18 practices (4-9 practices per hospital) randomized to crossover to the direct admission intervention at each
step/crossover point, stratified by hospital and practice group. Given the closure or merger of 3 practices during the first 
project year, an additional 2 practices will be randomized to either Group 3 or Group 4, beginning during the 18 month 
block (Block 3).   
Justification for Study Design. Our proposed SWD, like all studies of this type, will randomize the primary care 
practices to a time point to receive the intervention, but analyze outcomes at the individual level. Our proposed 
randomization approach is ethical, given uncertainty regarding the relative timeliness and patient-centeredness of direct 
and ED admissions for children.50 However, randomization at the level of the individual is impractical, as consent and 
randomization would need to occur by hundreds of PCPs in their office settings – a feat that would be extraordinarily 
difficult to accomplish. Beyond this, we have selected the SWD for several reasons. First, this study design has an 
advantage over conventional cluster randomized trials in that it allows stepwise rollout of the direct admission 
intervention to practices, requiring fewer resources than would be needed to offer the intervention to all clusters 
simultaneously. Relatedly, the SWD will allow the PHM services accepting direct admissions to start with a relatively low 
number of direct admissions, and to make necessary organizational and infrastructure changes if needed to support our 
direct admission evaluation during a lower-volume time period. This study design has an analytic advantage as well, 
allowing us to make comparisons within groups over time, as well as between groups at the same time period.  Finally, 
because of the perceived benefits of direct admission and to optimize the enrollment of PCP practices, this design is 
advantageous to maintain control practice engagement in the study.50–52 
5. Statistical Analysis Plan  Overview. Our analytic approach addresses some key features of SWD. First, because the 
design involves unidirectional crossover from the control (ED) to the intervention arm, our models will include an 
indicator specific to the practice and time point. Second, our analyses will take into account the hierarchical structures of 
the data. Each patient contributes a single observation of each outcome; hence, the data are cross-sectionally structured 
at the patient level and longitudinally structured at the clinic level with patients and clinics nested within hospitals. Third, 
our analyses will carefully evaluate secular time trends, to separate the effect of the intervention from the effect of time. 
Fourth, to identify the populations that experience the greatest benefits from direct admission, we will test several a 
priori hypotheses.
Data Adequacy & Management. With the exception of our parent-reported outcome, all required data for this project 
will be extracted from the hospitals’ EHRs; all proposed variables exist in EHRs, thereby minimizing data collection 
burden and missingness as a result of non-response. During the pre-implementation period, all sites will conduct pilot 
data extraction from EHRs to assess data source adequacy; any deficiencies will be addressed prior to the rollout. 
Following data extraction at hospital sites, de-identified data will be sent via HIPAA-secure methods to a secure 
Dartmouth Research Computing Server, where all data analyses will be conducted and subsequently shared with sites. 
Prior to study rollout, a comprehensive Data Management Plan will be created with feedback from study sites and 
stakeholders. Although we anticipate very low rates of missing data based on our preliminary work, we will monitor for 
and report missing data and use multiple imputation to protect analyses against the missing-at-random assumption.53,54 
In all analyses, we will examine the sensitivity of this approach, and consider this in our interpretation and conclusions.
Aim 1. Determine the effect of a pediatric direct admission intervention on timeliness of healthcare provision (our 
primary outcome), family experience of care, and rates of clinical deterioration compared to hospital admission 
beginning in the ED.   

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
9 
We hypothesize that our direct admission intervention will be associated with more rapid initiation of clinical care and 
improved family experience of care with no significant differences in rates of clinical deterioration (as measured by rapid 
response calls and unexpected transfer to the ICU). We test two variants of this hypothesis: (i) a patient-level version 
and (ii) a clinic-level version. The patient-level version is direct in that it evaluates the aim and tests the hypothesis as 
stated. The clinic-level version is indirect in that it tests whether the intervention has an overall effect across all patients, 
whether or not they were directly admitted.  
We will first determine crude differences in baseline clinical and sociodemographic characteristics, and in our primary 
and secondary outcomes, between children admitted directly and through EDs, using t-tests and chi-square tests as 
appropriate. However, because these unadjusted analyses do not adjust for observed predictors related to direct and ED 
admission, including temporal trends, for our main analyses we will use regression models. Because all patients receive 
some form of clinical care at the hospital irrespective of form of admission, there is no censoring associated with the 
measurement of time to clinical care. Therefore, standard regression models may be used. Patient characteristics 
incorporated in models will include diagnosis, medical complexity,49 level of care (observation versus inpatient), as well 
as age, gender, race/ethnicity and payer. The statistical model for our direct patient-level comparison of direct 
admission (DA) versus ED admission patients (analysis i) is  
𝑌𝑌𝑖𝑖𝑖𝑖ℎ𝑡𝑡= 𝛽𝛽0 + 𝛽𝛽1𝐷𝐷𝐴𝐴𝑖𝑖𝑖𝑖ℎ𝑡𝑡+ 𝛽𝛽2𝑥𝑥𝑖𝑖𝑖𝑖ℎ𝑡𝑡+ 𝛽𝛽3𝑡𝑡+ 𝛽𝛽4ℎ+ 𝜃𝜃𝑗𝑗ℎ+ 𝜀𝜀𝑖𝑖𝑖𝑖ℎ𝑡𝑡 
in which 𝑌𝑌𝑖𝑖𝑖𝑖ℎ𝑡𝑡 is the outcome variable for patient i seen by clinic j at hospital h at time t; 𝐷𝐷𝐴𝐴𝑖𝑖𝑖𝑖ℎ𝑡𝑡 is a binary variable 
indicating whether the patient was admitted via direct admission (𝐷𝐷𝐴𝐴𝑖𝑖𝑖𝑖ℎ𝑡𝑡= 1 ) or the ER (𝐷𝐷𝐴𝐴𝑖𝑖𝑖𝑖ℎ𝑡𝑡= 0 ), 𝑥𝑥𝑖𝑖𝑖𝑖ℎ𝑡𝑡 is a vector 
of patient-level controls and 𝜃𝜃𝑗𝑗ℎ is a random effect for clinic within hospital. In addition, the model includes fixed-effects 
for time-period, 𝛽𝛽3𝑡𝑡, and hospital, 𝛽𝛽4ℎ, to capture the general unstructured trend across calendar time and hospital-
specific effects, respectively. The key coefficient of interest is 𝛽𝛽1, which captures the association of a patient being 
admitted directly versus by an ED with the outcome. A crucial feature of this analysis is that the predictor of primary 
interest, DA, varies within clinic and time-period mitigating almost all forms of clustering. Therefore, although the data 
are clustered by clinic and time, the primary source of information about the association of DA with the outcome is from 
contrasts within clinic and time-period. Hence, accounting for time-period and clinic effects is likely to strengthen the 
statistical significance of our results. 
When estimating the model in (i), for our primary analysis we plan to only use observations from the post-period. This 
avoids any contamination from outcomes of ED observations when DA was not available. However, to gain some insight 
into the closeness of the results from this analysis with those that we’d expect to see if it was possible to have patient-
level randomization, in a supplemental analysis we will include the pre-intervention ED observations and test if there is a 
change in the mean ED outcome in the post-intervention period. A result consistent with minimal change over time will 
support a finding that, in the absence of the intervention, the outcomes for DA patients would have been the same as 
those observed for ED patients. 
The statistical model for analysis (ii) has a similar form: 
𝑌𝑌𝑖𝑖𝑖𝑖ℎ𝑡𝑡= 𝛽𝛽0 + 𝛽𝛽1𝑃𝑃𝑃𝑃𝑃𝑃𝑡𝑡𝑗𝑗ℎ𝑡𝑡+ 𝛽𝛽2𝑥𝑥𝑖𝑖𝑖𝑖ℎ𝑡𝑡+ 𝛽𝛽3𝑡𝑡+ 𝛽𝛽4ℎ+ 𝜃𝜃𝑗𝑗ℎ+ 𝜀𝜀𝑖𝑖𝑖𝑖ℎ𝑡𝑡 
differing only in that the direct admission indicator is replaced with 𝑃𝑃𝑃𝑃𝑃𝑃𝑡𝑡𝑗𝑗ℎ𝑡𝑡, a time-varying practice within hospital 
indicator of whether the intervention has been rolled-out (1 = intervention transition has occurred by time period t and 
0 = direct admission intervention has not occurred by time-period t). The key coefficient of interest is 𝛽𝛽1, which captures 
the structural shift in the outcome that occurs when a practice receives the direct admission intervention. Because 
𝑃𝑃𝑃𝑃𝑃𝑃𝑡𝑡𝑗𝑗ℎ𝑡𝑡 does not vary within a clinic and time-period, the information in the observations made within a clinic and time-
period depends on the size of the effects of clinic and time-period with the statistical significance of inferences about 𝛽𝛽1 
is likely to be reduced by the clustering of observations in clinics. An additional component of clustering arises because 
the four groups are aligned with nine practice groups that in turn are embedded in three hospitals. We acknowledge 
that there may be separate levels of clustering in the data due to hospital and below that due to practice groups. 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
10 
However, the nine practice groups emanating from the 3 hospitals will be randomly distributed across the four groups of 
the stepped-wedge design in a manner that balances the four groups with respect to the characteristics of these groups. 
Because the practice groups and hospitals are cross-classified with the SWD groups, their effects on the statistical 
precision of the results will be much less than under a purely hierarchical arrangement. 
Recognizing that the addition of two practices at CHP midway through the project, randomized to either Group 3 or 
Group 4, has the potential to have some unique or differential effect on our outcomes of interest, we will conduct a 
sensitivity analysis excluding these two practices. This will allow us to determine the extent to which our findings are 
associated with the addition of these two practices.  A substantial difference in results would be suggestive of 
heterogeneous effects of the intervention across the sites and our conclusions would discuss such a finding. The 
prolongation of Steps 3 and 4 from 6 months to 9 months, and the addition of a 9-month maintenance block (months 
37-45), will be naturally addressed with our statistical model due to observations being at the level of the individual. 
Each observation will be coded according to the time-period the study is in at the time of the observation. The width of 
the intervals will not have an effect on the results of the analysis. If we have concerns about trends within an interval, 
we will add an additional covariate (including in the vector 𝑥𝑥𝑖𝑖𝑖𝑖ℎ𝑡𝑡)  that reflects the precise date when the observation is 
made. These analytic additions will apply to both Aim 1 and Aim 2. 
Aim 2. Identify the pediatric populations and conditions that experience the greatest benefits from direct admission with 
respect to timeliness of healthcare provision and family experiences of care.  
To test our hypothesis that children with UTI, skin and soft tissue infections, and 
co-morbid complex chronic illness will experience the greatest benefits from 
direct admission relative to children admitted with other clinical diagnoses, we 
will evaluate group-level heterogeneity of treatment effects (HTE) by conducting 
subgroup analyses as specified a priori. Although there is no prior literature about 
the relative benefits of direct admission for these subgroups, this hypothesis is 
derived from our deliberative discussions with multidisciplinary stakeholders, 
based on their lived experiences. Our anticipated sample size for subgroups is 
derived from national statistics about pediatric inpatient stays 
(HUCPnet.ahrq.gov), shown in Table 3. Although our assessment of HTE will follow 
methodological guidance that all subgroups be specified a priori, we will consider 
other subgroup analyses based on recommendations from Parent Partners and 
hospitals’ DALTs.55,56 Subgroup analyses will be performed by estimating an analogous model to that for analysis (i) given 
above for the subgroups of interest. To evaluate whether the effect of the direction admission intervention varies 
significantly across subgroups, we will augment the model specified in Aim 1 analysis (i) with a predictor for the product 
of the subgroup-defining variable and DA; the coefficient of the additional variable is referred to as an interaction effect 
and its value captures the extent to which the effect of DA differs between one level of the subgroup to the other. This 
approach is based on the recommendations by Kent et al., who recommend analysis and reporting of multivariate risk-
based HTE to account for the fact that patients have multiple characteristics simultaneously that affect the likelihood of 
benefiting from an intervention.56,57  
Aim 3. Applying mixed qualitative and quantitative methods, identify barriers to and facilitators of implementing direct 
admission processes.  
Table 3. Projected subgroup sample sizes 
Condition / Population 
(Total sample n=1696) 
Anticipated 
n (%) 
Medical Complexity: 
 
  No chronic disease 
 644 (38%) 
  Chronic, non-complex  
 441 (26%) 
  Complex chronic disease 
611 (36%) 
Diagnosis: 
 
  Pneumonia 
551 (33%) 
  Skin / soft tissue infection 
262 (16%) 
 Gastroenteritis/dehydration 
317 (19%) 
  Urinary Tract Infection 
182 (11%) 
  Viral infection not 
otherwise specified 
293  (18%) 
  Influenza 
51 (3%) 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
11 
To determine the impact of our direct admission intervention and to inform post-project implementation in other health 
systems, we will conduct a mixed-methods process evaluation applying the RE-AIM implementation framework to 
assess reach, effectiveness, adoption, 
implementation and maintenance of our direct 
admission intervention.24 These domains, 
definitions, and our proposed approach are 
summarized in Table 4. This approach 
combines analysis of our primary and 
secondary outcomes with analysis of process 
measures and qualitative interviews with 
stakeholders. Semi-structured interviews 
focused on Adoption, Implementation, and 
Maintenance domains (Table 4) will be 
conducted with parents of hospitalized 
children, referring PCPs, and inpatient 
healthcare team members including nurses, 
resident physicians, hospitalists, and other key 
stakeholders. We anticipate completing 12-16 
in-depth interviews with stakeholders each 
calendar year, for a total of 36-48 interviews 
per site over the 4-year implementation 
period. However, interviews will be continued 
until thematic saturation is reached.58 Interviews will be recorded with permission and transcribed verbatim, and 
analyzed iteratively (concurrent with ongoing qualitative work) using a general inductive approach.59–61 Analysis will be 
led by Drs. Leyenaar and McDaniel, in partnership with Ms. Arthur (Co-investigator, Years 1-3), Ms. Jacob-Files (Years 3-
5), Ms. Taylor (Years 3-5) and Ms. Stevens (Parent partner). Results summaries will be shared with interview 
participants, and with hospitals’ DALTs, both to return results to participants and to member-check the credibility of 
findings.62,63  
6. Study Population and Setting 
Eligibility Criteria. We will limit our analysis to children < 18 years of age admitted with the following common medical 
reasons for hospitalization: gastroenteritis/dehydration, skin and soft tissue infection, UTI, pneumonia, viral infection 
not otherwise specified, and influenza. These diagnoses comprise approximately 25% of all unscheduled PHM 
admissions nationally, and represent conditions that have been identified as appropriate for direct admission but not 
currently accepted for direct admission at the participating hospitals.25,39 Children with planned admissions (i.e. 
chemotherapy), those admitted to non-PHM Services (i.e. the ICU), and those transferred from other hospitals will be 
excluded.  
Selection of Study Sites. To achieve our overall goal of implementing and evaluating a standardized direct admission 
intervention, we aimed to select hospital sites with the following characteristics: (i) low baseline direct admission rates, 
allowing us to implement a standardized direct admission system; (ii) institutional support to increase direct admission 
rates using a standardized direct admission system; and (iii) interest from and capacity among pediatric primary care 
practices to refer children for direct admission when indicated. Beyond this, to optimize the external validity of our 
study findings, we aimed to work with geographically and structurally diverse hospitals, including both general (non-
children’s) hospitals and children’s hospitals. Relatedly, we sought project partners that had differing 
relationships/levels of integration between PHM services and PCP practices (i.e. primary care practice ownership versus 
independence). To solicit interest from a diverse prospective sample of partner sites, we received the support of the 
Pediatric Research in Inpatient Settings (PRIS), a research network with more than 100 participating hospitals nationally. 
Table 4.  Components of mixed methods process evaluation using RE-AIM Framework  
Domain and Definition 
Approach 
REACH: the % and 
characteristics of children 
eligible for the direct 
admission intervention who 
were admitted via this 
approach  
Monthly reports to primary care practices reporting the 
number and % of eligible children admitted via direct 
admission; quarterly reports displaying clinical and 
sociodemographic characteristics of children admitted 
directly compared to those admitted via the ED 
EFFICACY: consideration of 
positive and negative 
outcomes of the intervention 
Quarterly reports to primary care practices of primary 
and secondary study outcomes  
ADOPTION: barriers to and 
facilitators of adopting this 
intervention 
Qualitative interviews with key stakeholders will focus on: 
(i) experience with the direct admission intervention and 
(ii) barriers to and facilitators of a) referral for direct 
admission, b) delivery of the intervention, including 
adherence to core components c) provision of timely and 
patient-centered care, and (d) assurance of patient safety 
IMPLEMENTATION: the 
extent to which the 
intervention is delivered as 
intended 
Qualitative interviews with stakeholders, and quarterly 
meetings of the Direct Admission Leadership Teams to 
discuss barriers to and facilitators of adherence to 
intervention 
MAINTENANCE: the extent to 
which the intervention is 
sustained over time 
Qualitative interviews with stakeholders, and meetings of 
the NCH and PRMC-E Direct Admission Leadership Teams 
to discuss barriers to and facilitators intervention 
maintenance 
 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
12 
The PRIS Executive Council, comprised of physician-researchers from 10 research institutes nationally, reviewed our 
proposed study design and approach, providing valuable methodological feedback. Following this review PRIS assisted 
with recruitment of potential sites via their national listserv. Through this process, we identified our 3 partner hospitals 
as those best meeting the key characteristics described above. (Please see LOS from PRIS Chair, Dr. Karen Wilson). 
Structure of Study Sites: CHP has a Pediatric ED with an annual volume of over 80,000 visits and a PHM Service that 
provides care to > 9000 children each year. At CHP, the ambulatory care partners will be Pediatric Alliance (11 practices), 
Kids Plus Pediatrics (3 practices), Children’s Community Pediatrics (18 practices including 14 primary care and 4 urgent 
care), and General Academic Pediatrics (2 practices). These practices are affiliated with CHP as part of a regional 
Pediatric Clinical Integrated Network. Excluding General Academic Pediatrics, these practice groups are independently 
owned and operated. NCH has a Pediatric ED annual volume of  >90,000 visits, and a PHM Service that provides care to 
6000 children each year. At NCH, the ambulatory care partner will be their NCH-owned Primary Care Network of 12 
clinics, and 6 NCH-owned urgent care centers. Dr. Ryan Bode, Site Principal Investigator at NCH is Division Chief in 
Hospital Medicine; he is fully supportive of randomizing their 12 primary care practices for this project. Finally, PRMC-E 
is a general hospital in Everett, Washington, where PHM services are provided by the Division of PHM at Seattle 
Children’s Hospital.  At this hospital, children seeking care in their ED are cared for in a general (non-children’s) ED by 
general (non-pediatric) emergency medicine physicians.  PRMC-E admits approximately 650 children annually. The 
ambulatory care partners at this site include the three primary care groups that together account for two-thirds of 
PRMC-E’s referrals: The Everett Clinic (TEC), comprised of 10 pediatric primary care practices; the Providence Medical 
Group (PMG), comprised of 6 pediatric primary care practices; and Community Health Clinics of Snohomish County 
(CHC), comprised of 2 pediatric primary care practices.   
7. Recruitment Plan for Prospective Studies. Each of our three hospital partners examined administrative data from the 
2017 calendar year to identify the number of children admitted to their PHM services, with an eligible diagnosis, from 
our partnering pediatric primary care practices. The resulting projected sample size is shown in Table 5, with hospital-
specific estimates provided in Appendix B.  Given the availability of historic data from all of our partner sites, we are 
confident in our sample size estimate. Because our primary outcome will be derived from EHR data, which is available 
for all hospitalized children by hospital protocol and procedures, we anticipate no missing data for this timeliness 
outcome. This is also true of our secondary outcomes derived from EHR data, including ICU transfer and rapid response 
calls. However, for our parent-reported measure of family experience of care, we estimate a response rate of 75%, 
accounting for potential parental refusals, absence of parents from the bedside, and inability to complete the survey in 
English, Spanish, Somali, Nepali or Arabic. (This 
response rate aligns with past work requesting 
parental participation in a survey during their 
hospital stay64,65). To collect this parent-reported 
outcome, we will approach parents/guardians 
during their child’s hospital stay, either by phone 
or in-person, keeping with each site’s COVID-19 
regulations. A research assistant will explain the 
study, and administer the survey via tablet or via 
personal cellphone to those parents/guardians 
who provide informed consent.  Our sample size 
estimate for the relative number of direct 
admissions is dependent upon three 
assumptions/components of the direct admission 
process – first, that approximately 13% of 
children who are admitted to hospital will be 
seen by their PCPs prior to hospital admission (an 
Table 5.  Estimated Sample Size, Attrition, and Monthly Enrollment  
1. Estimated number of potentially eligible study participants: Estimated 
number of children meeting eligibility criteria during two year 
implementation period, derived from participating hospitals’ 
administrative data (2017).  
2288 
2. Total number of study participants expected to be screened: All  
2288 
 
3. Total number of study participants expected to be eligible of those 
screened: All directly admitted patients, 70% of patients admitted through 
EDs at children’s hospitals, and 95% of children admitted through EDs at 
our participating community hospital  
1696 
 
4. Target sample size (use same number stated in milestones): Target sample 
size for direct admission intervention = 190, for ED admission = 1506 
1696 
 
5. If applicable, total number of practices or centers that will enroll 
participants: 70 primary care and urgent care practices will be randomized 
70 
6. Projected month first participant enrolled (month after project initiation): 
6 mo 
7. Projected month last participant enrolled (month after project initiation): 
45 mo 
8. Projected rate of enrollment:  estimated number of enrolled children/mo  
70 
9. Estimated percentage of participant dropout: We anticipate a zero drop-
out rate for our timeliness primary outcome given its derivation from EHR 
data.  
0% 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
13 
estimate derived from administrative data at NCH); second that these PCPs will contact the hospital for direct admission 
(a component of our direct admission intervention education); and third, that the PHM service will deem the child 
appropriate for direct admission and have a bed available. For this third criteria, we conservatively estimate that 80% of 
children referred for direct admission from PCPs will be accepted. Based on these assumptions, we estimate that 10% of 
children cared for by these PCPs will be directly admitted following rollout (Figure 2). Please note that our estimate that 
13% of children who are admitted to hospital will be seen by their PCP is conservative; our PCP stakeholder partners 
have estimated that between 15 and 50% of their admitted patients are seen in their clinics prior to admission. For 
children considered for direct admission from urgent care centers, we anticipate that eligible urgent care providers will 
call to request direct admission in 80% of cases, and that 80% of these patients will be accepted. Our projected sample 
size of children admitted through the ED takes into account our assumption that these children may be more acutely 
unwell and/or clinically unstable. Correspondingly, we estimate that 70% of children who are admitted through the ED 
at children’s hospitals with our target conditions will be eligible for inclusion, and that 95% of those who are admitted 
through the ED at our community hospital will be eligible for inclusion (given lower levels of acuity of children admitted 
to community hospitals than children’s hospitals). 
8. Sample Size and Power. Our projected sample size is shown in Table 5 and Figure 2. Based on the conservative 
assumptions above, we anticipate that 190 children and adolescents will be directly admitted and 1506 will be admitted 
through the ED. Of those admitted through the ED, 870 will be during post time-periods. Therefore, for analysis (i) there 
will be 190 direct admissions and 870 concurrent control ED admissions. Because the effect of all forms of clustering in 
analysis (i) of Aim 1 are minimal, we illustrate the statistical power for this analysis by using a two-group t-test. To 
inform our effect size determination, the PHM program at Lowell General Hospital (Lowell, MA), which has a well-
established direct admission program, collected timeliness data from April-July 2018, applying the same eligibility 
criteria and definition proposed for this study. ED timeliness data was similarly derived from Lowell and PMRC-E. We 
found that mean time to clinical care for ED admissions was 85 min [SD 83 min], and mean time for direct admissions 
was 40 minutes [SD 28 min].  
With the above means and standard deviations, a two-sided 0.05-level test that allows for unequal standard deviations 
performed in the context of analysis (i) of Aim 1 has power well in excess of 0.99. There is also substantial power for sub-
group analyses. For example, the power of a subgroup analysis that involves 10% of the sample (applied evenly across 
clinics and time-intervals) has power of 0.98. It is only when the subgroup is as small as 5% that power falls below 0.80. 
These power calculations can be performed using self-calculation or a variety of software packages. In this instance, it 
was performed using Satterwaite’s approximation (i.e., evaluating a pooled variance to approximate the sampling 
distribution of the actual test-statistic by a t-distribution) under nQuery. 
For analysis (ii) we only compute power for the effect of Post, the effect of a clinic receiving the intervention and 
entering the post period, for the analysis of the entire sample. The method of computing the power for this stepped-
wedge design (SWD) with unequal standard deviations in the prior and post time-periods is to first determine the 
design-effect for the SWD and then compute the power for a two-population comparison using the effective-sample-
sizes based on the design-effect. The design-effect is estimated using the expression in Woertman et al (2013), also 
described in Hemming (2016). Because hospital and practice group are cross-classified across the four SWD groups, and 
because little is known about likely levels of interclass correlation coefficients (i.e., the level of clustering) for each of 
hospital, practice-group and clinic, we perform illustrative power calculations assuming that the net impact of clustering 
is equivalent to hospital-level clustering alone with an ICC of hospital of 0.05. Based on results for past cluster-
randomized studies at the clinic level, an intraclass correlation of 0.05 appears to be a conservative estimate of 
clustering by clinic66,67 and so it is not unreasonable for 0.05 to still be conservative once the upper-level sources of 
cross-classified clustering are absorbed. The other inputs are 70 practices (clusters), 4 waves of intervention onset, no-
baseline period, 1,696 patients in total and equal numbers of patients per clinic per time-interval. Under this scenario 
the design-effect is 2.385. Therefore, the effective sample-size (ESS) is 711.  

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
14 
The second part of the calculation is to determine the power of a two-group comparison of a continuous outcome in the 
absence of clustering when the total sample-size is 711. Because there are 10 group-times in the post-period and 6 in 
the pre-period each with 106 observations, the split in the ESS between the post and the pre period is 444 and 267. The 
mean and standard deviation of the outcome in the pre-period is given by the above values for ED admissions; the mean 
is 85 and the standard deviation is 83. To determine the mean and standard deviation of the outcome expected in the 
post period, we take weighted averages of the DA and ED admission means and standard, using the total theorem of 
probability to compute marginal effects. The weight of DA to total patients is 19/106 = 0.094. The weighted average 
marginal mean and weighted average marginal standard deviation of the observations in the post period are 76.93 and 
84.99, respectively. Because the sample-sizes are reasonably large, an asymptotic normal approximation is well justified 
(i.e., the true distribution of the outcome is bimodal will be averaged out). Given the ESS’s and these means and 
standard deviations, a two-sided 0.05-level test has power of 0.23. In order to have 0.80 power, we would need a much 
larger effect-size. Specifically, the mean for the post-period would need to be approximately 66.5. Due to the small 
fraction of ED patients, this will only be plausible if there is a spillover effect of the intervention such that the outcome 
reduced for DA and ED patients, not just for DA patients. 
9. Outcomes. Our primary outcome, prioritized by multidisciplinary stakeholders including parents and healthcare teams 
in our preliminary work,39 will be timeliness of clinical care provision. Timeliness of clinical care will be defined as the 
time from arrival at the hospital until initiation of clinical care by the accepting healthcare provider (including diagnostic 
testing and/or medical management). This outcome will be derived from EHR time stamps; all project sites use EHRs to 
document times of patient registration, orders, administration of medications and therapies, and diagnostic tests. EHR 
time stamps have been previously validated using time motion data;68 during the pre-implementation period we will 
similarly validate the accuracy of time stamps by conducting time motion studies at each hospital.69 The feasibility of 
collecting timeliness outcome data is supported by preliminary data from Lowell General Hospital and PRMC-E, as 
described above (see Power Calculation, effect size estimation).  
Secondary outcomes will include: (i) parent-reported family experience of care (detailed below); (ii) unanticipated 
transfer to the ICU within 6 hours of hospital admission, a marker of clinical deterioration;38 and (iii) rapid response calls 
within 6 hours of hospital admission, a second marker of clinical deterioration defined as calls to the hospital’s medical-
emergency team placed by any person concerned about signs of critical clinical 
deterioration. Outcomes (ii) and (iii) will be extracted from the EHR; these are 
tracked routinely by PHM programs as quality markers.70,71  Family experience of 
care will be determined by caregiver/parent-report using questions derived from 
the Child Hospital Consumer Assessment of Healthcare Providers and Systems 
Survey (Child HCAHPS) hospital admission module and global ratings72 and the 
Consumer Quality Index Inpatient Hospital Care admission module.73 The Child 
HCAHPS is a validated and publicly available parent-reported measure of pediatric 
inpatient family experience of care, available in English and Spanish. It contains 
several modules and composite measures; to evaluate hospital admission 
experience we will use the hospital admission module, which focuses on 
communication and medication reconciliation during the hospital admission, as 
well as the global hospital rating. We will augment the Child HCAHPS with 
questions derived from the validated Consumer Quality Index Inpatient Hospital Care admission module (Table 6).73 
These parent-reported measures will be reported on a 0-100 scale, with high scores indicating better experiences of 
care. Six to seventy-two hours following hospital admission (and prior to hospital discharge), parents will be approached 
by a research team member, in person or via phone, to explain the study and request completion of the survey via tablet 
or via their personal electronic devices. This time period was selected to allow sufficient time to have passed to allow 
parents to reflect on their hospital admission, yet to minimize bias in their responses due to post-admission experiences. 
In our past studies we have had response rates of >75% using this approach.64,65 In contrast, when administered by mail 
Table 6.  Consumer Quality Index Inpatient 
Hospital Care admission module measures 
Your rights as patient (complaint procedure, 
etc.) 
What will happen during this hospitalization 
The person in the hospital whom you can 
contact if you have questions 
What medications you are taking 
Any dietary and nutritional requirements 
Any hypersensitivity to 
substances/medication 
Your provisional discharge date 
Your personal needs during the 
hospitalization 
Did the healthcare providers/ staff, upon 
arrival to the ward, have enough time for 
you? 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
15 
or telephone following hospital discharge, the Child HCAHPS had a response rate of <20%, which substantially limits the 
generalizability of findings.72 To account for all patients, we will categorize and report reasons for non-participation, 
including refusals, absence from the bedside, and inability to complete the survey in an available language. To facilitate 
future quality measure development, we will request all parents of children admitted directly from participating 
practices (with all diagnoses during the study period) to complete the family experience of care survey.  
 
10. Engagement Plan 
Planning the Study: Broadly speaking, key stakeholders in the hospital admission process include: (i) children and their 
families, (ii) referring healthcare providers, (iii) healthcare teams that provide hospital-based care (including ED 
physicians, pediatric hospitalists, resident physicians, and nurses), (iv) hospital administrators, (v) practice managers, (vi) 
payers, and (vii) policymakers (including professional societies such as the American Academy of Pediatrics). 
Stakeholders from these groups have played an integral role in prioritizing components of our proposed intervention, 
and in selecting the outcomes they deemed most relevant to evaluating this intervention. In our formative work, 
multidisciplinary stakeholders engaged in interviews, deliberative discussions and a Delphi process to identify the 
populations and systems appropriate for pediatric direct admission, and to prioritize outcomes to evaluate hospital 
admission processes.29,39 This preliminary work has directly informed our proposed intervention and outcomes.   
Conducting the Study: Members of the multistakeholder panel that developed the direct admission guidelines in our 
formative work will scaffold the proposed project as members of our DAAP, providing feedback on quarterly conference 
calls to guide project planning, implementation and analysis (Please see their LOS). Because this is a clinical intervention 
being implemented in real-world settings, implementation at each site will be lead by a PHM Clinical Leader (Co-
investigator). All of our Clinical Leaders are full time pediatric hospitalists who provide direct clinical care, actively 
engage with multidisciplinary stakeholders daily, and understand the clinical context. His/her roles will be supported by 
site-specific DALTs comprised of parents, PCPs, and healthcare providers who will participate in monthly implementation 
meetings to make key decisions about how to adapt the intervention to the local context, data collection considerations 
to maximize participation while minimizing disruptions to clinical operations, adherence to the intervention, and 
designing and revising monthly data reports and qualitative analysis summaries. As described above, Ms. Cathy Stevens, 
Parent Partner, will work with the research team to identify key concepts and themes in interview transcripts.  
Disseminating the Study Results:  We aim to disseminate our results broadly, providing results that are both meaningful 
and timely. The site-specific DALTs will, in their monthly meetings, design and revise monthly data reports and 
qualitative analysis summaries to ensure that data presentation is useful and meaningful. Paige Stein, TDI Director of 
Communications has also agreed to support these dissemination efforts via social media, infographics and press 
releases. Members of the DAAP have agreed to disseminate results through their professional networks (see their LOS). 
To reach pediatric hospitalists, we will disseminate findings via a PRIS network webinar. Beyond these strategies, the 
DAAP and DALTs will brainstorm additional means of disseminating results to key stakeholders beyond traditional 
methods. 
11. Timeline. The timeline for rollout of our direct admission intervention is summarized in Figure 2. During the first six-
month period, we will pursue IRB approvals and all sites will conduct a time-motion study to validate EHR time stamps 
and EHR data extraction protocols. In addition, all sites will convene their multi-disciplinary DALTs to adapt the direct 
admission intervention and educational materials as needed to their local context, and Clinical Leaders will conduct site 
visits with PCP practices randomized to the first group. The implementation period will span 45 months, with the final 6   
months of the funding period dedicated to analyses and results dissemination. 
 
 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
16 
 
 
 
REFERENCES CITED  
For detailed instructions, refer to the Application Guidelines for the PFA. Do not exceed 10 pages. 
 
Follow scholarly citation practice and list the source material cited in your Research Plan. PCORI suggests using 
American Medical Association citation style, but other citation styles are accepted. 
 
1.  
Institute of Medicine. Hospital-based emergency care: At the breaking point. Washington, DC: National 
Academies Press. 2006. 
2.  
Merrill C, Owens PL. Reasons for Being Admitted to the Hospital through the Emergency Department for Children 
and Adolescents, 2004. Healthcare cost and untilization project (HCUP) Statistical Brief #33.  
3.  
Merrill C, Owens PL. Hospital Admissions That Began in the Emergency Department for Children and Adolescents, 
2004. Healthcare costs and utilization project (HCUP) Statistical Brief Number 32. 
4.  
Tang N, Stein J, Hsia RY, Maselli JH, Gonzales R. Trends and characteristics of US emergency department visits, 
1997-2007. JAMA. 2010;304(6):664-670.  
5.  
Leyenaar JK, Shieh M-S, Lagu T, Pekow PS, Lindenauer P. Direct Admission to Hospitals Among Children in the 
United States. JAMA Pediatr. 2015;169(5):500-502. 
6.  
Schuur J, Venkatesh A. The growing role of emergency departments in hospital admissions. N Engl J Med. 
2012;367(5):391-393.  
7.  
Bourgeois FT, Monuteaux MC, Stack AM, Neuman MI. Variation in Emergency Department Admission Rates in US 
Children’s Hospitals. Pediatrics. 2014;134(3):539-545.  
8.  
Amit G, Cafri C, Gilutz H, Ilia R, Zahger D. Benefit of direct ambulance to coronary care unit admission of acute 
myocardial infarction patients undergoing primary percutanoues intervention. Int J Cardiol. 2007;119:355-358.  
9.  
Bryant M, Kelly A-M. “Point of entry” treatment gives best time to thrombolysis for acute myocardial infarction. 
Aust Heal Rev. 2001;24(1):157-160. 
10.  
Burns J, Hogg K, Rae A, Hillis W, Dunn F. Impact of a policy of direct admission to a coronary care unit on use of 
thrombolytic treatment. Br Hear J. 1989;61:322-325. 
11.  
Leurent G, Fougerou C, Pennec P, et al. Door-to-balloon delays before primary angioplasty in the Regional Acute 
Myocardial Infarction Registry of Brittany . An analysis of the Observatoire Régional Breton sur l’Infarctus du 
myocarde ( ORBI ). Arch Cardiovasc Dis. 2009;102:777-784.  
12.  
Majumder B, Mavroudis C, Smith C, Coghlan JG, Shiu M, Rakhit RD. Superior outcome with direct catheter 
laboratory access vs ED-activated primary percutaneous coronary intervention. Am J Emerg Med. 
2012;30(7):1118-1124.  
13.  
Melville MR, Hons BS, Gray D, Hinchley M. The Potential Impact of Prehospital Electrocardiography and 
Telemetry on Time to Thrombolysis in a United Kingdom Center. ANE. 1998;3(4):327-333. 
14.  
Van De Loo A, Saurbier B, Kalbhenn J, Koberne F, Zehender M. Primary Percutaneous Coronary Intervention in 
Acute Myocardial Infarction : Direct Transportation to Catheterization Laboratory by Emergency Teams Reduces 
Door-to-Balloon Time. Clin Cardiol. 2006;29:112-116. 
15.  
Carstensen S, Nelson GCI, Hansen PS, et al. Field triage to primary angioplasty combined with emergency 
department bypass reduces treatment delays and is associated with improved outcome. Eur Heart J. 
2007;28:2313-2319.  
16.  
Dorsch MF, Greenwood JP, Priestley C, et al. Direct ambulance admission to the cardiac catheterization 
laboratory significantly reduces door-to-balloon times in primary percutaneous coronary intervention. Am Heart 
J. 2008;155:1054-1058.  

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
17 
17.  
May MR Le, Davies RF, Dionne R, et al. Comparison of Early Mortality of Paramedic-Diagnosed ST-Segment 
Elevation Myocardial Infarction With Immediate Transport to a Designated Primary Percutaneous Coronary 
Intervention Center to That of Similar Patients Transported to the Nearest Hospital. Am J Cardiol. 2006;98:1329-
1333.  
18.  
Steg PG, Cambou J, Goldstein P, et al. Bypassing the emergency room reduces delays and mortality in ST 
elevation myocardial infarction : the USIC 2000 registry. Heart. 2006;92:1378-1383.  
19.  
Bång A, Grip L, Herlitz J, et al. Lower mortality after prehospital recognition and treatment followed by fast 
tracking to coronary care compared with admittance via emergency department in patients with ST-elevation 
myocardial infarction. Int J Cardiol. 2008;129:325-332.  
20.  
Joy A V, Adamowicz M, Furber R, Thomas B, Furber R. Reduction in treatment delay by paramedic ECG diagnosis 
of myocardial infarction with direct CCU admission. Heart. 1997;78:456-461. 
21.  
Ortolani P, Marzocchi A, Marrozzini C, et al. Clinical impact of direct referral to primary percutaneous coronary 
intervention following pre-hospital diagnosis of ST-elevation myocardial infarction. Eur Heart J. 2006;27:1550-
1557.  
22.  
Kocher KE, Dimick JB, Nallamothu BK. Changes in the source of unscheduled hospitalizations in the United States. 
Med Care. 2013;51(8):689-698.  
23.  
Powell ES, Khare RK, Courtney DM, Feinglass J. Lower mortality in sepsis patients admitted through the ED vs 
direct admission. Am J Emerg Med. 2012;30(3):432-439.  
24.  
Glasgow RE, Vogt TM, Boles SM. Evaluating the Public Health Impact of Health Promotion Interventions : The RE-
AIM Framework. Am J Public Health. 1999;89(9):1322-1327. 
25.  
Leyenaar JK, Ralston SL, Shieh M, Pekow PS, Mangione-Smith R, Lindenauer PK. Epidemiology of Pediatric 
Hospitalizations at General Hospitals and Freestanding Children ’ s Hospitals in the United States. J Hosp Med. 
2016;0(0):1-7.  
26.  
Lassman D, Hartman M, Washington B, Andrews K, Catlin A. US health spending trends by age and gender: 
selected years 2002-10. Health Aff (Millwood). 2014;33(5):815-822.  
27.  
Moore B, Levit K, Elixhauser A. Ph.D.Costs for Hospital Stays in the United States, 2012. Healthcare Cost and 
Utilization Project 181. Available at: http://www.hcup-us.ahrq.gov/reports/statbriefs/sb181-Hospital-Costs-
United-States-2012.pdf.  
28.  
Thomson J, Shah SS, Simmons JM, et al. Financial and Social Hardships in Families of Children with Medical 
Complexity. J Pediatr. 2016;172:187-193.e1.  
29.  
Leyenaar J, Pizzo P, O’Brien E, Lindenauer P. Paediatric hospital admission processes and outcomes: a qualitative 
study of parents’ experiences and priorities. BMJ Qual Saf. 2018;Epub ahead of print. 
30.  
Snow V, Beck D, Budnitz T, et al. Transitions of Care Consensus Policy Statement American College of Physicians-
Society of General Internal Medicine-Society of Hospital Medicine-American Geriatrics Society-American College 
of Emergency Physicians-Society of Academic Emergency Medicine. J Gen Intern Med. 2009;24(8):971-976.  
31.  
Leyenaar J, O’Brien E, Leslie L, Lindenauer P, Mangione-Smith R. Families’ priorities regarding hospital-to-home 
transitions for children with medical complexity. Pediatrics. 2017;139(1):pii: e20161581. 
32.  
Coleman E a. Safety in numbers: physicians joining forces to seal the cracks during transitions. J Hosp Med. 
2009;4(6):329-330.  
33.  
Jack BW, Chetty VK, Anthony D, Greenwald JL, Sanchez GM. A Reengineered Hospital Discharge Program to 
Decrease Rehospitalization: a randomized trial. Ann Intern Med. 2009;150:178-187. 
34.  
Kangovi S, Grande D. Hospital readmissions--not just a measure of quality. JAMA. 2011;306(16):1796-1797.  
35.  
Leyenaar JK, O’Brien ER, Malkani N, Lagu T, Lindenauer PK. Direct Admission to Hospital: A Mixed Methods 
Survey of Pediatric Practices, Benefits, and Challenges. Acad Pediatr. August 2015:1-8.  
36.  
Keng BMK, Thallner EA, Elson P, et al. Reducing Time to Antibiotic Administration for Febrile Neutropenia in the 
Emergency Department. J Oncol Pract. 2015;11(6):450-457. 
37.  
Leyenaar JK, Shieh M, Lagu T, Pekow PS, Lindenauer PK. Variation and outcomes associated with direct admission 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
18 
among children with pneumonia in the United States. JAMA Pediatr. 2014;168(9):829-836. 
38.  
Reese J, Deakyne S, Blanchard A, Bajaj L. Rate of preventable early unplanned intensive care unit transfer for 
direct admissions and emergency department admissions. Hosp Pediatr. 2015;5(1):27-34. 
39.  
Leyenaar JK, Rizzo P, Shevenell M, Hill V, Lindenauer P. Multi-stakeholder informed guidelines for pediatric direct 
admission to hospital. J Pediatr. 
40.  
Morganti KG, Bauhoff S, Blanchard J, et al. The Evolving Role of Emergency Departments in the United States. 
RAND Corporation; 2013. 
41.  
Gosdin C, Simmons J, Yau C, Sucharew H, Carlson D, Paciorkowski N. Survey of academic pediatric hospitalist 
programs in the US: organizational, administrative, and financial factors. J Hosp Med. 2013;8(6):285-291.  
42.  
Oshimura J, Sperring J, Bauer BD, Rauch D a. Inpatient staffing within pediatric residency programs: work hour 
restrictions and the evolving role of the pediatric hospitalist. J Hosp Med. 2012;7(4):299-303.  
43.  
Wachter R, Goldman L. Zero to 50,000 - the 20th Anniversary of the Hospitalist. N Engl J Med. 2016;375:1009-
1011. 
44.  
Fitch K, Bernstein SJ, Mcdonnell J, Kahan JP. The RAND / UCLA Appropriateness Method User ’ S Manual.; 2000. 
45.  
Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(3):166S-206S. 
46.  
Prabaker KK, Hayden MK, Weinstein R a, Lin MY. Use of the point of origin code from a universal billing form, UB-
04, to efficiently identify hospitalized patients admitted from other health care facilities. Am J Infect Control. 
2012;40(7):659-662.  
47.  
Leyenaar JK, Shieh M, Lagu T, Pekow P, Lindenauer PK. Variation and Outcomes Associated With Direct Hospital 
Admission Among Children With Pneumonia in the United States. JAMA Pediatr. 2014;168(9):829-836.  
48.  
Aarons GA, Green AE, Palinkas LA, et al. Dynamic adaptation process to implement an evidence-based child 
maltreatment intervention. Implement Sci. 2012;7(32):1-9.  
49.  
Simon TD, Cawthon ML, Stanford S, et al. Pediatric medical complexity algorithm: a new method to stratify 
children by medical complexity. Pediatrics. 2014;133(6):e1647-54.  
50.  
Hargreaves JR, Copas AJ, Beard E, et al. Five questions to consider before conducting a stepped wedge trial. 
Trials. 2015;16(350):1-4.  
51.  
Copas AJ, Lewis JJ, Thompson JA, Davey C, Baio G, Hargreaves JR. Designing a stepped wedge trial : three main 
designs , carry-over effects and randomisation approaches. Trials. 2015;16(352):1-13.  
52.  
Hemming K, Haines TP, Chilton PJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, 
analysis, and reporting. Br Med J. 2015;350:h391.  
53.  
Cummings P. Missing Data and Multiple Imputation. JAMA Pediatr. 2013;167(7):8-13.  
54.  
Rubin DB, Schenker N. Multiple imputation in health-care databases: an overview and some applications. Stat 
Med. 1991;10(4):585-598. 
55.  
Dahabreh IJ, Hayward R, Kent DM. Using group data to treat individuals : understanding heterogeneous 
treatment effects in the age of precision medicine and patient-centred evidence. Int J Epidemiol. 
2016;45(6):2184-2193.  
56.  
Kent DM, Rothwell PM, Ioannidis JPA, Altman DG, Hayward RA. Assessing and reporting heterogeneity in 
treatment effects in clinical trials : a proposal. Trials. 2010;11(85):1-10. 
57.  
Brookes ST, Whitely E, Egger M, Davey G, Mulheran PA, Peters TJ. Subgroup analyses in randomized trials : risks 
of subgroup-specific analyses ; power and sample size for the interaction test. J Clin Epidemiol. 2004;57:229-236.  
58.  
Guest G, Bunce A, Johnson L. How Many Interviews Are Enough?: An Experiment with Data Saturation and 
Variability. Field methods. 2006;18(1):59-82.  
59.  
Creswell J, Miller D. Determining validity in qualitative inquiry. Theory Pract. 2000;39(3):124-130. 
60.  
Creswell J. Qualitative Inquiry and Research Design: Choosing among Five Traditions. 3rd ed. Thousand Oaks, CA: 
Sage Publications Inc.; 2013. 
61.  
Morse JM. Determining Sample Size. Qual Health Res. 2000;10(1):3-5. 
62.  
Carlson JA. Avoiding Traps in Member Checking. Qual Rep. 2010;15(5):1102-1113.  

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
19 
63.  
Birt L, Scott S, Cavers D, Campbell C, Walter F. Member Checking : A Tool to Enhance Trustworthiness or Merely a 
Nod to Validation ? Qual Health Res. 2016;26(13):1802-1811.  
64.  
Kascak L, Leyenaar JK. Sleep quality during pediatric hospital stays: a pilot study of patient experiences. 
Presented at Academic Pediatric Association Region 1 Meeting, Waltham MA, March, 2018. 
65.  
Leyenaar JK, Shevenell M, Rizzo P, O’Neill M, Blaine K, Hussain, K, Mangione-Smith R, Berry JG. Hospital-to-home 
needs assessment for children with medical complexity: a pilot study of feasibility and acceptability. Pediatric 
Academic Societies Meeting, San Francisco, May 2017. 
66.  
Kul S, Vanhaecht K, Panella M. Intraclass correlation coefficients for cluster randomized trials in care pathways 
and usual care: hospital treatment for heart failure. BMC Health Serv Res. 2014;14(84):1-7.  
67.  
Thompson DM, Fernald D, Mold J. Intraclass Correlation Coefficients Typical of Cluster-Randomized Studies : 
Estimates From the Robert Wood Johnson Prescription for Health Projects. Ann Fam Med. 2012;10:235-240.  
68.  
Arndt B, Beasley J, Watkinson M, et al. Tethered to the EHR: Primary Care Physician Workload Assessment Using 
EHR Event Log Data and Time-Motion Observations. Ann Fam Med. 2017;15:419-427. 
69.  
Lopetegui M, Yen P-Y, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking 
about? J Biomed Inf. 2014;49:292-299.  
70.  
Humphreys S, Totapally BR. Rapid response team calls and unplanned transfers to the pediatric intensive care 
unit in a pediatric hospital. Am J Crit Care. 2016;25(1):9-14. 
71.  
Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of Rapid Response System 
Implementation on Critical Deterioration Events in Children. JAMA Pediatr. 2014;168(1):25-33.  
72.  
Toomey SL, Zaslavsky AM, Elliott MN, et al. The Development of a Pediatric Inpatient Experience of Care 
Measure: Child HCAHPS®. Pediatrics. 2015;136(2):360-369.  
73.  
Smirnova A, Lombarts KMJMH, Arah OA, Van Der Vleuten CPM. Closing the patient experience chasm : A two- ­ 
level validation of the Consumer Quality Index Inpatient Hospital Care. Heal Expect. 2017;20:1041-1048.  
74.  
Leyenaar J, Bergert L, Mallory L, et al. Pediatric Primary Care Providers’ Perspectives Regarding Hospital 
Discharge Communication: A Mixed Methods Analysis. Acad Pediatr. 2015;15(1):61-68. 
75.  
Leyenaar JK, Rizzo PA, Khodyakov D, Leslie LK, Lindenauer PK, Mangione-smith R. Importance and Feasibility of 
Transitional Care for Children With Medical Complexity : Results of a Multistakeholder Delphi Process. Acad 
Pediatr. 2018;18(1):94-101.  
76.  
Leyenaar JK, Andrews CB, Tyksinski ER, Biondi E, Parikh K, Ralston S. Facilitators of interdepartmental quality 
improvement : a mixed- methods analysis of a collaborative to improve pediatric community- acquired 
pneumonia management. BMJ Qual Saf. 2018.  
77.  
Coghlin DT, Leyenaar JK, Shen M, et al. Pediatric discharge content: a multisite assessment of physician 
preferences and experiences. Hosp Pediatr. 2014;4(1):9-15.  
78.  
Bruce ML, Lohman MC, Greenberg RL, Bao Y, Raue PJ. Integrating Depression Care Management into Medicare 
Home Health Reduces Risk of 30- and 60-Day Hospitalization : The. J Am Geriatr Soc. 2016;64:2196-2203.  
79.  
Bruce ML, Raue PJ, Reilly CF, et al. Clinical Effectiveness of Integrating Depression Care Management Into 
Medicare Home Health The Depression CAREPATH Randomized Trial. JAMA Intern Med. 2015;175(1):55-64.  
80.  
Desai AD, Burkhart Q, Parast L, et al. Development and Pilot Testing of Caregiver-Reported Pediatric Quality 
Measures for Transitions Between Sites of Care. Acad Pediatr. 2016;16(8):760-769.  
81.  
Parast L, Burkhart Q, Desai AD, et al. Validation of New Quality Measures for Transitions Between Sites of Care. 
Pediatrics. 2017;139(5):e20164178. 
82.  
Desai AD, Simon TD, Leyenaar JK, et al. Utilizing Family-Centered Process and Outcome Measures to Assess 
Hospital-to-Home Transition Quality. Acad Pediatr. 2018.  
83.  
Leyenaar JK, Desai AD, Burkhart Q, Parast L, Roth CP. Quality Measures to Assess Care Transitions for 
Hospitalized Children. Pediatrics. 2016;138(2):1-9. 

 
 
 
PRINCIPAL INVESTIGATOR: Leyenaar, JoAnna, Kristine 
 
 
 
PCORI Cycle 2 2018 Broad PFA: Research Plan Template                                                                        
20 
 
APPENDIX (optional)  
For detailed instructions, refer to the Application Guidelines for the PFA. Do not exceed 10 pages.