Using the Electronic Health Record to Identify and Promote Goals-of-Care Communication for Older Patients With Serious Illness
Phase: NA
Conditions: Dementia, Chronic Disease, Neoplasm Metastasis, Lung Neoplasm, Pulmonary Disease, Chronic Obstructive, Heart Failure，Congestive, Liver Cirrhosis, Kidney Failure, Chronic, Lung Diseases, Interstitial, Peripheral Vascular Disease, Diabetes With End Organ Injury
Sponsor: University of Washington

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
1 
 
Official Study Title: Project to Improve Communication About Serious Illness--Hospital Study: Pragmatic Trial 
(Trial 1) 
 
NCT Number: NCT04281784 
 
Document date: 04/25/22 
 
 
 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
2 
 
 
Study Protocol: Project to Improve Communication About Serious Illness--Hospital Study (PICSI-H) 
 
Principal Investigator:  J. Randall Curtis, MD, MPH 
Co-Principal Investigator: Ruth A. Engelberg, PhD 
 
Cambia Palliative Care Center of Excellence 
University of Washington, Seattle WA 
 
Table of Contents 
1) 
Brief Overview ......................................................................................................................................... 3 
2) 
Specific Aims: .......................................................................................................................................... 3 
3) 
Research Plan .......................................................................................................................................... 4 
4) 
Research Design and Methods ................................................................................................................ 5 
5) 
Outcomes ................................................................................................................................................ 8 
6) 
Analyses ................................................................................................................................................. 10 
7) 
Sample size ............................................................................................................................................ 11 
8) 
Data management and quality control to achieve scientific rigor ........................................................ 11 
9) 
Protocol modifications .......................................................................................................................... 11 
10) 
Potential limitations and alternative approaches ............................................................................. 12 
11) 
Anticipated findings .......................................................................................................................... 13 
 
 
 
 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
3 
 
1) Brief Overview 
 
For hospitalized patients with chronic illness, a key component of high-quality care includes goals-of-care 
discussions (GOCD) conducted early during a hospital stay to identify how patients’ goals of care should inform 
current care plans. These goals-of-care discussions are associated with improved patient and family outcomes 
and reduced intensity of care at the end of life. Despite their importance, GOCD during a hospitalization often 
do not occur. The failure to conduct these conversations and to ensure that care received is aligned with 
patients’ wishes for care is particularly a concern for older adults with chronic illness, and particularly those 
with Alzheimer’s disease and related dementias (ADRD).  
 
This study, funded by NIA, evaluates the Jumpstart intervention, the key to which is the “Jumpstart Guide.” 
There are two versions of the Jumpstart Guide being tested, one that is “clinician-facing” and includes 
information identifying the dates of prior advance care planning documents (e.g., living wills, healthcare 
directives, durable power of attorney for healthcare) and Physician Orders for Life Sustaining Treatments 
(POLST forms) in the electronic health record (EHR) to the clinician. It also includes “just-in-time” suggestions 
for having a GOCD. The other guide is “patient-specific” and “bi-directional”.  It is patient-specific because it 
provides goals of care information from patient’s self-reported surveys and is then shared with his/her 
clinician.  It is bi-directional because there is one Jumpstart that goes to the clinician and another version that 
goes to the patient or, if the patient doesn’t have decisional capacity, a family member. These one-page 
Jumpstart Guides include not only indications of prior advance care planning documents (both guides) and 
preferences (patient-specific guide only) but also tips to improve this communication that are tailored either 
for clinicians or patients and family members.  
 
The study comprises two linked, complementary randomized trials.  Trial 1, a large pragmatic trial, compares 
usual care with the clinician-facing Jumpstart for hospitalized older adults with serious illness (target n=2000; 
ADRD subset, target n=400). Trial 2, a comparative effectiveness trial is a three-arm trial that compares the 
clinician-facing Jumpstart, the bi-directional, patient-specific Jumpstart and usual care (n=600). Subjects are 
enrolled from the UW Medicine hospitals (Harborview Medical Center, UW Medical Center - Montlake, UW 
Medical Center – Northwest).  For Trial 1, eligible patients are ≥ 55 years, admitted for a minimum of 12 and a 
maximum of 96 hours prior to study enrollment to inpatient services, without a documented GOCD during 
admission, and meet criteria for serious illness; hospitalized patients >80 years are also eligible. For Trial 2, 
eligible patients meet the same criteria and, in addition, must be sufficiently fluent in English to complete 
questionnaires and not be under COVID precautions to allow in-person recruitment.  For Trial 2, eligible family 
may include any of the following:  legal guardians, durable power of attorney for healthcare, spouses, adult 
children, parents, siblings, domestic partners, other relatives, and friends. In addition, we will conduct 
qualitative interviews with key stakeholders including patients/families (n=40, subgroup of patients enrolled 
for Trial 2) and clinicians (n=50 across Trial 1 and Trial 2). 
 
2) Specific Aims: 
 
Specific Aim 1 (Trial 1): Evaluate the effectiveness of a novel clinician-facing Jumpstart, compared with usual 
care, for improving the quality of care; the primary outcome is EHR documentation of a goals-of-care 
discussion during the 30 days after enrollment. Secondary outcomes focus on intensity of care: ICU use, ICU 
and hospital free days, costs of care during the hospitalization, and 7- and 30-day hospital readmission. 
 
Specific Aim 2 (Trial 2): Evaluate the efficacy of the bi-directional, patient-specific Jumpstart compared to the 
clinician-facing Jumpstart and usual care for improving quality of care; the primary outcome is EHR 
documentation of a goals-of-care discussion 30 days after enrollment.  Secondary outcomes include: intensity 
of care outcomes from Aim 1 and patient- and family-reported outcomes assessed by surveys at 3-5 days and 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
4 
4-6 weeks after randomization, including occurrence and quality of goals-of-care discussions in the hospital, 
goal-concordant care, psychological symptoms, quality of life, and palliative care needs.  
 
Specific Aim 3:  Conduct a mixed-methods evaluation of the implementation of both interventions, guided by 
the RE-AIM framework for implementation science, incorporating quantitative evaluation of intervention reach 
and adoption, as well as qualitative analyses of interviews with participants, to explore barriers and facilitators 
to future implementation and dissemination.   
 
3) Research Plan 
 
a) Background and Significance 
 
People near the end of life often receive care they would not choose.48,49 The National Academy of 
Medicine has documented these discrepancies in care and identified advance care planning and goals-of-care 
discussions as primary mechanisms for addressing them.48 This type of communication is a focus for 
improvement for two key reasons: 1) when goals-of-care discussions occur, they are associated with improved 
quality of care and patient- and family-centered outcomes including increased quality of life, reduced 
symptoms of psychological distress, and fewer intensive treatments at the end of life;1,7,50,51 and 2) clinicians 
frequently do not have goals-of-care discussions with their patients until very late in the illness.1,3-6 
 
The value of advance care planning discussions with healthy individuals is a topic of debate; however 
improving goals-of-care discussions for those with serious illness facing difficult treatment decisions is widely 
agreed upon as an urgent need that can improve patient outcomes.1-6 There is an emerging consensus on this 
important distinction between ACP for healthy individuals and goals-of-care discussions for those with chronic 
life-limiting illness and on the critical importance of timely goals-of-care discussions.1,7,8 Furthermore, even if 
advance care planning does occur in the outpatient setting, effective goals-of-care discussions (a component of 
advance care planning for more proximal decision-making) are still needed for hospitalized patients whose 
prior preferences may have changed or may not have been specific to the current circumstances.52,55,56 In 
short, for hospitalized patients with chronic illness, a key component of high quality care includes goals-of-care 
discussions conducted early during a hospital stay that build on prior advance care planning and identify how 
patients’ goals inform current care.4,53,57 These early hospital discussions are supported by the National Quality 
Forum.58 Despite their key importance to a large number of patients, early hospital goals-of-care discussions 
often do not occur.4,59 A recent research agenda for serious illness communication, supported by the National 
Institute on Aging and published in JAMA Internal Medicine, highlights the importance of promoting high-
quality goals-of-care discussions, as well as the potential opportunity to use the EHR to both identify those 
patients who would benefit from goals-of-care discussions and to guide clinicians in high-quality discussions.8 
We propose two complementary trials to examine the effectiveness of such interventions, and we use an 
innovative hybrid effectiveness-implementation approach that evaluates the interventions and their 
implementation.23 
 
b) Innovation 
 
Use of the EHR to identify seriously ill, hospitalized patients without a goals-of-care discussion: Recent research 
agendas highlight the lack of research utilizing the EHR to implement interventions that improve serious illness 
communication.8,9 We will use a validated EHR-based quality metrics program to identify hospitalized patients 
over age 55 with chronic serious illness who do not have EHR documentation of a goals-of-care discussion. We 
have developed an innovative NLP/ML protocol to identify inpatient and outpatient documentation of goals-
of-care discussions in any type of EHR note, including admission notes, progress notes, and discharge 
summaries (see preliminary data). In this way, we not only target a population likely to benefit from the 
intervention but also do so with methods that are generalizable and scalable.105 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
5 
 
Examine both a clinician-facing Jumpstart and a patient-specific Jumpstart in an innovative study design: The 
intervention is based on our recently completed trial in the outpatient setting of the patient-specific 
intervention (Jumpstart), which is an individualized communication-priming intervention, targeting both 
patients and clinicians and providing each with information obtained from patient surveys in order to guide a 
goals-of-care discussion.2 This intervention is innovative because it is one of the few that involves clinicians as 
well as patients and family members (bi-directional). However, a question raised by reviewers of our prior trial2 
was whether the individualized survey-based component was necessary, especially given the resources needed 
to implement it. Therefore, we have developed this innovative study design that combines a large pragmatic 
trial of a clinician-facing Jumpstart compared to usual care (using a waiver of informed consent to facilitate 
enrollment and the pragmatic approach) and a smaller comparative effectiveness trial of a bi-directional, 
patient-specific Jumpstart that will include individual patient or surrogate consent for participation as well as 
completion of surveys to generate the patient-specific Jumpstart. 
 
Develop an innovative effectiveness-implementation approach that advances implementation and 
dissemination: Few evidence-based communication interventions are widely adopted. Barriers to 
implementation include factors at the level of individual patients, clinicians, operating clinical units, and 
healthcare systems.106,107 The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) 
framework systematically addresses these factors and, in turn, translates interventions like Jumpstart into the 
“real world”.24-28 This study will include a novel hybrid effectiveness-implementation approach that can 
accelerate implementation and dissemination of the interventions after the study by allowing us to evaluate–
during the study–strategies and outcomes that will facilitate uptake of the interventions in the future.23,28,108-110 
This innovative design offers the opportunity to advance implementation science in palliative care and 
increases the utility of this study. 
 
4) Research Design and Methods 
 
a) Overview 
 
We will conduct two linked, complementary randomized trials of a clinician-facing or a bi-directional, 
patient-specific intervention to promote and guide goals-of-care discussions for older, seriously ill, hospitalized 
patients using an innovative method for identifying eligible participants through the EHR. Trial 1 assesses 
effectiveness of the clinician-facing Jumpstart compared with usual care in a large pragmatic trial not requiring 
patient or family consent (Aim 1, Trial 1). Trial 2 examines the efficacy of the patient-specific Jumpstart Guide 
as compared with the clinician-facing Jumpstart and usual care (Aim 2, Trial 2). In addition, we will examine the 
implementation of the interventions using the RE-AIM framework24-27 (Aim 3). Figure 1 provides an overview of 
the design. 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
6 
 
 
b) Setting 
 
We will conduct this study with a diverse population drawn from 3 hospitals at UW Medicine, including 
a university hospital (UW Medical Center, UWMC), a county, safety-net hospital (Harborview Medical Center, 
HMC), and a community hospital (Northwest Hospital and Medical Center, NWHMC). UWMC provides 
specialist care for the Pacific Northwest region and has 529 acute care beds and 75 ICU beds. HMC is operated 
by the University of Washington and has 413 acute care beds and 94 ICU beds. HMC is the only Level 1 Trauma 
Center serving five states, and its mission population includes inner city poor, recent immigrants, and persons 
with HIV/AIDS. NWHMC is a community hospital with 218 acute care beds and 15 ICU beds, serving north 
Seattle with a large geriatric and nursing home resident population. Our prior studies have included each of 
these sites.29,30,79 These 3 sites offer the advantage of caring for diverse patients while also using a unified EHR 
incorporating EPIC systems into a platform we have been able to access and use.16-21  
 
c) Patient population 
 
Eligible patients over 55 will be identified by ICD-10 codes for one or more of the nine chronic conditions used 
by the Dartmouth Atlas:134 dementia, malignant cancer/leukemia, chronic pulmonary disease, coronary artery 
disease, heart failure, chronic liver disease, chronic renal disease, diabetes with end-organ damage, and 
peripheral vascular disease. These nine conditions account for 90% of deaths among Medicare beneficiaries in 
the US.135,136 To increase inclusivity of important and under-studied populations, we will also include all 
hospitalized patients over age 80. Among patients meeting these criteria, we will include only those with no 
identified documentation of goals-of-care discussions during the current hospitalization and prior to 
enrollment. For Trial 2, eligible patients or their surrogates will have sufficient English language proficiency to 
complete surveys.  
 
d) Description of participants 
 
 
 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
7 
We will collect the following information about study participants.  
 
Patients: Age, race, ethnicity, sex, comorbidities and acute severity of illness including the National Early 
Warning Score (NEWS). For Trial 2, we will additionally collect education, marital status, health status (SF-1) 
and income.  
 
Families: Age, race, ethnicity, sex, education, marital status, health status (SF-1), relationship to patient, living 
situation related to patient.  
 
Clinicians: Age, race, ethnicity, sex, education.  
 
e) Sampling patients with ADRD 
 
Given the dramatically increasing prevalence of ADRD in the US and the rising intensity of care among these 
patients,11,13 it is particularly important that we understand the effect of interventions to improve quality of 
palliative care in this group. We designed both trials to be powered to examine heterogeneity of treatment 
effects (HTE) in this important group. For Trial 1, we will plan to continue recruitment to enhance power in this 
subgroup. For Trial 2, we will prioritize recruitment of patients with ADRD to maximize this subgroup sample 
size. We hypothesize that the interventions will be equally effective in this population, but that the proportion 
of patients with decisional capacity will be lower (enrolling more family members). However, we believe it is 
important to explicitly examine this hypothesis given the unique features of intensity of care at the end of life 
for this important and increasing population. 
 
f) Randomization 
 
Patients are randomized in a 1:1 ratio in Trial 1 and 1:1:1 ratio in Trial 2 using variable size blocks and stratified 
for hospital and ADRD vs. no ADRD. Participating family members or legal surrogate decision makers are 
assigned to the same arm as the corresponding patient.   
 
g) Intervention 
 
Clinician-facing Jumpstart:  First, we use automated methods to examine inpatient and outpatient EHR notes 
prior to the current admission, identifying current code status as well as all prior POLST forms and advance 
directives; this information is included on Jumpstart Guides to inform discussions. Second, we deliver the 
Jumpstart Guide to the primary hospital team (all attending and resident physicians and advanced practice 
providers) via secure email and either a page (Trial 1) or Epic SecureChat message (Trial 2) alerting the 
physicians to the presence of the Jumpstart Guide in their email, with the addition of in-person delivery of a 
paper version in Trial 2 only.     
 
Bi-directional Jumpstart (Trial 2 only):  First, information about the patient is abstracted from the EHR in the 
same way as for the Clinician-facing Jumpstart. Second, patients or their legal surrogate decision-maker 
complete baseline survey items assessing three domains: a) preferences for discussions about goals of care; b) 
barriers and facilitators for having such discussions; and c) current goals of care. Third, using the EHR and 
baseline survey, we use the automated algorithm from our prior trial9 adapted to the hospital setting using 
human-centered design methods10 to create a survey-informed Jumpstart Guide to prompt and guide goals-of-
care discussions between the patient and hospital team or, if the patient isn’t able, the family member and the 
hospital team (see supplement for sample Jumpstart Guide). Finally, in the fourth step, we deliver the 
Jumpstart Guides to the primary team via secure email similar to Trial 1, as well as in-person delivery to 
members of the team.  We also provide a survey-informed Jumpstart Guide to the patient or family, adapted 
with phrasing and terminology specifically for the patient and family.  All Jumpstart Guides are delivered on 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
8 
the day of randomization with the goal of prompting a goals-of-care discussion early during hospitalization, as 
supported by the National Quality Forum.11 The bi-directional Jumpstart includes both EHR- and patient-
tailored suggestions for conducting goals-of-care discussions based on survey responses.  The suggestions are 
guided by the educational experience of VitalTalk, a nationally-acclaimed program for teaching serious illness 
communication, and adapted to the inpatient setting,12,13 as well as by a human-centered design exercise with 
hospital clinicians.10 
 
 
5) Outcomes 
 
a) Outcomes from the EHR and Death Certificates (Trials 1 and 2): 
 
The primary outcome for both trials is EHR documentation of goals-of-care discussions within 30 days after 
randomization. Our rationale for this as the primary outcome is that this is the primary target for all 
interventions and important to diverse stakeholders including patients and their families.1,14-17 We will use 
NLP/ML methods to identify goals-of-care discussions.18  We will manually review the EHR for goals-of-care 
discussions using our standard EHR abstraction methods19-21 for a randomly selected subset of patients in each 
trial to evaluate potential misclassification with  NLP/ML methods. 
  
Additional outcomes for both trials, obtained from the EHR, include utilization metrics associated with 
intensity of care (i.e., any ICU admissions, any ED visits, any palliative care consultations, and ICU- and hospital-
free days); these outcomes will be assessed at 30 and 90 days after randomization. ICU- and hospital-free days 
are defined as the number of days alive and outside of the ICU (or hospital) within the specified time period 
after randomization (i.e. 30 days or 90 days).22,23 We will also examine the following outcomes: 1) time to first 
goals-of-care discussion during the 30 days after randomization; 2) occurrence of any hospital readmissions 
within 7 and 30 days after discharge from the index hospitalization; and 3) mortality status at 90 days and 1 
year after randomization. Costs of care during hospital admission and 30- and 90-days following randomization 
will be obtained from institutional billing systems.  Washington State death certificate data will be used to 
examine mortality after hospital discharge (Table 1).  
 
b) Outcomes derived from patient- and family-reports (Trial 2 only): 
 
Additional outcomes for Trial 2 will be obtained from patient or family surveys. Surveys will be completed 
targeting three time points: 1) baseline; 2) 3-5 days after randomization; and 3) 4-8 weeks after randomization. 
Surveys may be completed in person, online, by mail, or by phone, based on respondents’ preferences.    
 
Occurrence and quality of discussions: We use previously validated items to assess the occurrence and quality 
of goals-of-care communication during the hospitalization after randomization.9,24-30 Communication 
occurrence is assessed with a single item.9,25 Quality of goals-of-care communication is assessed with the end-
of-life communication scale (QOC_eol) of the Quality of Communication (QOC) survey, developed from 
qualitative interviews and focus groups with a diverse group of patients, families, and clinicians.24,26,28  
 
Goal-concordant care: Concordance between the care patients want and the care they are receiving will be 
measured with two questions from SUPPORT.31 The first question defines patients’ priorities for extending life 
or ensuring comfort. The second question assesses patients’ perceptions of their current treatment using the 
same two options.31 Concordance is defined as a match between preference for care and the type of care 
currently received, as reported by patients or families.  Although most patients want both quality and life-
extending care, requiring respondents to pick one is a useful way to identify patients’ top priority.32-34 If 
patients are unable to respond, goals of care are elicited from family as they would be in clinical practice.35   
 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
9 
Symptoms of anxiety and depression: Patient and family symptoms of anxiety and depression are assessed 
with the Hospital Anxiety and Depression Scale (HADS).36,37 Patients and families will complete these surveys 
for themselves only; we do not ask for surrogate report of patients’ psychological symptoms.  The goal is not to 
diagnose the clinical syndromes of anxiety or depression, but rather to identify the burden of symptoms.  
 
Utilization:  In addition to measuring hospital readmissions through the EHR, we will use patient or family 
reports of patient emergency department visits, hospitalizations, and outpatient visits following hospital 
discharge.  By using both sources of data, we expect to capture utilization that occurs outside of UW Medicine. 
 
c) Implementation outcomes (Trials 1 and 2)  
 
Assessment of the implementation of the interventions in Aim 3 is guided by the RE-AIM Framework for 
implementation research38-41 and the Consolidated Framework for Implementation Research (CFIR).42 RE-AIM 
is a multidimensional framework for evaluating the public health impact of efforts to translate research into 
practice.39 The five dimensions of RE-AIM are reach of the intervention within the target population, 
effectiveness of the intervention, adoption by target staff members or settings, implementation consistency 
and quality, and maintenance of intervention delivery and effects.38-41 CFIR is a pragmatic meta-theoretical 
framework that synthesizes constructs related to implementation of evidence-based interventions. The five 
overarching domains are intervention characteristics, outer setting, inner setting, characteristics of individuals, 
and process, and include a total of 37 constructs that can be used to understand what works, and why, in a 
certain setting.42 We collect quantitative and qualitative data on reach, effectiveness, adoption, 
implementation, and maintenance (RE-AIM) of the intervention. Quantitative data are collected as routine 
tracking as part of the implementation of both trials, with data on participation, intervention use, fidelity to 
the intervention, and changes over time (Table 2). Qualitative data are collected through short, semi-
structured interviews (10-30 minutes) guided by the CFIR domains.  The interviews are conducted in-person or 
by phone with patients (n=20) and family members (n=20) from Trial 2, and clinicians (n=50) from either Trial 1 
or 2 after study involvement.  All participants are selected using purposive sampling to ensure a diverse group 
based on level of participation, race, ethnicity, age, gender and, for clinicians, specialty, and year of training. A 
trained qualitative interviewer will interview participants using an interview guide, and interviews will be 
audio-recorded and transcribed.16,43-59.  
 
Table 1: Outcome measures and data collection 
MAJOR OUTCOME MEASURES 
CONCEPT 
DATA COLLECTION:  
SOURCE & TIME 
Aims 1 and 2 outcomes 
EHR documentation of goals-of-
care discussion (Primary outcome 
for both Trials) 
Goals-of-care discussion 
EHR: 30 days post randomization 
ICU use, ICU and hospital free days 
Intensity of care 
EHR: 30 days post randomization 
7- and 30-day ICU and hospital 
readmissions 
Intensity of care 
EHR: 7 and 30 days following hospital 
discharge 
Costs of care 
Intensity of care/intervention 
costs 
EHR: During hospital stay, 30 and 90 
days post-randomization. 
All-cause mortality at 90 days and 
1 year 
All-cause mortality 
Washington State death certificates 
Aim 2 outcomes (not used in Aim 1 since the Trial 1 is a pragmatic trial without contact with patients or 
family members) 
Patient/family-reported discussion 
of goals9,25 
Goals-of-care discussion 
occurrence 
Survey: 3-5 days & 4-6 weeks post-
randomization 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
10 
Quality of Communication 
(QOC)24,26,28 
Quality of communication 
Survey: 3-5 days post randomization 
SUPPORT question31 
Goal-concordant care 
Survey: 3-5 days & 4-6 weeks post-
randomization 
HADS – anxiety and depression36,37 
Symptoms of anxiety & 
depression 
Survey: 4-6 weeks post-randomization 
EQ-5D-5L  
Health-related QOL 
Survey: 4-6 weeks post-randomization 
Patient/family reported ED, 
hospitalization and outpatient 
utilization  
Healthcare utilization 
Survey: 4-6 weeks post-randomization 
CollaboRATE60 
Shared decision-making 
Survey: 3-5 days post randomization 
 
 
 
6) Analyses 
 
We will follow the intention-to-treat principle for all analyses.  
 
a) Primary Outcome (presence of goals of care discussion within 30 days after randomization) 
 
The effect of intervention on the primary outcome will be quantified by the difference in proportions and 
evaluated with a linear regression model with robust standard errors. The predictor of interest is 
randomization arm (Clinician-facing Jumpstart or usual care for Trial 1; or Clinician-facing Jumpstart, Bi-
directional Jumpstart, or usual care for Trial 2). The model will adjust for hospital site and ADRD status, since 
randomization is stratified on these factors. This model assumes the effect of intervention is the same for 
patients with and without ADRD. We will also include an interaction between randomization arm and ADRD, 
which allows the effect of intervention to vary by ADRD status and allows evaluation of the effect among those 
with and without ADRD.  We will evaluate the timing of goals-of-care discussions with a Cox proportional 
hazards model.  
 
b) Additional Outcomes 
 
For the analysis of the other outcomes, we will use a strategy similar to that for the primary outcome. For 
continuous outcomes (e.g., ICU-free days, HADS score), the effect of intervention will be quantified by a 
difference in means. For survey outcomes which are collected at more than one time point after 
randomization, we will use a mixed model to account for the correlation between repeated measures. Our 
initial model will allow the average response to be different at each time point, but assume the intervention 
has the same effect at each time. We will also allow the effect of intervention to be different across time by 
including an interaction between time and intervention. The advantage of using the data at the multiple time 
points and a mixed model approach is that we can gain precision; it also allows for missing responses, 
assuming responses are missing at random. Missing data are more of an issue for the survey outcomes than 
the primary outcome; we will quantify the amount and type of missing data, evaluate associations of 
missingness with participant characteristics, and apply appropriate methods to account for missing data.61 
 
c) Evaluate implementation and identify barriers and facilitators to future implementation. 
 
We will perform thematic content analysis of transcribed interviews to explore feedback on the intervention, 
ways to improve intervention implementation, and aspects of care not adequately addressed by the 
intervention.62-64 Interview guides and analyses will be guided by the RE-AIM and CFIR frameworks as 
described above.38-42 Qualitative data will be imported to analytic software (Dedoose), where investigators will 
perform the following analytic steps using an iterative approach to thematic analysis65: 1) initially code 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
11 
material, devising a coding framework and using that framework to reduce the text into smaller segments; 2) 
identify themes from the coded text; 3) construct thematic networks that include basic themes, organizing 
themes and global themes; 4) describe and summarize thematic networks; and 5) interpret patterns that have 
emerged in and across thematic networks.  
 
7) Sample size  
 
a) Sample size considerations for the primary outcome   
 
The focus for sample size considerations is the primary outcome: proportion of patients with documented 
goals-of-care discussions within 30 days after randomization. 
 
Trial 1: With a total sample size of 2000 (1000 per group), two-sided significance level (α) of 0.05, and a 
variance estimate based on the proportion in the control group only, we have 80% power to detect a 
difference in proportions between those randomized to Clinician-facing Jumpstart and usual care of at least 
0.06. We assumed a proportion in the control group of 0.54 based on preliminary data. If the total number of 
patients with ADRD in Trial 1 is 400 (200 per group), we would have 80% power with α=0.05 to detect a 
difference in proportions of 0.14 among those with ADRD. 
 
Trial 2: With a total sample size of 600 (200 per Clinician-facing Jumpstart, 200 per Bi-directional Jumpstart, 
and 200 per usual care), we have 80% power to detect a difference in proportions of 16% for each of the 3 
pairwise comparisons assuming an overall α=0.05 and a Bonferroni adjustment for the 3 comparisons (α=0.017 
for each comparison) and variance based on a proportion of 0.54.    
 
b) Sample size for qualitative analyses 
 
For Aim 3 qualitative analyses, it is important to achieve theoretical saturation (no new themes emerging).64,66 
Based on our prior studies, we anticipate achieving saturation by 80 interviews for understanding 
patients/families and clinician perspectives.16,46,54-59 We will monitor for saturation and will recruit additional 
participants if needed. 
  
8) Data management and quality control to achieve scientific rigor 
 
This project requires the creation, maintenance, and analysis of a database that includes a variety of measures 
from multiple sources.  This study, like all studies, depends on the quality of the data and therefore systematic 
data collection, quality control, and data-management procedures will be implemented: 1) protocols for data 
collection; 2) rigorous training, certification, and periodic re-training of study staff, with ongoing monitoring of 
adherence to protocols; 3) regular review of questionnaire response rates, respondent burden,67 and missing 
items to identify and correct problems; 4) verification of all data through custom-designed data entry systems; 
and 5) weekly team meetings to provide feedback to study staff to ensure problems are resolved quickly. To 
ensure reliability and validity of data, we will use our current methods for training and quality control.68-72 Staff 
conducting EHR review will undergo the following training: instruction on the protocol, guided practice 
abstraction, and independent abstraction with reconciliation by a trainer.  
 
9) Protocol modifications 
 
1. Changes to inclusion/exclusion criteria for Trials 1 and 2 (except where noted) 
a. Lowering the inclusion criteria age from ≥65 to ≥55 years 
b. Removing “English speaking” as an inclusion criterion (Trial 1 only) 
c. Adding hospital admission for a minimum of 12 hours as an inclusion criterion 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
12 
d. Removing markers of frailty as an eligibility criterion 
e. Adding pregnancy, suicide attempt and same-day discharge as exclusion criteria 
f. Adding COVID-19 as a specific inclusion diagnosis (Trial 1 only) 
2. Study design 
a. For Trial 2, in addition to comparing the bi-directional (survey-based) Jumpstart Guide to the 
clinician-facing (EHR-based) Jumpstart guide, we will also include a third arm in which patients 
receive usual care.  
b. We have increased the target sample size from n=400 to n=600 to accommodate the addition 
of a third “usual care” arm to the study. 
c. We have modified the questionnaires that patients receive at baseline such that only patients 
in the bi-directional Jumpstart arm are presented with items that are used to create the bi-
directional (survey-based) Jumpstart Guide. These items are an integral part of the bi-
directional intervention and therefore not appropriate for the other arms to complete.  
3. Procedures 
a. Removing data collection for acute severity of illness (e.g., SOFA score) for Trial 2 and replacing 
with a NEWS score. 
b. Revisions to instruments included in surveys for Trial 2 
c. Increasing subject numbers for clinician interviews from 20 to 50 across both trials 
d. Addition of demographic items (required by study sponsor, NIH) to the clinician interview 
4. Materials 
a. Updated instructional video for clinicians in Trial 1 
b. Revision of the Jumpstart Guide into an HTML format as a delivery option. This version 
includes optional feedback buttons at the bottom (options: will definitely use; will use if time 
allows; maybe, will consider; not appropriate; will not see this patient; already done; opt out; 
other/free text) 
c. Updated language and formatting for all versions of the Jumpstart Guide (patient, clinician 
EHR-based, and clinician survey-based) using a human-centered design approach.10  
 
10) Potential limitations and alternative approaches 
 
a) Including ADRD and other diseases in same study 
 
Patients with ADRD receive different intensity of care at the end of life compared to other chronic 
diseases,73,74 and there may be differences in the effectiveness of the interventions. Our hypothesis is that 
these interventions will work for all diseases, but because of the unique issues of increasing ICU use among 
those with ADRD,73,74 we will target adequate sample size for patients with ADRD to be able to test this 
hypothesis for the primary outcome of each trial. We could have proposed separate trials for ADRD and other 
illnesses, but this would decrease the generalizability of the interventions. 
 
b) Generalizability 
 
This study occurs in a single healthcare system but includes three diverse hospitals thus enhancing 
generalizability. 
 
c) Misclassification of goals-of-care discussions 
 
Goals-of-care discussions may be misclassified for two reasons: 1) the sensitivity and specificity of the NLP/ML 
algorithm is not perfect; and 2) documentation of goals-of-care discussions in the EHR will never perfectly 
reflect actual discussions. This misclassification could affect outcome assessment and patient identification. 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
13 
For outcome assessment, we will assess the accuracy of the NLP/ML algorithm against manual EHR review in a 
randomly selected sample of patients to evaluate the extent of misclassification.  For our final algorithm, we 
will use human abstractors to verify the first documentation of a goals-of-care discussion for each patient to 
maximize positive predictive value and specificity, as has been done by others.75,76  However, since our goal is 
to prompt and guide more discussions than would happen without the interventions, the limitations of the 
NLP/ML algorithm for patient identification do not invalidate the randomized trials. We could have included all 
hospitalized patients regardless of prior documentation, but we believe that untargeted prompts might limit 
the impact of the intervention. In addition, we will use the interviews in Aim 3 to understand clinicians’ 
perspectives on the effect of misclassification. 
 
d) Contamination 
 
It is possible that this intervention might change behavior for clinicians caring for patients randomized to the 
comparator arms. Our prior studies suggest that most clinicians require a patient-specific prompt to have 
timely goals-of-care discussions, which may mitigate this concern.9,25 However, we will assess for an increase in 
goals-of-care discussions in the comparator groups over time, which might signify contamination or temporal 
trends, but could be used to assess the potential degree of contamination if present. This issue would bias the 
results toward the null hypothesis and only be a major issue for a negative study. 
 
e) Scalability of surveys in Trial 2 
 
Study staff will distribute surveys, which is challenging for implementation in clinical practice. Aim 3 will 
provide insights into how best to address scalability for implementation.  
 
f) Objective assessments of the quality of goals of care discussions 
 
Our NLP/ML approach identifies goals-of-care discussions without assessing their quality. Since our prior trials 
demonstrated increased patient assessed quality with the Jumpstart intervention, this is less of a concern.9,25 
Future NLP/ML advances may permit quality assessments. Trial 2 assesses quality of communication from 
patient and family perspectives. 
 
g) Costs assessments focus on UW Medicine 
 
Cost assessments for Aim 1 are limited to costs available in the UW Medicine EHR, and we will not be able to 
assess costs from other healthcare systems after hospital discharge. Most of the benefits we anticipate for this 
intervention will occur during the hospitalization, although there may be ongoing reductions in costs after 
hospitalization related to changes in the goals of care as a result of the intervention. We will evaluate for such 
effects in Trial 2, and the limitation of not having access to these costs from outside UW Medicine in Trial 1 is 
diminished somewhat because this is a randomized trial. 
 
11) Anticipated findings 
 
These interventions use the EHR to identify patients who should have documentation of a goals-of-care 
discussion but do not, and then prompt and guide such discussions with either: a) a clinician-facing prompt and 
guide for clinicians only, along with information about prior advance care planning completed prior to the 
hospitalization; or b) a bi-directional, patient-specific intervention that provides patient-specific support to 
clinicians, patients and family members. We anticipate that both interventions will be effective compared to 
the usual care arm, and that this study will provide important options for healthcare systems. Economic 
analyses will allow us to evaluate the effect on costs of care, after factoring in the costs of implementing these 
interventions, to enhance dissemination. If either or both of these interventions are not effective, the results 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
14 
and the interviews in Aim 3 will provide important information to shape, direct and deliver future 
interventions. 
 
References 
 
1. 
Sudore RL, Lum HD, You JJ, et al. Defining Advance Care Planning for Adults: A Consensus Definition From 
a Multidisciplinary Delphi Panel. J Pain Symptom Manage 2017;53:821-32 e1. 
2. 
Childers JW, Back AL, Tulsky JA, Arnold RM. REMAP: A Framework for Goals of Care Conversations. J 
Oncol Pract 2017;13:e844-e50. 
3. 
Morrison RS, Meier DE, Arnold RM. What's Wrong With Advance Care Planning? JAMA 2021;326:1575-6. 
4. 
Halpern SD. Toward Evidence-Based End-of-Life Care. N Engl J Med 2015;373:2001-3. 
5. 
Halpern SD. Goal-Concordant Care - Searching for the Holy Grail. N Engl J Med 2019;381:1603-6. 
6. 
Halpern SD, Loewenstein G, Volpp KG, et al. Default options in advance directives influence how patients 
set goals for end-of-life care. Health Aff (Millwood) 2013;32:408-17. 
7. 
Mohan D, Sacks OA, O'Malley J, et al. A New Standard for Advance Care Planning (ACP) Conversations in 
the Hospital: Results from a Delphi Panel. J Gen Intern Med 2021;36:69-76. 
8. 
Sudore RL, Heyland DK, Lum HD, et al. Outcomes That Define Successful Advance Care Planning: A Delphi 
Panel Consensus. J Pain Symptom Manage 2018;55:245-55 e8. 
9. 
Curtis JR, Downey L, Back AL, et al. A patient and clinician communication-priming intervention increases 
patient-reported goals-of-care discussions between patients with serious illness and clinicians: a 
randomized trial. JAMA Intern Med 2018;178:930-40. 
10. Abedini NC, Merel SE, Hicks KG, et al. Applying Human-Centered Design to Refinement of the Jumpstart 
Guide, a Clinician- and Patient-Facing Goals-of-Care Discussion Priming Tool. J Pain Symptom Manage 
2021;62:1283-8. 
11. National Quality Forum, Measure 1626: Patients Admitted to ICU who Have Care Preferences 
Documented within 48 hours (RAND Corporation). 2017. (Accessed January 2, 2018, at 
https://www.qualityforum.org/News_And_Resources/Endorsement_Summaries/Endorsement_Summari
es.aspx.) 
12. Back AL, Arnold RM, Baile WF, et al. Efficacy of communication skills training for giving bad news and 
discussing transitions to palliative care. Arch Intern Med 2007;167:453-60. 
13. Back AL, Arnold RM, Tulsky JA, Baile WF, Fryer-Edwards KA. Teaching communication skills to medical 
oncology fellows. J Clin Oncol 2003;21:2433-6. 
14. Curtis JR, Sathitratanacheewin S, Starks H, et al. Using Electronic Health Records for Quality 
Measurement and Accountability in Care of the Seriously Ill: Opportunities and Challenges. J Palliat Med 
2018;21:S52-S60. 
15. Sudore RL, Schillinger D, Knight SJ, Fried TR. Uncertainty about advance care planning treatment 
preferences among diverse older adults. J Health Commun 2010;15 Suppl 2:159-71. 
16. Curtis JR, Wenrich MD, Carline JD, Shannon SE, Ambrozy DM, Ramsey PG. Understanding physicians' skills 
at providing end-of-life care: Perspectives of patients, families, and health care workers. J Gen Intern 
Med 2001;16:41-9. 
17. Steinhauser KE, Christakis NA, Clipp EC, McNeilly M, McIntyre LM, Tulsky JA. Factors considered 
important at the end of life by patients, family, physicians, and other care providers. JAMA 
2000;284:2476-82. 
18. Lee RY, Brumback LC, Lober WB, et al. Identifying Goals of Care Conversations in the Electronic Health 
Record Using Natural Language Processing and Machine Learning. J Pain Symptom Manage 2021;61:136-
42 e2. 
19. Curtis JR, Nielsen EL, Treece PD, et al. Effect of a quality-improvement intervention on end-of-life care in 
the intensive care unit: a randomized trial. Am J Respir Crit Care Med 2011;183:348-55. 
20. Curtis JR, Treece PD, Nielsen EL, et al. Integrating palliative and critical care: evaluation of a quality-
improvement intervention. Am J Respir Crit Care Med 2008;178:269-75. 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
15 
21. Curtis JR, Treece PD, Nielsen EL, et al. Randomized Trial of Communication Facilitators to Reduce Family 
Distress and Intensity of End-of-Life Care. Am J Respir Crit Care Med 2016;193:154-62. 
22. Khandelwal N, Brumback LC, Halpern SD, Coe NB, Brumback B, Curtis JR. Evaluating the Economic Impact 
of Palliative and End-of-Life Care Interventions on Intensive Care Unit Utilization and Costs from the 
Hospital and Healthcare System Perspective. J Palliat Med 2017;20:1314-20. 
23. Auriemma CL, Taylor SP, Harhay MO, Courtright KR, Halpern SD. Hospital-Free Days: A Pragmatic and 
Patient-centered Outcome for Trials among Critically and Seriously Ill Patients. Am J Respir Crit Care Med 
2021;204:902-9. 
24. Engelberg R, Downey L, Curtis JR. Psychometric characteristics of a quality of communication 
questionnaire assessing communication about end-of-life care. J Palliat Med 2006;9:1086-98. 
25. Au DH, Udris EM, Engelberg RA, et al. A randomized trial to improve communication about end-of-life 
care among patients with COPD. Chest 2012;141:726-35. 
26. Curtis JR, Engelberg RA, Nielsen EL, Au DH, Patrick DL. Patient-physician communication about end-of-life 
care for patients with severe COPD. Eur Respir J 2004;24:200-5. 
27. Curtis JR, Patrick DL, Caldwell E, Greenlee H, Collier AC. The quality of patient-clinician communication 
about end-of-life care: A study of patients with AIDS and their primary care clinicians. AIDS 1999;13:1123-
31. 
28. Janssen DJ, Curtis JR, Au DH, et al. Patient-clinician communication about end-of-life care for Dutch and 
US patients with COPD. Eur Respir J 2011;38:268-76. 
29. Hanson LC, Zimmerman S, Song MK, et al. Effect of the Goals of Care Intervention for Advanced 
Dementia: A Randomized Clinical Trial. JAMA Intern Med 2017;177:24-31. 
30. White DB, Angus DC, Shields AM, et al. A Randomized Trial of a Family-Support Intervention in Intensive 
Care Units. N Engl J Med 2018;378:2365-75. 
31. Teno JM, Fisher ES, Hamel MB, Coppola K, Dawson NV. Medical care inconsistent with patients' 
treatment goals: association with 1-year Medicare resource use and survival. J Am Geriatr Soc 
2002;50:496-500. 
32. Coast J, Huynh E, Kinghorn P, Flynn T. Complex Valuation: Applying Ideas from the Complex Intervention 
Framework to Valuation of a New Measure for End-of-Life Care. Pharmacoeconomics 2016;34:499-508. 
33. Finkelstein EA, Bilger M, Flynn TN, Malhotra C. Preferences for end-of-life care among community-
dwelling older adults and patients with advanced cancer: A discrete choice experiment. Health Policy 
2015;119:1482-9. 
34. Flynn TN, Bilger M, Malhotra C, Finkelstein EA. Are Efficient Designs Used in Discrete Choice Experiments 
Too Difficult for Some Respondents? A Case Study Eliciting Preferences for End-of-Life Care. 
Pharmacoeconomics 2016;34:273-84. 
35. Appelbaum PS. Clinical practice. Assessment of patients' competence to consent to treatment. N Engl J 
Med 2007;357:1834-40. 
36. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand 1983;67:361-70. 
37. Bjelland I, Dahl AA, Haug TT, Neckelmann D. The validity of the Hospital Anxiety and Depression Scale. An 
updated literature review. J Psychosom Res 2002;52:69-77. 
38. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J 
Public Health 2013;103:e38-46. 
39. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: 
the RE-AIM framework. Am J Public Health 1999;89:1322-7. 
40. Harden SM, Gaglio B, Shoup JA, et al. Fidelity to and comparative results across behavioral interventions 
evaluated through the RE-AIM framework: a systematic review. Syst Rev 2015;4:155. 
41. Shoup JA, Gaglio B, Varda D, Glasgow RE. Network analysis of RE-AIM framework: chronology of the field 
and the connectivity of its contributors. Transl Behav Med 2015;5:216-32. 
42. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of 
health services research findings into practice: a consolidated framework for advancing implementation 
science. Implement Sci 2009;4:50. 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
16 
43. Engelberg RA, Wenrich MD, Curtis JR. Responding to families' questions about the meaning of physical 
movements in critically ill patients. J Crit Care 2008;23:565-71. 
44. Engelberg RA, Downey L, Wenrich MD, et al. Measuring the quality of end-of-life care. J Pain Symptom 
Manage 2010;39:951-71. 
45. Howell AA, Nielsen EL, Turner AM, Curtis JR, Engelberg RA. Clinician perceptions of the role of an 
interprofessional communication facilitator to improve family outcomes in the ICU Am J Crit Care 
2014;23:380-6. 
46. Curtis JR, Engelberg R, Young JP, et al. An approach to understanding the interaction of hope and desire 
for explicit prognostic information among individuals with severe chronic obstructive pulmonary disease 
or advanced cancer. J Palliat Med 2008;11:610-20. 
47. Curtis JR, Engelberg RA, Wenrich MD, Au DH. Communication about palliative care for patients with 
chronic obstructive pulmonary disease. J Palliat Care 2005;21:157-64. 
48. Curtis JR, Engelberg RA, Wenrich MD, et al. Studying communication about end-of-life care during the 
ICU family conference: Development of a framework. J Crit Care 2002;17:147-60. 
49. Curtis JR, Engelberg RA, Wenrich MD, Shannon SE, Treece PD, Rubenfeld GD. Missed opportunities during 
family conferences about end-of-life care in the intensive care unit. Am J Respir Crit Care Med 
2005;171:844-9. 
50. Reinke LF, Engelberg RA, Shannon SE, et al. Transitions regarding palliative and end-of-life care in severe 
chronic obstructive pulmonary disease or advanced cancer: themes identified by patients, families, and 
clinicians. J Palliat Med 2008;11:601-9. 
51. Reinke LF, Shannon SE, Engelberg RA, Young JP, Curtis JR. Supporting hope and prognostic information: 
nurses' perspectives on their role when patients have life-limiting prognoses. J Pain Symptom Manage 
2010;39:982-92. 
52. West HF, Engelberg RA, Wenrich MD, Curtis JR. Expressions of nonabandonment during the intensive 
care unit family conference. J Palliat Med 2005;8:797-807. 
53. White DB, Engelberg RA, Wenrich MD, Lo B, Curtis JR. Prognostication during physician-family discussions 
about limiting life support in intensive care units. Crit Care Med 2007;35:442-8. 
54. Back AL, Young JP, McCown E, et al. Abandonment at the end of life from patient, caregiver, nurse, and 
physician perspectives: loss of continuity and lack of closure. Arch Intern Med 2009;169:474-9. 
55. Carline JD, Curtis JR, Wenrich MD, Shannon SE, Ambrozy DM, Ramsey PG. Physicians' interactions with 
health care teams and systems in the care of dying patients: Perspectives of dying patients, family 
members, and health care providers. J Pain Symptom Manage 2003;25:19-28. 
56. Curtis JR, Patrick DL, Caldwell E, Collier AC. Why don't patients with AIDS and their clinicians talk about 
end-of-life care? Barriers to communication for patients with AIDS and their primary care clinicians. Arch 
Intern Med 2000;160:1690-6. 
57. Curtis JR, Wenrich MD, Carline JD, Shannon SE, Ambrozy DM, Ramsey PG. Patients' perspectives on 
physicians' skills at end-of-life care: Differences between patients with COPD, cancer, and AIDS. Chest 
2002;122:356-62. 
58. Wenrich MD, Curtis JR, Ambrozy DM, et al. Provision of emotional support and personalized care by 
physicians to patients nearing the end of life. J Pain Symptom Manage 2003;25:236-46. 
59. Wenrich MD, Curtis JR, Shannon SE, Carline JD, Ambrozy DM, Ramsey PG. Communicating with dying 
patients within the spectrum of medical care from terminal diagnosis to death. Arch Intern Med 
2001;161:868-74. 
60. Elwyn G, Barr PJ, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and 
frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns 
2013;93:102-7. 
61. Little R, Rubin D. Statistical analysis with missing data. Third ed. New York: Wiley; 2019. 
62. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277-
88. 
63. Corbin J, Strauss A. Methods of qualitative research. Thousand Oaks, CA: Sage Publications; 1993. 

Study Protocol: PICSI-H  
PI: J. Randall Curtis 
Page 
17 
64. Strauss AL, Corbin J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded 
Theory. Thousand Oaks: Sage Publications; 1998. 
65. Attride-Stirling J. Thematic networks: An analytic tool for qualitative research. . Qualitative Research 
2001;1:385-405. 
66. Giacomini M, Cook DJ, for the Evidence-Based Medicine Working Group. Qualitative research in health 
care: What are the results and how do they help me care for my patients? JAMA 2000;284:478-82. 
67. Kross EK, Nielsen EL, Curtis JR, Engelberg RA. Survey burden for family members surveyed about end-of-
life care in the intensive care unit. J Pain Symptom Manage 2012;44:671-80. 
68. Glavan BJ, Engelberg RA, Downey L, Curtis JR. Using the medical record to evaluate the quality of end-of-
life care in the intensive care unit. Crit Care Med 2008;36:1138-46. 
69. Gries CJ, Engelberg RA, Kross EK, et al. Predictors of symptoms of posttraumatic stress and depression in 
family members after patient death in the ICU. Chest 2010;137:280-7. 
70. Gries CJ, Curtis JR, Wall RJ, Engelberg RA. Family member satisfaction with end-of-life decision making in 
the ICU. Chest 2008;133:704-12. 
71. Gerstel E, Engelberg RA, Koepsell T, Curtis JR. Duration of withdrawal of life support in the intensive care 
unit and association with family satisfaction. Am J Respir Crit Care Med 2008;178:798-804. 
72. Kross EK, Engelberg RA, Gries CJ, Nielsen EL, Zatzick D, Curtis JR. ICU care associated with symptoms of 
depression and posttraumatic stress disorder among family members of patients who die in the ICU. 
Chest 2011;139:795-801. 
73. Curtis JR, Engelberg RA, Teno JM. Understanding variability in end-of-life care in the ICU for the elderly. 
Intensive Care Medicine 2017;43:94-6. 
74. Teno JM, Gozalo P, Khandelwal N, et al. Association of Increasing Use of Mechanical Ventilation Among 
Nursing Home Residents With Advanced Dementia and Intensive Care Unit Beds. JAMA Intern Med 
2016;176:1809-16. 
75. Udelsman B, Chien I, Ouchi K, Brizzi K, Tulsky JA, Lindvall C. Needle in a Haystack: Natural Language 
Processing to Identify Serious Illness. J Palliat Med 2019;22:179-82. 
76. Lindvall C, Deng CY, Moseley E, et al. Natural Language Processing to Identify Advance Care Planning 
Documentation in a Multisite Pragmatic Clinical Trial. J Pain Symptom Manage 2022;63:e29-e36.