Evaluation of Methods for Implementation of a Comfort Care Order Set
Phase: NA
Conditions: Terminal Conditions, End of Life
Sponsor: VA Office of Research and Development

Version 6/20/17 
  
 
Page 1 
 
 
Protocol 
Evaluation of Methods for Implementation of a Comfort Care Order Set – Beacon II 
Principal Investigator: Kathryn L. Burgio, PhD 
6/20/2017 
Background 
It is well documented that patients at life’s end have unrelieved physical suffering with significant emotional, 
spiritual, and social distress. Too many patients die an undignified death with many uncontrolled symptoms. 
Emotional, social and spiritual needs of the patient and the family often are not addressed adequately. Studies 
report most patients have up to 10 different symptoms during the last week of life, many severe and 
unrelieved. Consistently, family accounts of the dying process and bereavement period point to unrelieved 
symptoms and exacerbating ineffective disease-modifying treatments as sources of distress.  
Hospice is a holistic approach to end-of-life care focusing on management of pain and non-pain symptoms, 
while addressing other forms of distress such as psycho-social stressors and existential angst. At its best, 
hospice care supports growth and healing, achieving a sense of completion that includes bereavement and 
grief support both before and after death. In this way hospice programs may help individuals have a “Good 
Death.”  One of the particular skills that the hospice programs have developed is excellence in care in the last 
days and hours of life. This is the specialty care of the actively dying. Patients referred to hospice often benefit 
from improved symptom management and practical assistance in the home, which may improve both quality 
and quantity of life. The hospice model is accepted as a model of excellence for care at life’s end.  
Despite the benefits of hospice care, research has shown most people do not receive such care when they die.  
For the foreseeable future, it appears that the majority of patients will die in a hospital or nursing home, where 
they will not receive care designed specifically to address suffering and support family members. Statistics 
from the National Hospice & Palliative Care Organization indicate that even among hospice patients, only 52% 
die at home, 22% die in a nursing facility and 10% in a hospital. Given these numbers, there is clearly a need 
to address the processes of care for the actively dying patient in the inpatient setting.   
Integrating hospice care into traditional medical practices can be challenging because the general approaches 
to care differ. In acute care settings, where the focus is on “cure,” attention to symptom management often is 
downplayed, because it is assumed that symptoms will abate if the disease is “managed” or “cured.” In this 
environment, transition away from disease-modifying treatments to symptom control can seem counter-
intuitive. Successful symptom management techniques and supportive care systems that are not technically 
sophisticated are not always available for inpatients. The subspecialty palliative care workforce is not sufficient 
to reach all patients dying in hospital settings, and acute care practitioners can fail to recognize shifts in patient 
illness trajectory indicating the need for palliative care consultation. The overarching goal of this program of 
research is to implement the best practices of traditionally home-based hospice care in the inpatient setting. 
This intervention could have significant impact in transferring the best practices of hospice care into 
institutional settings where most end-of-life care actually takes place.  
In the past, other acute care palliative care interventions have not achieved their goals. The SUPPORT study 
sought to improve care for seriously-ill hospitalized patients using specially trained nurses to increase doctor-
patient communication regarding care preferences. This carefully designed study of 9,105 adults in five 
teaching hospitals failed to improve care at end of life. Investigators in another study used a multi-modality 
intervention to improve quality of end-of-life care in 12 intensive care units. Despite a robust intervention that 
included clinician champions, clinician education, academic detailing, quality-of-care feedback and system 
support, they were unable to identify improvements in quality-of-care outcomes based on family satisfaction. 
Early Work and Development of the Comfort Care Education Intervention 
Dr. Amos Bailey has many years of experience in the field of palliative care, most of which has focused on 
inpatient settings. In 1998, he founded the Balm of Gilead, a palliative care unit within an acute care public 
hospital. The goal was to replicate the best practices of home hospice care in the acute care hospital. In 2001, 
he established the Birmingham VAMC Safe Harbor Palliative Care Program, a comprehensive initiative that 

Version 6/20/17 
  
 
Page 2 
 
includes active case finding, inpatient and outpatient consultation, staff education, and physician/nurse case 
management of inpatients and home hospice patients referred to community programs. Recent feedback from 
the PROMISE family surveys indicate the BVAMC is above national norms on several scores, including overall 
ratings of “excellent.”  
It was in these settings that Dr. Bailey developed the Comfort Care Order Set (CCOS) based on home hospice 
best practices for care in the last days or hours of life and modified for use in the acute inpatient care setting. 
Subsequently, it has been refined, condensed to a pocket card, and translated into a CPRS software patch. As 
a set of modifiable standing orders, it functions as a decision support tool. With funding from The Project on 
Death in America (PDIA) Faculty Scholars Program (2002-2003), Dr Bailey developed and evaluated a 
Palliative Medicine Training Program. He also developed and published an instructional manual, The Palliative 
Response,  a comprehensive guide to end-of-life care. Based on adult learning theory, the manual includes 44 
individual 5-minute education sessions. The book has been used to train internal medicine residents and guide 
daily didactic teaching sessions for palliative care, oncology, and geriatric fellows. This text will be used with 
other educational materials to train providers in the proposed project.  
Pilot Study: Intervention to Improve End-of-Life Care in the Birmingham VA Medical Center 
To formally assess the impact of the Safe Harbor Palliative Care Program within the Birmingham VAMC, a pilot 
study was conducted using educational materials and pocket cards to assist physicians in the identification of 
dying patients, and the Comfort Care Order Set (CCOS) to implement a plan of care. Data on 13 symptoms 
and processes of care were collected for the 6 months prior to initiation of the Program in 2001 and compared 
to data from the first 6 months of 2003. Between the two years, providers increased documentation for 10 of 
the13 end-of-life symptoms (p < .05) and increased care plans for 9 of 13 symptoms (p < .05). Number of 
documented end-of-life symptoms increased from 1.7±2.1 to 4.4±2.7 (p < .001) and documented care plans 
increased from 0.4±0.9 to 2.7±2.3 (p < .001). Opioid medication orders increased dramatically from 56% to 
84% (p < .001) and DNR orders increased from 62% to 89% (p < .001). There was a significant decline in 
deaths in the ICU from 47.4% to 25% (p = .002). Hospice care offerings increased significantly from 21% to 
35% (p < .03); In addition, overall costs of caring for patients at end of life decreased. In secondary analyses, 
there was a significant increase in orders for morphine as a preferred opioid from 47.4% to 81.7% (p < .001) 
and administration of an opioid in the last 72 hours of life increased from 13.9% to 71.3%( p<.0001. 
Based on the pilot data from the Birmingham VAMC, the team concluded that the Comfort Care Education 
Intervention was a promising, feasible approach to inpatient end-of-life care. With the teaching methods and 
materials developed, including the published training manual, six laminated pocket cards, PowerPoint slide 
presentations, and the CCOS (built and integrated into CPRS), the team sought Merit Review funding to test 
the effectiveness of the education intervention for improving the quality of end-of-life care in other VAMCs. 
Multi-site Implementation Trial: "Intervention to Improve Care at Life's End in VA Medical Centers" (BEACON) 
(funded by VA HSR&D; PI: Burgio; Co-PI: Bailey) (in press, Journal of General Internal Medicine)  
“Best Practices for End-of-life Care and Comfort Care Order Sets for Our Nation's Veterans” (BEACON) is a recently 
completed real-world implementation trial designed to evaluate the Comfort Care Education Intervention in six 
VAMCs in the Southeastern US. The Comfort Care Education Intervention consisted of two weeks of in-person staff 
training plus supporting written materials (pocket cards and book), creation of a CCOS in each site’s CPRS system, 
and follow-up consultation. Staff training targeted hundreds of VAMC staff members in teaching basic palliative care, 
identification of the dying veteran, and application of the CCOS modified as needed for individual patients. Several 
processes of care were identified as quality endpoints for end-of-life care (last 7 days) and abstracted from the 
electronic medical records of veterans who died before or after intervention in the participating VAMCs. 
Primary endpoints were proportion of patients with an order for opioid pain medication at time of death, do-not-
resuscitate order, location of death, nasogastric tube, intravenous line infusing, and physical restraints. Due to 
the possibility that secular trends could bias or confound intervention estimates, records were abstracted 
across the entire 6-year study period for all hospitals (n=6,066). Generalized estimating equations were 
conducted adjusting for longitudinal trends over the 6 years of the study.  
Each of the intervention components was delivered to all sites. Sign-in sheets documented that training was 
attended by at least 1,621 staff, including 131 physicians, residents, and medical students, 66 physician 

Version 6/20/17 
  
 
Page 3 
 
assistants and nurse practitioners, 943 nurses, 135 nursing assistants and students, 53 social workers, 24 
chaplains, 100 pharmacy staff, 5 mental health staff, 24 respiratory therapists, 15 dietary staff, 54 other allied 
health professionals, 44 administrative staff, and 27 other staff. There were no significant differences in patient 
characteristics between the pre-and post-intervention groups. Changes were in the expected direction for all 17 
of the process of care endpoints. Longitudinal (secular) trends were significant only for the proportions of 
patients dying with a do-not-resuscitate (DNR) order, family present, or an order for death rattle medication. 
For all other variables, no longitudinal trends achieved statistical significance. After adjusting for these 
longitudinal trends, the intervention effects remained significant for the proportion of patients dying with an 
active opioid order, with an antipsychotic order, and with an order for death rattle medication (Figure 1). The 
longitudinal analysis also indicated significant intervention effects for presence of advance directives, 
nasogastric tube, benzodiazepine order, and sublingual administration. 
Conclusion: This broadly targeted 
intervention to change practice 
patterns for end-of-life care led to 
modest but statistically significant 
changes in several processes of care. 
These changes become highly relevant 
when considering the potential impact on 
the 21,500 veterans who die each year in 
VA facilities. 
A secondary analysis of baseline data 
(2005 only, n=1,068) described pain 
management practices for imminently 
dying patients and found that 64.2% had 
an active order for an opioid at the time of 
death.  Of these, 69.8% of patients had 
received the medication at some time 
within the last 7 days of life, 61.2% within 
the last 48 hours, and 47.0% within the 
last 24 hours. These findings indicate a 
need for improving availability of opioids for end-of-life care in the inpatient setting, especially within 
the last 24 hours.   
Formative Evaluation of the BEACON Intervention  
After completing the BEACON trial, we conducted a formative evaluation (supplementary study funded by 
HSR&D) to understand the processes by which best practices were adopted and to inform the design of the 
proposed Enhanced Intervention Approach. We completed semi-structured telephone interviews with key 
informants from each site in the trial (n=14) and queried them about the roles played by key personnel, as well 
as perceived barriers, facilitators, needs, and suggestions for future implementation. Qualitative analysis was 
led by the team medical sociologist (Dr. Williams).  
In characterizing the optimal instructional modality, informants’ accounts of the favorable aspects of the 
BEACON training focused on the flexible presentation strategies, standardized instructional tools, and 
interactive teaching methods. Informants detailed how small group instruction, in particular, facilitated active 
learning, encouraging participants to engage with the instructional process, to discern what palliative care is all 
about, and to explore the “whys” and “hows” for doing palliative care. Teaching in small groups facilitated 
interactive learning, allowing participants to connect to the learning process by relating their personal stories of 
illness, death, and dying, and stimulating others to share their stories as well. Small interactive settings and 
efficient use of time also heightened interest in the training sessions. In typifying the characteristics of the ideal 
trainer, informants expressed the importance of the presenter’s particular experience in the field, his ability to 
relate to and communicate with trainees, and his content area expertise.  
Figure 1: Odds ratios and 95% confidence intervals for process of 
care endpoints adjusting for six-year longitudinal trends, N=6,066 

Version 6/20/17 
  
 
Page 4 
 
Reflecting on challenges and barriers, respondents mentioned the difficulties involved in making staff available 
for training (including the burden of competing training and other duties), getting buy-in from individuals, 
administration, and physician leadership, staff shortages, and staff turn-over. A major theme across sites was 
how interest in the intervention faded in the post-intervention period, after the team left the facility, and the 
corresponding need for refresher training and regular formal and informal training opportunities. As 
suggestions, they specifically mentioned using a train-the-trainer model and "getting key physicians talking 
about Palliative Care," so their residents would learn about it as part of their regular rotations. In one 
respondent's words:  
"Well I think you really need to work closely with somebody within the facility who has the credibility of the 
staff, who has experience with quality end-of-life care who can do the training within that facility. I mean, it’s 
always nice to have that external expert come in, but once they’re gone, the immediacy or the importance of 
it tends to fade. Whereas if you’ve got somebody there who can continue to keep it moving forward, that’s 
going to be really important. ...I think that people who are dealing with the clinical issues are the ones who 
need to be doing the training and the constant cheerleading oversight monitoring of the program." 
These findings support the design of the proposed Enhanced Implementation Approach, particularly the face-
to-face, interactive learning model and the enduring role of the Champion embedded within each facility.  
Dissemination Activities 
In the final stage of the BEACON project, all materials were refined and a Comfort Care Education Intervention 
Packet was created for training providers and a “train the trainer” model for disseminating best practices. The Packet, 
including all of the written materials, presentations, policy and procedures, and family brochure is posted on the 
VA Sharepoint and Dr. Bailey has used it to train Palliative Care Consult Teams in VISN 11. In addition, the CCOS 
for CPRS has been disseminated to over 15 other VAMCs with encouragement from the Comprehensive End-
of-life Care Initiative (CELC) and support from the CELC VA Implementation Center.  
Rationale for the Proposed Implementation Study 
With the completion of the BEACON study, we conclude that it is possible to change provider behavior across a 
number of endpoints that reasonably can be expected to improve end-of-life care for veterans. However, 
investigators encountered challenges conducting this trial that revealed barriers to the adoption of best practices, 
including the difficulty of doing intensive training at another site, as well as some factors that appear to promote 
uptake, such as positioning on-site clinical champions to lead the efforts to change practice on an ongoing basis. 
In the BEACON trial, Dr. Bailey, the physician leader and developer of the intervention, personally travelled to each 
site and spent two weeks providing intensive staff training. While this approach allowed us to demonstrate the 
effectiveness of the Comfort Care Education Intervention, it was difficult to accomplish and cannot be replicated on a 
broad scale by Dr. Bailey himself. This poses significant challenges to the feasibility of national implementation.  
Further, while the changes were statistically significant across several important variables, they were modest in 
magnitude, particularly in comparison to changes observed during the pilot study at the BVAMC. This suggests a 
potential implementation effect and underscores the need to reassess and refine the intervention delivery strategy.  
Although the Education Intervention Packet is fully developed and available to all VA providers, the best strategy for 
delivering the education to providers and facilitating adoption of best practices (CCOS) is not yet known.  
A goal of the BEACON study was to train as many staff as possible that might have contact with actively dying 
patients. However, in achieving this goal, it is possible that the depth and breadth of knowledge necessary to achieve 
long-term culture and practice change, as well as sustainable patient outcomes was not reached. Because hundreds 
of staff were exposed to some training, no one individual or group of individuals received intensive training, which 
may have diffused the efficacy of the intervention. Further, although intensive in nature, the intervention occurred 
over a short period of time. This did not provide the continuity over time needed to accomplish large changes in 
practice and to address staff turn-over. As such, findings from the BEACON study suggest the need for a train-the-
champion model as an alternate approach to ensuring that each VA facility has an in-house subject matter expert 
capable of training others and providing continuity and longer-term availability. A train-the-champion model, 
through which training could be ongoing at each site, would allow for a more sustained delivery of training and 

Version 6/20/17 
  
 
Page 5 
 
the potential for regular refreshers and updates. We propose to test this train-the-champion model in the next 
phase of this research program (BEACON 2). 
As a result of the 2003 VA mandate, every VAMC now has a formal Palliative Care Consult Team (PCCT) to 
promote and oversee Palliative Care programs in local facilities. These PCCTs include a physician, nurse, social 
worker, mental health provider, and pastoral care provider. These teams have received core training with the 
EPEC for Veterans and ELNEC for Veterans, which allows these individuals to easily understand the concept 
and need for the comfort care clinical interventions and quickly adopt them in their care setting to improve their 
delivery of palliative care. Compared to what was previously available within VA, PCCT members represent a 
more highly trained group of providers who can serve as clinical champions  
We reason that PCCTs are now well established and that team members are trained at least in the basics of 
palliative care principles. We propose to capitalize on this existing infrastructure and utilize these leaders as change 
agents for delivering the Comfort Care Education Intervention using two implementation strategies. Training existing 
teams of palliative care providers creates the potential for in-house change agents to be present for a longer period 
of time. This potentially could increase the uptake of comfort care interventions as opposed to the presence of an 
expert for only two weeks. Activating and empowering a PCCT, rather than an individual champion, has the potential 
for greater impact. In their formal roles, members of the PCCTs already are leaders, recognized and charged with 
improving palliative care and presumably motivated to implement what they have learned and to pass this training 
onto others in an ongoing way. By training and supporting PCCTs to provide state-of-the-art, evidence-based care 
we propose to optimize adoption and utilization of best practices in a sustainable and reproducible manner.   
The purpose of the proposed project is to evaluate two methods of delivering this Comfort Care Education 
Intervention utilizing the established infrastructure of Palliative Care Consult Teams (PCCT): a Basic 
Implementation Approach using a teleconference to review educational materials and activate PCCTs to 
educate other providers, and an Enhanced Implementation Approach utilizing in-person, train-the-champion 
workshops to prepare PCCT members to be clinical champions and trainers at their home sites.  
With these findings, we will be able to determine the better method of delivery for the Comfort Care Education 
Intervention and modify the training program, the education materials, and any other aspect of the 
implementation strategy to ensure a successful future dissemination. 
Conceptual Framework  
The proposed study is guided by constructs from the PRECEDE (Predisposing, Reinforcing, and Enabling 
Constructs in Ecosystem Diagnosis and Evaluation) framework. This framework, which has its origins in health 
promotion planning and has been applied more recently in the field of implementation science, provides a 
comprehensive structure for diagnosing needs, as well as developing, and implementing, and evaluating 
interventions targeted to meet those needs. In the current context, the framework guides a multi-modal 
approach to changing practice patterns for end-of-life care including: 1) predisposing providers to make the 
desired changes in practice through intensive case-based and experiential training and by utilizing in-house 
“change agents”; 2) enabling providers to change by providing them with a computerized clinical decision 
support tool, as well as a packet of written educational materials; and 3) reinforcing the changes in practice 
through a regular audit and performance feedback mechanism. This framework also provides a structure for 
evaluating the process, impact, and outcome of the interventions.  
Significance 
The total number of veterans predicted to die next year, both inside and outside of the VA, is greater than 
600,000, comprising more than 25% of all individuals who die in the United States. Within VA facilities, nearly 
21,500 veterans die each year. The VA has long recognized that veterans at the end-of-life have unique needs for 
a healthcare system that can respond to the need for pain and symptom management, as well as psychosocial 
support. However many patients still have poorly controlled symptoms and multiple unmet needs.  
VA has embarked on a robust program to improve capacity for hospice and palliative care services in all VA facilities. 
This program includes training initiatives, and the creation of over 50 new inpatient hospice-palliative care units. 
Further, Palliative Care Consult Teams (PCCT) are mandated by VHA at all VA facilities. Dr. Amos Bailey was 
a member of the Hospice and Palliative Care Field Advisor Group for VA Central Office that drafted this 2003 

Version 6/20/17 
  
 
Page 6 
 
directive. In the last three years the Comprehensive End-of-Life Care (CELC) Initiative has supported the 
PCCTs and other staff in the VAMC by the development of "Education in Palliative and End-of-life Care" 
(EPEC) for Veterans, "End-of-Life Nursing Education Consortium" (ELNEC) for Veterans, and Hospice and 
Palliative Nurses Association (HPNA) nursing assistants training. The PCCTs and other staff have received 
mandatory training with these new curricula to improve proficiency, and they are primed to use the CCOS.  
The proposed project will build on this infrastructure by providing the PCCTs at participating facilities with 
training, education materials, tools, and technical support to help accomplish their mission. Further, it will 
provide valuable information on the effectiveness of two implementation strategies for delivering this Comfort 
Care Education Intervention, developed in the BEACON project, and teaching PCCT members to train other 
providers and serve as clinical champions. The long-term goal of this program of research is to improve the quality 
of end-of-life care for patients dying in VA Medical Centers. Findings will be used to refine the Comfort Care 
Education Intervention and implementation approach in preparation for nationwide dissemination of best 
practices for end-of-life care within the VA Healthcare System, and potentially within health care systems 
throughout the nation providing the bulk of health care services to veterans.  
Research Design and Methods 
Overview 
This study will be a group (cluster) randomized trial in which 50 PCCTs will be assigned to receive the Comfort 
Care Education Intervention via a Basic or Enhanced Implementation Approach. Data on processes of end-of-
life care, before and after implementation of the Education Intervention, will be derived from the records of 
veterans who have died in the 50 VAMC hospitals. Following intervention, semi-structured telephone 
interviews will be conducted with PCCT members to explore the implementation process at each site, including 
barriers, facilitators and suggestions for improving the implementation process. Questionnaires will be used to 
evaluate the perceptions of PCCT members and providers who receive training from the PCCT regarding 
training and its impact on their attitudes, skills, practice, and perceived efficacy to care for patients. 
Recruitment 
PCCT teams at 50 VA Medical Centers will be recruited to participate in the trial. To avoid ceiling effects, 
recruitment will be preferential, targeting teams at sites with the lower scores on the PROMISE family after-
death survey, which measures several parameters of family satisfaction with care received by the veteran. To 
avoid enrolling sites that might be especially resistant to change, teams at VAMCs with scores in the lowest 
10th percentile will not be recruited. To compensate for potential dropout, 50 sites will be recruited to ensure 
the participation of 48 VA Medical Centers. The study will be announced on the twice monthly calls held with 
the Palliative Care Coordinators for each VISN. Emails will also be used to announce the study and invite 
participation. (See letter of support from Scott Shreve, DO, National Director of Hospice & Palliative Care) 
Currently, the Comprehensive End-of-Life Care (CELC) palliative care programs have a high profile among 
VISN and VAMC leadership. All VISNs and VAMCs are receiving quarterly PROMISE reports, which document 
deficiencies in quality of care, and identifying areas that need improvement and is expected to increase 
facilities' motivation to participate in the study. Each PCCT can decide individually if they wish to participate 
in the study. If there is not a prescribing provider from the PCCT (enabling use of the CCOS) who wishes to 
participate, that site will not be included in the trial. 
Once the PCCT members are identified as participants, their personally identifiable, i.e., name, telephone, 
email, will be maintained on VA secure networks. If paper copies exist, they will be maintained within locked 
office of the GRECC. 
Power 
Power calculations were based on data on the primary endpoint from the BEACON study, change in proportion 
of patients with an active opioid order at time of death. Sample sizes of 2400 patients in the Basic 
Implementation Approach and 2400 patients in the Enhanced Implementation Approach, which will be obtained 
by randomizing 24 PCCTs from VA Medical Centers with 100 patients on average to each group, will achieve 
86% power to detect a difference between the group proportions of 0.05, where the proportion of patients post-
intervention with an active order for opioid at death in the Basic group is 0.65 and the proportion in the 

Version 6/20/17 
  
 
Page 7 
 
Enhanced group is 0.70. The significance level was 0.05 and an intraclass correlation coefficient (ICC) is 
assumed to be 0.005, similar to the ICC observed in the original BEACON Trial. Calculations were performed 
using PASS 11.0. 
Randomization 
To avoid baseline differences between groups of sites in quality and processes of end-of-life care, PCCT sites 
will be stratified on high vs. low baseline scores on the PROMISE Bereaved Family Survey. PROMISE scores 
should reflect the general quality of care provided. Within this stratum, teams will be randomized to the Basic 
or Enhanced Approach to implementing the Intervention.  
In most group-randomized clinical trials, all groups or clusters are recruited and randomized at baseline. 
However, we will not be able to recruit and randomize all PCCTs prior to initiation of intervention. For this 
reason, training of the PCCTs will be conducted in separate waves. When five PCCTs have been recruited and 
randomized to a group, training will be initiated with those five teams as a group. This will produce twelve 
waves of 6 teams (six waves for each implementation approach) to complete the study. While wave #1 is 
receiving intervention, 10 more teams will be recruited in preparation for wave #2. These waves of recruitment, 
randomization, and training will continue until 25 teams have been assigned to each group.  
The BEACON Comfort Care Education Intervention - a Multi-modal Approach 
The Education Intervention is targeted to providers. There has been considerable debate about the effects of 
continuing medical education (CME) on provider behavior in the practice setting. CME courses have been 
found to be effective for increasing provider knowledge; but knowledge has often not generalized into their 
practice patterns. Didactic presentations and distributing printed information only have little or no beneficial 
effect in changing physician practice. It has been suggested that the ineffectiveness of CME accounts for the 
discrepancy between evidence and practice or, at lease, contributes to this gap. Understanding what CME 
tools and techniques are most effective in disseminating and retaining medical knowledge is critical to 
improving CME and diminishing the gap between evidence and practice. 
Several health services research (HSR) areas focus on improving patient quality of care by changing provider 
behavior. For instance, academic detailing, audit and feedback, benchmarking, continuing medical education 
(CME), and computerized reminder systems are all methodologies used to change provider behavior. The 
literature is consistent in reporting that interactive techniques are the most effective at changing physician care 
and patient outcomes. Clinical practice guidelines and opinion leaders are less effective. However, there is 
mixed evidence that these interventions alone can change and sustain behavior. The literature is clear that 
there does not appear to be a “magic bullet” in changing provider behavior, but that multi-modal interventions 
may be more effective compared to a single intervention methodology. 
The proposed study will use a multi-modal approach to changing practice patterns for end-of-life care. This will 
be accomplished in part by offering an intervention that includes three core adult learning and development 
concepts: 1) training will be timely, relevant, insightful, practical and useful; 2) it will employ experiential 
learning and action learning strategies, methods and practices; and, 3) the central focus will be on participants’ 
interaction and collaboration. Specifically, using the PRECEDE phases of the conceptual framework, 1) 
providers will be predisposed to make the desired changes in practice through intensive case-based and 
experiential training and by utilizing in-house “change agents”; 2) providers will be enabled to change by 
providing them with a computerized clinical support tool, as well as a packet of written educational materials; 
and, 3) the changes in provider practice will be reinforced through structured consultation with trainers.  
The two groups will receive the same Pre-Implementation Preparation and the same Education Intervention 
Packet Materials. They will differ on the implementation approach used to deliver the education intervention. 
Table 1 
Components of Comfort Care Education Intervention/Implementation 
Teleconference 
Train-the-Champion 
A. Pre-Implementation Preparation 
A. Pre-Implementation Preparation 

Version 6/20/17 
  
 
Page 8 
 
Communicate with administrative leaders 
Communicate with administrative leaders 
Communicate with PCCT members 
Communicate with PCCT members 
Communicate with IRM representative 
Communicate with IRM representative 
Technical assistance to build Comfort 
Care Order Set in CPRS 
Technical assistance to build Comfort Care Order 
Set in CPRS 
B. Education Intervention Package Materials 
B. Education Intervention Package Materials 
Case Identification 
Case Identification 
Comfort Care Interventions 
Comfort Care Interventions 
Comfort Care Order Set  
Comfort Care Order Set  
Pocket cards 
Pocket cards 
Sample policies and procedures 
Sample policies and procedures 
PowerPoint slides for training other providers 
PowerPoint slides for training other providers 
Sign-in sheets to document training other 
providers 
Sign-in sheets for training other providers 
C. Implementation Strategy 
C. Implementation Strategy 
Activate Comfort Care Order Set  
Activate Comfort Care Order Set  
Teleconference 
Train-the-Champion Program 
Orientation and review of Education Packet 
Materials (80 minutes) 
In-person workshop at BVAMC (2 days) 
 
Interactive teaching on comfort care interventions 
 
Case-based coaching to build care plans using 
CCOS 
 
Hands-on experiences with using the CCOS 
 
Teaching to become trainers and change agents 
(with manual) 
 
Review of baseline facility-specific PROMISE data 
(feedback) 
Charge to train providers at home facility 
Charge to train providers at home facility 
Team available for consultation as needed by 
PCCT members  
Bi-weekly conference calls to provide consultation 
for 4 months 
A. Pre-Implementation Preparation 
Identify Lead Champion. (point of contact) It is critical that all sites have a lead champion, a physician or 
nurse practitioner who will lead the efforts to promote ongoing palliative care education and use of the CCOS. 
The quality of the lead champion may vary, but is expected to be fairly uniform in the proposed study due to 
the standard training they receive in the PCCT role.  
CCOS preparation: Each PCCT team will be provided with the assistance of an experienced Computer 
Applications Coordinator (CAC; see Letter of Support from Dr. Cleveland) and a clinical provider (Dr. Bailey, Dr. 
Perna, or Dr. Kvale) at the Birmingham VAMC, all of whom will work remotely with the local PCCT and CAC to 
construct a CCOS tailored and integrated into that VAMC's CPRS using existing and new orders. To assist the 
local team to build the CCOS, materials are provided in the Education Packet that include a description of the 
components and screen shots of the sample CCOS in the CPRS system. In addition, the BVAMC CAC has 
written an instructional manual to assist the CACs at other VAMCs to construct their order set.  
The CCOS will be tested by the investigators and refined to ensure that it functions as planned and has fidelity to the 
best practices. It can be tested remotely in Birmingham and errors or sections that do not work as planned can 
be repaired easily. This is an iterative process unique to each VA, because it is built using orders that already 
exist within the local system and the programming varies across VAMCs. Based on experience with other sites, 

Version 6/20/17 
  
 
Page 9 
 
this process can be completed in 30 hours, divided over an 8- week period. The CCOS will not be activated until the 
training for that site is launched. The BVAMC CAC has built a “Reminder Report” to identify patients for whom the 
CCOS was used. This programming will allow investigators to track use of the electronic CCOS by site.  
B. Education Intervention Package Materials  
Dr. Bailey has developed a Training Manual for PCCTs to use to deliver Comfort Care Interventions in the 
inpatient setting, specifically in their local VAMCs. The Palliative Response includes didactic information on 
how to identify and care for patients who are dying in the hospital setting, the CCOS, pocket cards, and other 
tools and training materials needed to train staff at their own hospital.  
Case Identification. PCCT members at the sites will receive a pocket card designed to aid them in the 
identification of patients who are near the end of life and for whom use of the comfort care order set is 
appropriate.  This tool has been adopted by VISN 7 as a screening tool for case finding. 
Comfort Care Clinical Interventions. PCCT members will receive materials describing a number of Comfort 
Care interventions appropriate for patients at the end of life. These interventions are derived from the best 
practices for care in the last days or hours of life as practiced in the home hospice setting and have been 
modified for use in the acute inpatient care setting. The Comfort Care interventions are based on the American 
Board of Medical Specialty guidelines and core principles for clinical policy and professional practice in end-of-
life care, subsequently endorsed by the National Quality Forum Consensus standards for Palliative Care and 
End-of-life Care. There is a growing body of evidence for each of the Comfort Care interventions. Evidence of 
safety and efficacy can be found in The 2004 Cochrane Library.  
CPRS Comfort Care Order Set (CCOS). These order sets can be customized for individual patients, and 
all or some of the orders can be selected as appropriate and integrated into existing care plans. The orders are 
designed to compliment and be used with disease-modifying therapies. There is no requirement to discontinue 
any other therapies or treatments if the CCOS is activated. The CCOS has also been condensed into a pocket 
card for easy reference and distributed to staff at each of the 48 sites.  
Policy and Procedure Changes: One of the barriers for the BEACON study was the need to add certain 
medications or routes to the pharmacy formulary. This required working with the P&T Committee and Nursing 
Committees to change policies and procedures, particularly those related to morphine concentrate, 
scopolamine patches for secretions, and use of subcutaneous lines and medications to provide pain pumps or 
small doses of medications. To facilitate such changes in the proposed study, samples of appropriate policies 
and procedures are provided in the Education Intervention Packet  
C. Implementation Strategies 
All PCCTs in both arms of the study will be exposed to their assigned implementation strategy for 4 months. 
Trainees in both arms will receive training in teaching and charged to train as many appropriate providers as 
possible over the next 4 months. PCCTs will be given sign-in sheets to use when they offer group or one-on-
one training. Sign-in sheets will include the name, email address, and VA unit/department where they work. 
They will be asked to return their sign-in sheets by fax or secure email (PKI) at least monthly and will be 
prompted when they are due. These sign-in sheets will be used later to identify providers for emailing 
questionnaires regarding their training experiences (Aim #4). During the 4 months of implementation, the 
training team will be available for telephone consultation to answer questions about patient cases or the 
training process at the site. In addition to offering training to other providers in the 4-month implementation 
window, PCCTs will be expected to develop a plan to sustain training opportunities, so that new employees 
can be exposed to the basic principles of Palliative Care and the CCOS. This is vital for residents who rotate 
through the VA on a monthly basis. 
Teleconference Teaching. An 80-minute teleconference will be scheduled to provide an orientation for PCCTs. 
A trainer from the BVAMC will review the Education Packet Materials and the basic functions of the CCOS. The 
session will be guided by a standard PowerPoint slide set for training providers. The session will include the 
importance of training other providers, the concept of culture change, and teaching "when to refer." Although the goal 
will be for PCCTs to complete training of existing staff within the 4-month implementation window, training is 
expected to continue beyond four months. We will contact the sites monthly to collect training records. 

Version 6/20/17 
  
 
Page 10 
 
In-Person Train-the-Champion Workshop. Two PCCT members from each site will be identified to attend the in-
person training. Ideally this will be a physician and nurse practitioner with prescribing privileges, so that both will have 
access to the CCOS and be able to train others. With each wave including five sites, each workshop will train 10 
PCCT members. This group will travel to BVAMC to attend a 2-day, in-person, centralized train-the-champion 
workshop with Dr. Bailey and the training team. The training team at the Birmingham VAMC will provide a 
standardized curriculum for each group of trainees. Training will include review of education materials; case-
based coaching to build care plans using CCOS; hands-on experiences with using the CCOS; coaching to 
become champions (trainers and change agents); review of baseline facility-specific PROMISE data 
(feedback); a charge to train providers at home facility; and expectations for training at the home facility. 
Review of Education Materials 
Trainers will review the hospice and palliative care education materials, with content derived from EPEC for 
Veterans, ELNEC for Veterans and The Palliative Response, particularly modules dealing with the introduction 
to palliative care, anorexia, dyspnea, delirium, pain and pain control, ethical issues, sharing bad news, and 
goals of care. All of the above will then be incorporated into sessions on "Care in the Last Hours of Life." 
Training will be interactive and involve question and answer sessions.  
Case-Based Coaching and Hands-on Experiences with Using the CCOS 
Early in the first day, trainees will receive a hands-on experience with their own CCOS. Three standardized 
patient scenarios have been developed that replicate typical scenarios, such as patients dying with 1) advanced 
cancer, 2) CHF/COPD, and 3) Dementia/stroke. The teams will discuss each case and be coached to use their 
personalized CCOS to build a care plan for the actively-dying patient described in the scenario. This will provide the 
team with hands-on experience, as well as comfort and familiarity with the CCOS built for their home facility. Small 
group discussion will be conducted to review experiences with the CCOS and compare care plans. In the 
afternoon there will be second session using a standardized patient scenario followed by feedback from 
trainers regarding most effective care plans. On the second day of training, the third standardized patient 
scenario will be used to develop a final care plan. There will be a certification checklist to ensure that all 
participants demonstrate competency with use of their CCOS.  
Coaching to Become Champions (Trainers and Change Agents)  
PCCT members will be coached in the use of the Education Packet materials to train other staff in identifying dying 
patients and implementing the orders in the CCOS. Using a training manual, PCCT members will review teaching 
concepts, techniques, and materials for training other providers at their VAMC. A brief training session will be 
demonstrated using a short 20-minute, PowerPoint presentation appropriate for in-service training for the 
majority of nursing and ancillary staff in their hospital. The participants will practice presenting the materials 
and will receive feedback from the faculty and fellow participants. PCCT members will be asked to 
demonstrate their skill as a trainer using either the module for training others physicians, or those with 
prescribing authority or the in-service module for the general staff. On the second day of training, a certification 
checklist will be used to ensure all participants demonstrate competency. We will also introduce the PCCTs to 
academic detailing, a small group training technique modeled after the successful change agent techniques of 
the pharmaceutical sales force. Extensive research on the effectiveness of these techniques suggests this kind 
of education intervention is more likely to change provider behaviors than traditional CME-type presentations. 
The importance of culture change will be emphasized, as will the need for all hospital staff caring for and 
interacting with patients and families to have a basic understanding of CCOS orders. The PCCT members will 
discuss how this may differ from previous approaches to care and the important role they will play.  
 
Charge and Expectations for Training at the Home Facility  
The PCCT teams will be charged to train all providers at their home facility who may have contact with patients 
who are dying. The goal will be for them to complete training of existing staff within the 4-month implementation 
window, but is expected to continue beyond four months whether or not this accomplished.  
Follow-up Consultation 

Version 6/20/17 
  
 
Page 11 
 
During the 4-month implementation, the team at the Birmingham VAMC will be available to assist teams with specific 
problems at their request. In addition, the training team will conduct bi-weekly conference calls for the most recently 
trained PCCT members to discuss the team's local implementation and to brain storm about solutions. Calls will 
include a very brief didactic to review an aspect of palliative care (either requested by the PCCT or identified by 
the training team) Then PCCT members will have time to review cases, ask questions, and receive advise from 
the training team and other PCCT members. Dr. Bailey and team have experience conducting similar calls 
nationally for the VA Implementation Center, including sessions for VISN champions and palliative care 
program managers, and the activity has been perceived as very helpful.  
Measurement 
Aim #1: Process-of-Care Endpoints 
The primary aim of the study is to compare the effectiveness of the Basic and Enhanced Implementation 
Approaches for improving processes of end-of-life care. Data on end-of-life care will be abstracted from the 
CPRS records of veterans who have died [in settings other than the emergency department] in the 48 VAMCs 
9 months before and after the intervention period. As detailed in the power calculation, the study requires an 
average of 100 deaths per site. It is likely that some hospitals will have many more than the required 100 
deaths during the sampling period, in which case, a random sample will be chosen, not to exceed 150. The 
limit is above 100 because it is likely some hospitals will have less than 100 patients die during the sampling 
period. Allowing some hospitals to be above 100 will enable us to satisfy the requirement of 100 on average. 
Methods of Data Collection and Management. Lists of veterans (names and real SSNs) who have died in 
the 48 participating VAMCs 9 months before and after the intervention period will be obtained via direct contact 
with the facilities. Information will be sent to assigned study personnel via encrypted email and stored on the 
secure shared drive (\\v07\bir\BIRResearch\REAP1\1548). Real SSNs will be necessary in order to access the 
health records. We will request remote “Special User” access via VistaWeb and/or CAPRI to the health records 
of deceased veterans at each facility, as well as access to local CPRS if needed. Health records will be 
reviewed and abstracted to capture data related to care at the end of life. Abstracted data will be entered onto 
paper abstraction forms and locked in a secure filing cabinet maintained in the investigator’s locked office in 
the GRECC (8th floor of BVAMC). Abstracted data will also be stored electronically using VA REDCAP. VA 
REDCAP is a web application that facilitates the collection and entry of research data. Electronic data will then 
be imported into VINCI (VA Informatics and Computing Infrastructure), a secure, central analytic platform for 
performing research and supporting clinical operations activities. Future analysis will be conducted within 
VINCI. Applications will be accessed using encrypted VA devices. 
Similar technology and chart abstraction procedures were used successfully in the BEACON project. 
Demographic variables, variables regarding hospital stay and variables related to processes of care will be 
captured by the chart review. The chart abstraction tool developed and used for data collection in the BEACON 
study will be used for this study. We will ensure data validity by checking for inter-rater agreement on 10% of 
the charts abstracted. Additional demographic and clinical data will be pulled from national VA datasets and 
merged with the data from chart abstractions. These variables will include: age (DOB), race, gender, marital 
status, living status, service connection, zip code, religion, income category (1-8), and diagnoses (all ICD-9 
codes).  
Endpoints. The primary endpoint will be presence of an active order for opioid medication at the time of 
death. Opioids are important medicines for the management of pain and dyspnea at life’s end. These 
medicines should be routinely available for treatment of symptom exacerbations. Numerous studies have 
documented that patients dying in the acute care setting are likely to have poorly controlled pain as reported by 
their families. Secondary endpoints will be other processes of care that represent the development of a 
comprehensive care plan for actively dying patients, including orders for and administration of appropriate 
medications, as well as orders that implement goals of care, palliative care consultation, orders to facilitate 
environmental changes that enhance patient safety and comfort, and psycho-social-spiritual support for patient 
and family.  
Orders for and administration of appropriate medications. In addition to the primary endpoint, we will abstract 
orders for and administration of antipsychotic medications for delirium or nausea and vomiting, 

Version 6/20/17 
  
 
Page 12 
 
benzodiazepines for anxiety and treatment or prevention of seizures, and medication to control excessive 
secretions (death rattle). While orders are necessary, they are not sufficient. If patients are in pain or have 
dyspnea, then the nursing services would need to administer the medication. There are numerous barriers to 
opioid administration, which this education intervention is designed to overcome by encouraging clinicians to 
write orders for opioids and  schedule dosing as "offer and patient may refuse" or "nurse will assess for 
symptoms with option to hold," as opposed to PRN. This approach prompts the nursing staff to frequently 
assess for pain and dyspnea and guides them in offering small but effective doses frequently if necessary. The 
educational intervention is designed to relieve opioid phobia and excessive anxiety associated with the 
unfounded fear that small doses will cause death in dying patients. Also, it is important that the orders are 
flexible, so that if the patient is not able to swallow a tablet, sublingual or parental medication is available. 
Finally the subcutaneous route is promoted since starting and maintaining an IV is difficult for many patients at 
end-of-life. The subcutaneous route offers an easy to place, less painful and effective route for parenteral 
opioid administration, insuring that lack of IV assess is not a barrier to pain control   
Communication about goals of care. Indicators of communication regarding goals of care include having a 
DNR order in place at the time of death (documenting goals of care discussion), timing of DNR order, and having 
an Advance Directive. In studies about Advance Directive use, it has been noted that the provider often is 
unaware of the patient’s preferences for end-of-life care, including resuscitation efforts. Often the DNR order is 
written very late in the course of the illness, despite the fact that the prognosis may have been clear from the 
time of admission. Patients and families who have discussions about resuscitation earlier in the course of the 
illness have more time to adjust to the illness, to prepare for declines in patient status, and to make decisions 
in a less stressful environment.  
Palliative care consultation. The Palliative Care Consult is a marker of the recognition of the severity of the 
veteran’s illness with attendant risk for distressing symptoms and potential risk for dying. The CCOS may be 
used by the non-palliative care providers, but they often need assistance to customize the treatment plan for 
optimal symptom control and family support. Over time, as staff improve their skill of identifying the actively 
dying and increasingly understand the importance of symptom control and support to families, the rate of 
Palliative Care Consults can increase and may occur earlier in the course of the illness.  
Environmental Changes. Environmental endpoints will include orders for fans for dyspnea, nasogastric tubes, 
intravenous lines infusing, and restraints, and location of death (intensive care unit versus palliative care unit 
versus other). It is often appropriate to reduce the use of nasogastric tubes and infusing IV or other medical 
instrumentation that is no longer associated with benefit, but often associated with physical discomfort from the 
devices. Environmental changes should encourage family presence when available and appropriate. 
Regarding location of death, patients at the end of life are frequently in the ICU setting for at least some portion 
of their hospitalization. The ICU setting is associated with isolation from family and other support and the use 
of aggressive and invasive treatments that, at the end of life, may not be associated with clinical benefit, and 
have a high risk for iatrogenic harm. Many veterans are admitted to the ICU setting when this may not be 
consistent with goals of care before ultimately being transitioning to a comfort care approach.  
Psycho-social-spiritual support. Indicators of psycho-social-spiritual support for patient and family will include 
pastoral care visits and family presence at or near the time of death.  
In addition to the chart abstractions, the “Reminder Report” programming embedded in the CCOS will identify 
patients for whom the CCOS was used and will allow investigators to track use of the electronic CCOS by site.  
Aim #2: Telephone Interviews with PCCT members 
Telephone interviews will be conducted to qualitatively explore and formatively evaluate PCCT member 
experiences with and perceptions of the Basic and Enhanced Implementation Approaches. To ensure that 
multiple perspectives are included, we will invite all members of the PCCT from each site to participate in an 
interview.  For sites in the enhanced Implementation group, it will be essential to include the two members of 
the team who attended the in-person, train-the-champion workshop, because they will be able to provide 
feedback on the workshop program. For sites in the Basic Implementation group, it will be important to include 
at a minimum, the lead physician and nurse, to provide feedback on the teleconference teaching session. 
Approximately 12-18 weeks following the end of the 4-month implementation window, interviews will be 

Version 6/20/17 
  
 
Page 13 
 
conducted by an interviewer trained in qualitative inquiry and not involved in the training. It is anticipated that 
interviews will not exceed an hour in length. Interview sessions will be tape-recorded and transcribed for 
coding and analysis. We will use semi-structured telephone interviews to guide the inquiry and achieve breadth 
in the evaluation. The Center for Advanced Palliative Care (CAPC) guidelines will be followed in designing the 
telephone interview instrument and conceptualizing the interview process. Separate guides will be developed 
for PCCT members who are prescribers, and those who are non-prescribers. The development of the interview 
guides will be shaped by consensus of the research team using their expertise in palliative care, behavioral 
medicine, geriatrics, education, medical sociology, and qualitative methods. Each item in the interview guide 
will be reviewed by an interdisciplinary team of MDs, PhDs, and nurses for relevance, clarity and readability.   
The interview guides will be used to format the interactions to insure the key research questions are 
addressed. Open-ended questions will cover experiences with and perceptions of the Basic and Enhanced 
Implementation Approaches, including: 1) perceptions of the training, 2) the process of teaching other 
providers, 3) barriers and challenges encountered, 4) responses and solutions to obstacles, 5) how prepared 
or confident they felt to overcome these barriers, 6) facilitators observed, 7) needs and preferences regarding 
the implementation approach, and 8) suggestions for improving training. After each site's interviews are 
completed and reviewed, the team will refine the interview guide to capture more fully critical concepts that 
have emerged and to incorporate them into subsequent interviews.  
Aim #3: Evaluation of PCCT Member Perceptions 
To quantitatively evaluate PCCT member perceptions of the training received with the Basic and Enhanced 
Implementation Approaches we will invite all PCCT members at all sites to complete a structured questionnaire 
after their training. The goal of the survey is to assess the impact of the implementation approach and the 
Comfort Care Education Intervention on trainee attitudes, skills, perceptions, and potential impact on practice 
patterns. The content of the questionnaire will be developed by consensus of the research team with feedback 
from experts in each area represented by the PCCT: physician, nurse, social worker, mental health provider, 
and pastoral care provider. The questionnaire will include closed-ended and open-ended questions. The 
structure of the questions will be guided by the education/evaluation expert and the statistician. The survey will 
include items about the usefulness of the training and its impact in terms of changing 1) their attitudes about 
end-of-life care, 2) ability to recognize dying patients, 3) self-efficacy in ordering appropriate care plans, 4) self-
efficacy to train other providers, and 5) provider behavior/practice (including readiness to use the CCOS). The 
results of the post-training questionnaire will inform ongoing refinement of the telephone interview guide.  
Aim #4: Evaluation of Downstream Impact (Providers Trained by PCCT) 
The fourth aim is to assess the downstream effect of the intervention on providers at all sites who receive 
training from PCCT members, including their perceptions of the usefulness of the training and its impact on 
knowledge, attitudes, skills, self-efficacy for ordering appropriate care, readiness to use the CCOS, and 
practice patterns. Separate questionnaires will be developed for prescribers (who can potentially use the 
CCOS) and for non-prescribers (who can potentially facilitate the identification of patients who are dying and 
assist knowledgeably in the delivery of Comfort Care interventions). Individual providers will be identified from 
the sign-in sheets used by PCCT trainers in both groups and invited via email to complete a questionnaire. 
Two reminders will be sent to non-responders to optimize response rates. Participation will be voluntary and 
confidential. Specifically, these providers’ supervisors and co-workers will not be informed of their participation 
(or non-participation) or their responses to the questionnaires. The sign-in sheets will also provide a count of 
providers who received the education as a measure of reach and description of uptake by discipline. 
Data Analysis Plan 
Aim #1: Analysis of Effectiveness for Improving Processes of Care. Analyses of chart abstraction data will 
begin by summarizing demographic variables and process of care endpoints with descriptive statistics and 
graphs. A key aspect of the descriptive statistics, as well as the analytic methods, will be identification of 
individuals as pre-implementation or post-implementation deaths within a hospital. Analysis of the primary 
endpoint (proportion of patients with an order for pain medication at the time of death) and secondary 
endpoints for the 48 randomized VAMCs will rely upon Generalized Estimating Equations (GEE) as developed 
by Zeger and Liang. Given the nature of the binary outcomes of this grant, one might assume that chi-square 

Version 6/20/17 
  
 
Page 14 
 
analyses or logistic regression could be used to analyze the resulting data. However, such an approach would 
be wrong. Because the probability of receiving an order for opioid naturally varies by hospitals, outcomes for 
individuals nested within a hospital are actually correlated. This correlation, called the intra-class correlation 
coefficient (ICC), violates a key assumption of chi-square analyses and logistic regression. Any variance 
estimates derived without taking into consideration the clustering effects are typically biased downward. GEEs, 
which can be thought of as logistic regression accounting for the clusters and the ICC, correctly estimate the 
variance of treatment effect as long as 30 or more clusters are used.  
In order to test whether the Enhanced Implementation Approach leads to a higher proportion of patients having 
desired outcomes compared to the Basic Implementation Approach, GEE analyses of the post-implementation 
data will include an explanatory variable indicating to which arm the hospital was randomized. These simple 
GEE models, to be developed separately for the primary and each secondary outcome, will produce an 
unadjusted 95% confidence interval for the odds ratio for the outcome at time of death within an Enhanced 
Approach hospital relative to a Basic Approach hospital. If the hypothesized effect of Enhanced Approach 
leading to higher proportions of desired endpoints is true, then these confidence intervals should completely 
exceed an odds ratio of 1. Although confounding should be minimized due to the use of randomization of 
VAMCs, multivariable GEE models will be developed to incorporate patient characteristics that have been 
identified in our previous studies to be associated with the outcomes [(age, race, terminal illness, location of 
death)]. The justification for multivariable models is two-fold. First, inclusion of the covariates will adjust for any 
potential imbalance remaining after randomization. Second, inclusion of covariates should allow for greater 
precision in estimating treatment differences. As with the simple models, it is anticipated that these adjusted 
95% confidence intervals should completely exceed an odds ratio of 1 for the endpoints. Add David’s section 
Aim #2: Qualitative Data Analysis and Interpretation. Analysis of the telephone interviews will be led by Dr. 
Williams, working in conjunction with Dr. Ahluwalia, and Dr. Burgio. Analysis will be conducted concurrently 
with data collection. The process will begin with each team member individually reading initial transcripts to 
gain an overall perspective on the data. The team will progress to initial coding using directed qualitative 
content analysis. Unlike conventional content analysis where initial codes are derived from the data, in directed 
content analysis, the initial coding is guided by a theoretical framework. In this analysis we will employ the 
theoretical framework of symbolic interactionism, a sociological school of thought developed by Herbert 
Blumer. It is based on the premise that human beings, in interaction with one another, create a context of 
meaning to make sense of everyday life. Symbolic interactionism is used widely in qualitative inquiry to 
understand how human beings interpret the events in their social world, create shared understandings of social 
situations, and modify their assumptions and actions to fit the situation. As such, the symbolic interaction 
theoretical framework is ideally suited to shape our initial coding scheme for exploring informants’ perspectives 
on, perceptions of, and experiences with the intervention.  
The team will code initial transcripts simultaneously and meet weekly by conference call to compare codes, 
discuss discrepancies, and reach consensus. During initial coding, the team will pay close attention to 
informants' stance toward palliative care training and characterizations of the trainer-the-champion model. 
Additionally we will focus on informants’ accounts of the dynamics of the training sessions, as well as the lens 
(personal, clinical, institutional) through which the training is processed and assessed. Once an initial 
codebook has been developed, the team will move to focused coding, which involves identifying the most 
relevant and common elements within the codes and applying them to the transcripts. Codes will be 
reassessed and reconsidered throughout this process and some codes will be eliminated, edited, or added. 
The team will continue to meet regularly to discuss emerging themes and interpretations of the data, 
continually referring back to the data to insure consistency and relevance. 
Aims #3 & #4: Analysis of PCCT and Provider Questionnaires. Descriptive statistics will be used to 
summarize responses for each of the questionnaire items completed by the PCCT and the providers they 
subsequently train. GEE and mixed linear models will be used to compare categorical outcomes and 
continuous outcomes respectively between the Basic and Enhanced Implementation arms. 
Dissemination and/or Implementation Plan 
Some of our dissemination work is already underway. For example, materials developed by the investigators 

Version 6/20/17 
  
 
Page 15 
 
are posted on the VA Hospice/Palliative Care Website and the entire text of The Palliative Response is 
available online at: http://www.uab.edu/medicine/palliativecare/training/palliative-response. It was also peer-
reviewed for the End of Life/Palliative Education Resource Center (EPERC) website (www.eperc.mcw.edu) 
and received its highest rating. The materials have also been included in the Growthhouse website 
MegaSearch (www.growthhouse.org), an international clearinghouse for high quality educational materials 
related to hospice and palliative care. A BEACON packet of extensive materials that can be used by sites to 
start CCOS is on the VA Palliative Care SharePoint. Future dissemination will be shaped by the findings of the 
proposed study, which will determine the relative value of the Basic and Enhanced Approaches to 
implementation. Results will be used to refine the Intervention and implementation strategies in preparation for 
nationwide dissemination of best practices for end-of-life care within the VA Healthcare System. 
Challenges and Limitations 
One potential challenge to implementation is providers’ resistance to training due to the tedium of multiple and 
recurring education requirements. In our formative evaluation, informants specifically mentioned using a train-
the-trainer model so the residents would learn about palliative care as part of the regular rotations. We plan to 
overcome resistance by using an approach based on the evidence that interactive, experiential learning 
techniques are more effective than didactic presentations. The Enhanced Implementation Approach 
incorporates three core adult learning concepts, seeking to engage participants in their training, retain their 
interest throughout, and facilitate their interaction with their instructor and fellow trainees. Further, the 
intervention strategy seeks to support behavior change by integrating the electronic order set into the system 
of care as a decision support tool and through the enduring role of the Champion embedded within each 
facility. As an additional measure, to avoid enrolling sites that might be especially resistant to change, VAMCs 
with scores in the lowest 10th percentile will not be recruited. 
Project Management Plan 
Timeline for Proposed Project 
Activity 
Year 1 
Year 2 
Year 3 
Year 4 
Recruit VAMCs to participate 
 
 
Site preparation (IRB, CCOS, admin) 
 
 
Preparation for training (telecon, in-person) 
 
 
Refine data collection tools 
  
  
  
  
  
  
 
 
Implementation (wave #1 - 8 sites) 
  
  
 
  
  
  
  
 
 
Implementation (wave #2 - 8 sites) 
  
  
  
 
  
  
  
 
 
Implementation (wave #3 - 8 sites) 
  
  
  
  
 
  
  
 
 
Implementation (wave #4 - 8 sites) 
  
  
  
  
  
 
  
 
 
Implementation (wave #5 - 8 sites) 
  
  
  
  
  
  
 
 
 
Implementation (wave #6 – 8 sites) 
 
 
 
 
 
 
 
 
 
Telephone interviews 
  
  
  
  
  
  
 
 
Qualitative analysis 
  
  
  
  
  
  
 
 
Chart abstractions 
 
 
Data analysis 
  
  
  
  
  
  
 
 
Preparation of manuscripts 
  
  
  
  
  
  
 
 
Refinement of Implementation Approach 
  
  
  
  
  
  
 
 
Research Team, Facilities, and Resources Required 
Our research team is led by Kathryn L. Burgio, PhD, an established clinical investigator in behavioral science 
and health services research, and F. Amos Bailey, MD, a clinician/educator and expert in palliative medicine. 
The core BEACON team has been working together since 2004. We have weekly, agenda-driven meetings to 
maintain communication within the team, discuss and plan studies, presentations, and manuscripts, provide 
feedback on ongoing activities, communicate expectations, assist each other, and maintain progress towards 
goals. As PI, Dr. Burgio provides scientific leadership and coordinates the interdisciplinary team’s activities. Dr. 
Bailey provides vision and clinical supervision. Beverly Williams, PhD, is a medical sociologist and expert in 

Version 6/20/17 
  
 
Page 16 
 
qualitative analysis, who will lead the qualitative initiatives. Patricia Goode, MD, is a geriatrician and expert in 
end-of-life care, who will assist with implementation. Our biostatistician David Redden, PhD, oversees data 
management/analysis. Marie Bakitas, DNSc, APRN is a professor of Nursing and an established palliative care 
investigator, who will assist the team to plan and conduct PCCT training. Note: Although Dr. Bailey will be 
moving to Denver, CO, he is maintaining employment with the Birmingham VAMC and will fulfill his role on the 
study. He is in the process of obtaining approvals for teleworking, which will allow him to interact with the 
PCCTs, assist with building the CCOS at each facility, monitor chart abstraction, and attend regular team 
meetings. He will conduct teleconferences for the Basic Approach group and return to BVAMC to lead the in-
person Train-the-Champion workshops for the Enhanced Approach group.