Improving Transplant Medication Safety Through a Technology and Pharmacist (ISTEP) Intervention in Veterans
Phase: NA
Conditions: Transplant
Sponsor: VA Office of Research and Development

 
i 
CLINICAL TRIAL PROTOCOL AND STATISTICAL ANALYSIS PLAN 
IIR 15-359 / NCT03860818 
Improving Transplant Medication 
Safety through a TEchnology and 
Pharmacist Intervention (ISTEP) 
 
Last updated August 11, 2021 
 
 
 
 
 
 
 
 
 
 
 

 
ii 
 
TABLE OF CONTENTS 
    
Page 
TABLE OF CONTENTS ............................................................................................................. ii 
PRÉCIS ......................................................................................................................................... iv 
Study Title ................................................................................................................................. iv 
Improving TraNsplant Medication Safety through a TEchnology and Pharmacist 
Intervention (ISTEP) ............................................................................................................... iv 
Objectives ................................................................................................................................. iv 
Design and Outcomes ............................................................................................................ iv 
Interventions and Duration ...................................................................................................... v 
Sample Size and Population .................................................................................................. v 
1 
Study objectives ................................................................................................................. 6 
1.1 
Primary Objective ......................................................................................................... 6 
1.2 
Secondary Objectives .................................................................................................. 6 
2 
BACKGROUND AND RATIONALE ................................................................................. 7 
2.1 
Background on Condition, Disease, or Other Primary Study Focus .................... 7 
2.2 
Study Rationale ............................................................................................................ 9 
3 
STUDY DESIGN................................................................................................................. 11 
4 
SELECTION AND ENROLLMENT OF study sites and PATIENTS ....................... 13 
4.1 
Inclusion Criteria (require both to be included) ...................................................... 14 
4.2 
Exclusion Criteria ....................................................................................................... 14 
4.3 
Study Enrollment Procedures .................................................................................. 14 
5 
STUDY INTERVENTIONS ............................................................................................... 15 
5.1 
Interventions ................................................................................................................ 15 
5.2 
Training for Study Interventions ............................................................................... 17 
5.3 
Usual Care................................................................................................................... 19 
5.4 
Adherence Assessment ............................................................................................ 19 
6 
STUDY PROCEDURES .................................................................................................... 20 
7 
SAFETY ASSESSMENTS ............................................................................................... 20 
7.1 
Serious Adverse Events (SAEs) .............................................................................. 21 
7.2 
Reporting Procedures ................................................................................................ 22 
8 
INTERVENTION DISCONTINUATION .......................................................................... 22 

 
iii 
 
9 
STATISTICAL CONSIDERATIONS ............................................................................... 22 
9.1 
General Design Issues .............................................................................................. 23 
9.2 
Sample Size and Randomization ............................................................................ 24 
9.2.1 
Treatment Assignment Procedures ................................................................. 24 
9.3 
Outcomes .................................................................................................................... 24 
9.3.1 Primary outcome ...................................................................................................... 24 
9.3.2 
Secondary outcomes ......................................................................................... 25 
9.4 
Data Analyses ............................................................................................................. 26 
10 
DATA COLLECTION AND QUALITY ASSURANCE ............................................. 29 
10.1 
Data Collection and Management ........................................................................... 29 
10.2 
Quality Assurance ...................................................................................................... 29 
10.3.1  Training ................................................................................................................... 30 
10.3.2 
Quality Control .................................................................................................... 30 
10.3.3 
Metrics .................................................................................................................. 31 
10.3.4 
Protocol Deviations ............................................................................................ 31 
10.3.5 
Monitoring ............................................................................................................ 31 
11 
PARTICIPANT RIGHTS AND CONFIDENTIALITY................................................. 32 
11.1 
Institutional Review Board (IRB) Review ................................................................ 32 
11.2 
Informed Consent Forms and Procedure for Obtaining Consent ....................... 32 
11.3 
Participant Confidentiality ......................................................................................... 34 
11.4 
Study Discontinuation ................................................................................................ 34 
12 
ETHICAL CONSIDERATIONS .................................................................................... 35 
13 
COMMITTEES ................................................................................................................ 35 
14 
PUBLICATION AND DISSEMINATION OF RESEARCH FINDINGS .................. 36 
15 
REFERENCES ............................................................................................................... 37 
16 
SUPPLEMENTS/APPENDICES ................................................................................. 41 
  
 

 
iv 
 
PRÉCIS 
Study Title  
Improving TraNsplant Medication Safety through a TEchnology and Pharmacist 
Intervention (ISTEP) 
Objectives 
 Primary 
1. 24-month rate of hospitalization and emergency room visits compared 
between the intervention and usual care groups 
 
Secondary 
2. 24-month total estimated health care costs, compared between the 
intervention and usual care groups 
3. Estimated graft survival rates, compared between the intervention and usual 
care groups and defined as the proportion of patients that continue to have a 
functioning allograft at the end of the 24-month study 
4. Estimated patient survival rates, compared between the intervention and 
usual care groups and defined as the proportion of patients alive at the end of 
the 24-month study 
5. Potential medication safety issues, defined as the proportion of patients with 
the following, based on automated reports for the transplant medication 
dashboard: 
a. Percent of patients with missing labs 
b. Percent of patients alarming lab values without follow up scheduled 
c. Immunosuppression adherence, estimated using the proportion of 
days covered (PDC) 
d. Percent of patients on significant drug interaction without a 
immunosuppressant level 
e. Percent of patients with hospital discharge or ED visit without follow up 
scheduled 
6. Proportion of alerts that were deemed clinical relevant and actionable by the 
intervention pharmacists 
7. Proportion of interventions that were deemed to be accepted when made to 
other providers based on dashboard reporting information 
8. Average time required to respond to each alert 
Design and Outcomes   
This is a 24-month, prospective, multicenter, cluster-randomized, parallel-arm, 
controlled clinical trial assessing the impact of a pharmacist-led technology-
enabled intervention designed to improve immunosuppression medication safety 

 
v 
 
in veteran organ transplant recipients.  The study includes 10 sites, five sites 
randomized to standard clinical care and five to standard care plus the 
technology-enabled pharmacist intervention.   
Study efficacy will be determined by comparing the rates of hospitalizations and 
ER visits between intervention and control study site veteran organ transplant 
recipients, while adjusting for baseline patient, provider and facility 
characteristics.  Secondary measures include comparing estimated healthcare 
costs and determining dashboard functionality, dashboard actionability and 
pharmacist intervention types and acceptance rates.  We will also assess the 
overall incidence and severity of drug-related problems and graft and patient 
survival rates and compare these between the intervention and control sites. 
Interventions and Duration  
The study will randomize 10 sites to usual care or usual care plus intervention in 
a parallel arm design and study duration will be 24-months.   
The intervention will include a pharmacist that utilized a dashboard system to 
identify veteran organ transplant recipients with the potential of 
immunosuppressant medication-safety issues.  The technology component of 
this intervention consists of the use of a dashboard system that performs 
population-level surveillance of organ transplant recipients and identifies those 
with potential drug-related problems, including non-adherence to 
immunosuppression medications, drug interactions, missing and worrisome 
trends in labs and recent hospitalizations and ED visits; then providing a real-
time alert to the pharmacist, who will determine its relevance and intervene in an 
appropriate protocol-guided manner.   
Sample Size and Population  
There will be 10 sites included in this study with an estimated average of 169 
veterans per site, totaling 1,689 veterans across the entire study population, 
divided approximately equally between the intervention and usual care groups 
The patient population will consist of veterans that received solid organ 
transplants (kidney, liver, pancreas, heart, or lung) and receive their 
immunosuppression medications through the VA system.  Patients will be 
identified using ICD-9 or 10 codes and VA medication prescription information.

 
6 
 
 
1 
STUDY OBJECTIVES 
1.1 Primary Objective 
Hypothesis:   
1. Veterans in the intervention group will have significantly lower rates of 
hospitalizations and emergency room visits, as compared to the control 
group, by the end of the 24-month study 
 
Objective:   
1. Through a 24-month prospective, parallel arm, cluster-randomized, 
controlled multicenter study, measure the effectiveness of a pharmacist-
led, technology-enabled intervention on reducing the rate of 
hospitalizations and emergency room visits in Veteran organ transplant 
recipients, as compared to usual care  
1.2 Secondary Objectives 
Hypotheses:   
1. Veterans in the intervention group will have significantly reduced health 
care costs, as compared to the control group, by the end of the 24-month 
study 
2. The immunosuppressant dashboard system will provide near real-time, 
actionable alerts that will allow the pharmacist to conduct timely intervenes 
on potential or ongoing drug-related problems 
 
Objectives:   
1. 24-month estimated health care costs, compared between the intervention 
and usual care groups 
2. Estimated graft survival rates, compared between the intervention and 
usual care groups  
3. Estimated patient survival rates, compared between the intervention and 
usual care groups  
4. Potential medication safety issues, defined as the proportion of patients 
with the following, based on automated reports for the transplant 
medication dashboard: 
a. Percent of patients with missing labs 
b. Percent of patients alarming lab values without follow up scheduled 
c. Immunosuppression adherence, estimated using the proportion of 
days covered (PDC) 

 
7 
 
d. Percent of patients on significant drug interaction without a 
immunosuppressant level 
e. Percent of patients with hospital discharge or ED visit without follow up 
scheduled 
5. Proportion of alerts that were deemed clinically relevant and actionable by 
the intervention pharmacists 
6. Proportion of interventions that were deemed to be accepted when made 
to other providers based on dashboard reporting information 
7. Average time required to respond to each alert 
 
2 
BACKGROUND AND RATIONALE  
 
2.1 Background on Condition, Disease, or Other Primary Study Focus 
Within organ transplant, despite dramatic improvements in acute rejection rates, long-
term graft survival has not improved to nearly the same degree.  In kidney transplant, 
since 2003, there has been a 50% reduction in acute rejection rates; yet, during this 
same period of time, the allograft half-life has only increased by a modest 0.6 years.6,7  
The most recent report from the Scientific Registry of Transplant Recipients (SRTR) 
demonstrates a historically low one-year acute rejection rate of 10%, with a suboptimal 
five-year graft survival rate of 70%.4  Drug-related problems, which encompass 
medication errors, non-adherence and adverse drug events, are a predominant cause 
of deleterious clinical outcomes in kidney transplant recipients; most notably, graft 
loss.22  Our previous research, as well as studies from other groups, has demonstrated 
that approximately two-thirds of transplant recipients will experience at least one 
medication error.11-13  Of more concern, nearly one in eight kidney transplant recipients 
experience a medication error which directly contributes to hospitalization and doubles 
the risk of graft loss.  These medication errors are usually the result of unintentional 
non-adherence: patients have difficulty obtaining medications, gets confused from the 
fragmented care they receive or forget to take medications in a timely manner.14  Non-
adherence has now been recognized as a major contributor to late acute antibody 
mediated rejection (AMR), the development of donor specific antibodies (DSA) and 
subsequent graft loss.  In a landmark study by Sellares et al, 315 kidney transplant 
recipients were followed for roughly three years post-transplant; 47% of the 50 allografts 
that failed during this time were due to AMR; 32% of patients were identified as non-
adherent and one-half of all AMRs were due to this.  Remarkably, medication non-
adherence was 10 times more frequent in patients with graft failure (32% vs. 3%, 
p<0.001).22  As most medication non-adherence is unintentional, with the proper 
monitoring and follow-up, this devastating risk factor is modifiable.  Other drug-related 
problems, beyond non-adherence, also significantly contribute to deleterious post-
transplant events.  A study by Friedman et al demonstrated that of 149 medication 
errors identified in 93 ambulatory organ transplant recipients, 44% were due to 
iatrogenic causes, including 13% from a prescribing error and 13% from a dispensing 
error.  Importantly, 26% of these errors led to clinical significant events, including 

 
8 
 
laboratory abnormalities (8%), physical issues (5%), hospital admission (8%) and an 
outpatient intervention (8%).  Thus, to improve medication safety, interventions should 
not only focus on the patient, but also engage the health care providers.11 
Although immunosuppression is effective at preventing rejection, adverse drug events 
are nearly universal and associated with significant morbidity.  Several studies suggest 
that adverse drug events, particularly infection from over-immunosuppression and 
calcineurin inhibitor nephrotoxicity, may be a predominant cause for the discordance 
between reductions in acute rejection and lack of improvements in graft survival.  
Parasuraman et al demonstrated that infectious etiologies have surpassed rejections as 
the leading cause of graft loss.9  Our previous research demonstrates that 
immunosuppressant adverse drug events are associated with medication errors; 
patients that experience medication errors leading to hospitalization have 2.3 times the 
risk of developing at least three adverse drug events (p=0.020).14  In other 
comorbidities, adverse drug events have clearly been established as a major risk factor 
for medication non-adherence.23-26 Therefore, early recognition of adverse drug events 
in transplant recipients will likely help prevent downstream clinical sequelae, including 
non-adherence and irreversible immunosuppressant toxicities.  Research demonstrates 
that clinical pharmacists have the unique education and training to both identify these 
events early, while also developing strategies to mitigate or resolve the associated 
sequelae.16,17,27-31  
Also of concern with the use of chronic immunosuppression therapy in the ambulatory 
care setting are drug interactions.  The calcineurin inhibitors and mTOR inhibitors, 
which are widely utilized immunosuppressants, are metabolized through the cytochrome 
P450 3A4/5 enzyme system, making them prone to interactions.  Due to their effects on 
the renal and nervous system, these drugs are also frequently associated with 
pharmacodynamic drug interactions and drug-disease interactions.  This includes 
interactions with classes of medications that are the most frequently prescribed in the 
outpatient setting, such as NSAIDs and ACE inhibitors.  These interactions can result in 
an acute and dramatic reduction in renal function and electrolyte disturbances, 
particularly hyperkalemia.32,33  The impact of drug-related problems leading to graft loss 
on clinical and economic outcomes cannot be overstated.  Annual death rates are more 
than three times higher in those with kidney allograft failure (9.4%), as compared to 
those with a functioning transplant (2.8%).34  A well-functioning kidney transplant has 
also been shown to dramatically reduce the progression of cardiovascular disease and 
associated events.35-37  In other organ types, graft loss universally leads to either 
retransplant or death.  In terms of cost, transplantation is highly cost-effective.  
However, due to high and varied peri-operative costs associated with this surgery, the 
break-even point can range from two to 11 years after transplant.38,39  Once a kidney 
transplant fails, patients return to dialysis and costs to provide care accrue at a 
significantly higher rate.40,41  Our research indicates that kidney transplant recipients 
that experience clinically significant medication errors spend five more days in the 
hospital for readmissions, costing more than $18,000 per case.14  These data, taken in 
its entirety, establishes the need for innovative interventions designed to improve 
medication safety and reduce drug-related problems in transplant recipients, by 
decreasing medication errors, non-adherence and adverse drug events;10 Such 
medication safety improvements are needed to demonstrate significant progression in 

 
9 
 
the optimization of long-term graft outcomes and patient survival. 
 
2.2 Study Rationale 
Previous research provides evidence that this clinical trial is grounded on a strong 
conceptual model explaining the etiologies of medication safety events in transplant 
patients and supported by empirical evidence demonstrating the use of technology and 
pharmacist-led interventions can improve outcomes in transplant patients.  The specific 
rationale for this study are based on the following: 
 
Medication Safety Issues in Transplantation: 
• Medication safety events are common in transplant recipients.  In an analysis we 
conducted of 200 prospectively monitored transplant recipients, significant 
medication errors occurred in 64% of patients, with immunosuppressants 
associated with the error 48% of the time.  Patient-related issues (taking the wrong 
dose prescribed or missing doses) caused 68% of these errors, with provider and 
systems issues accounting for the remaining third.  Significant adverse drug 
events, defined as the Common Terminology Criteria for Adverse Events (CTCAE) 
grade 3 or higher, occurred a total of 327 times and within 87% of the study cohort.  
Patients that experienced a significant medication error contributing to a 
hospitalization developed more adverse drug events (2.1±1.0 vs. 1.6±1.1, 
p=0.030).14 
• Previous research has demonstrated that transplant recipients numerous risk 
factors for the development of medication safety events, including: taking >10 
medications concomitantly with more than 30 doses ingested per day, being 
prescribed narrow therapeutic index medications that are prone to drug 
interactions, taking chronic immunosuppressants with known debilitating side 
effects and having frequent dosage adjustments that occur during distantly, over 
the phone, during chart reviews.  Additionally, long-term ambulatory transplant 
recipients usually receive care across multiple health care organizations; thus 
fragmented care, omissions, duplications and discrepancies in medication 
regimens are common within these patients.  We have also established this as a 
major issue facing Veteran transplant recipients.3,10-13 
  
Pharmacist-Led Interventions to Improve Medication Safety in Transplantation: 
• Previous studies demonstrate that interdisciplinary quality improvement endeavors, 
which include the empowerment of pharmacists to implement initiatives aimed at 
improving medication use, can substantially improve medication safety in 
transplant recipients during the perioperative phase.  This data demonstrated a 
40% reduction in adverse drug events leading to hospitalization, 14% reduction in 
length of stay >3 days and 50% reduction in 14-day readmissions.  Our follow-up 
study, which measured the impact of a pharmacist-led formal medication 
reconciliation initiative, demonstrated that, on average, 2.3 medication errors per 

 
10 
 
transplant recipient discharge were prevented (3.4 vs. 1.1, p<0.001, respectively); 
although, medication errors were still apparent in a significant number of 
discharges, establishing the need of pharmacist involvement in the ambulatory 
phase of care for these high-risk patients.16,17 
 
Medication Safety in Veteran Transplants: 
• Veteran transplant recipients are embedded within highly complex inter-facility 
systems of care, such that medication safety monitoring and care coordination in 
the ambulatory care setting is often fragmented and suboptimal.  Our previous 
research has demonstrated that nearly two-thirds of Veteran transplant recipients 
are dual-users; with 62% having multiple providers managing the same conditions.  
This leads to a significant number of duplications and omissions in care.  
Medication discrepancies between systems are nearly universal as well.  Thus, 
provider and system-level issues represent substantial reinforcing and enabling 
factors driving medication safety events in Veteran transplant recipients.3    
• We completed an interrupted time series analysis, demonstrating significantly 
improved adherence to appropriate immunosuppression laboratory monitoring by 
using a population surveillance dashboard to identify Veteran transplant recipients 
with missing laboratory data, allowing the pharmacist to efficiently intervene by 
engaging the Veterans to either schedule labs or document non-VA care.  Using 
segmented regression, we demonstrated a 38% improvement in monitoring 
(p<0.001), at a rate of 4.7% improvement per month for the first six months of the 
intervention (p<0.001).18  Thus, we have strong preliminary data that the 
dashboard is functional and can be an effective instrument to identify Veterans 
needing attention in a time-sensitive and efficient manner.  This dashboard system 
is currently being utilized at multiple sites (one in VISN 7 and several in VISN 12). 
For this proposal, we plan to expand the dashboards functionality and incorporate 
its use by pharmacists as part of clinical care at the 5 sites randomized to the 
intervention arm of this study. 
 
The cluster-randomized design of this study was chosen for a number of important 
reasons.  First, this study design will allow investigators to test a promising intervention 
against a similar control group with respect to patient constitution and time.  The use of 
randomization at the site level will allow us to complete the 24-month study within a 
reasonable period and within the budget constraints.  Using the cluster randomized 
parallel design at 10 sites will provide sufficient patient numbers to meet statistical 
power to detect clinically meaningful differences in health care utilization.  
Randomization at the patient-level, as opposed to the site, would not be feasible, as 
there would be a high probability of cross-contamination based on the intervention 
proposed and the technology component, which uses site-specific population 
surveillance.  Additionally, randomization at the patient-level would substantially prolong 
enrollment, thus making completion within 4 years infeasible.  Other trial designs were 
considered and discussed with the investigational group, including stepped-wedge, 
interrupted time series, one group pre-post test, one group post-test only and a 

 
11 
 
traditional patient-level randomization.  The stepped wedge is the best alternative to our 
proposed study but would provide limited time to evaluate outcomes at some of the 
sites.  Given the proposed intervention type, the use of population-surveillance 
technology, the number of sites recruited and the time and budget constraints, the 
investigative team felt that the parallel-group, cluster randomized controlled trial was the 
strongest in terms of methodology, internal validity, statistical power, feasibility and 
likelihood of success. 
 
3 
STUDY DESIGN 
This is a 24-month prospective, parallel arm, cluster-randomized, controlled multicenter 
study in veteran organ transplant recipients receiving immunosuppressant medications 
through the VA system.  The trial will randomize 10 VA sites, 5 to standard of care and 
5 to standard of care plus the intervention.  The intervention consists of a pharmacist-
led health services endeavor utilizing a transplant medication dashboard system that 
monitors and reports on potential immunosuppression-related safety issues.  The study 
will use prospective methodology to monitor patients for 24-months following site-
randomization, comparing outcomes between the intervention and control sites.  Clinical 
outcomes that will be assessed include health care encounters (hospitalizations and ER 
visits), estimated health care costs, and graft and patient survival.  The functionality of 
the dashboard system will also be assessed by analyzing the proportion of actionable 
alerts created by the system as well as the proportion of missing labs, proportion of 
alarming lab values without follow up scheduled, estimated immunosuppression 
adherence (using the proportion of days covered [PDC]), proportion of significant drug 
interactions without an immunosuppressant level and the proportion of hospital and ER 
visits without follow up. 
The study population will consist of veteran organ transplant recipients that receive 
immunosuppressant medications through the VA system.  Organ transplant recipients 
will be identified using ICD-9/10 codes from the VA electronic health record.  Once 
identified, active medications will be queried and those receiving immunosuppressant 
medications to prevent rejection (tacrolimus, cyclosporine, azathioprine, 
mycophenolate, sirolimus, everolimus, or belatacept) will be included in the study. 
The study will take place at 10 VA sites, with 5 being randomized 5 to standard of care 
and 5 to standard of care plus the intervention.  The 10 site locations are listed in the 
participating sites section of this protocol.  After randomization and site initiation, all 
intervention sites will go live on the dashboard and pharmacist-led intervention 
simultaneously.  Following this, patients at both the intervention and standard of care 
sites will be prospectively monitored for 24 months, in a parallel fashion.   
The sites randomized to the intervention arm will continue to use current standard of 
care procedures within their sites, while also utilizing the dashboard system daily to 
identify veteran transplant recipients with potential medication safety issues.  Each day, 
the pharmacists will open the dashboard system, which is updated at approximately 
7:00 AM each morning.  The system will identify new patient-specific alerts that will 
inform the pharmacist on potential medication safety issues.  The pharmacist will then 
serve as a patient navigator, intervening to resolve the medication safety issue, by using 

 
12 
 
detailed protocols within the standard operating procedure (SOP) manual that the 
pharmacists will be trained on prior to study initiation.  Once the pharmacist validates 
that the dashboard alert is a relevant issue, they will develop a management plan using 
these protocols (Figure above), discuss the recommendations with the providers (when 
necessary), agree on a plan, and implement the plan.  The primary medication safety 
issues the intervention pharmacists will be alerted to, and address are laboratory 
abnormalities, medication non-adherence, drug interactions and medication 
coordination/communication issues.  For laboratory abnormalities, common issues 
encountered within transplant patients include electrolyte irregularities, organ 
dysfunction, drug level outliers and cytopenias.  For each of the laboratory values that 
will be monitored and reported in the dashboard, a detailed algorithm in the standard 
operating procedures (SOP) manual delineating how to address the issues will be used 
for the pharmacist-led intervention.  For instance, it is well known that magnesium levels 
are low in transplant recipients due to a side effect of calcineurin inhibitors.50  Strategies 
to address this include dietary interventions and supplementation.  Issues with out-of-
range drug levels will also be addressed.  These include ensuring the patient is taking 
the correct dose, ensuring the level is a true trough value, checking for new drug 
interactions and adjusting the dose when necessary.  When medication non-adherence 
is identified as a potential issue, the pharmacist will determine if it is deliberate or 
unintentional.  Strategies to address deliberate non-adherence include removing 

 
13 
 
perceived or actual barriers, using motivational interviewing, and addressing side effects 
and cost issues.  For unintentional non-adherence, pharmacists can implement trigger 
reminder strategies, simplify regimens, and re-educate.  Within drug-interaction alerts, 
the intervention will focus on reducing the impact of these through changing of regimens 
(when appropriate), educating patients and/or providers and increased monitoring and 
surveillance.51  Alerts for medication coordination issues will encompass discharges 
from the ER and hospital and missed laboratory assessments.  The intervention 
pharmacist will ensure accurate and safe medication regimens through medication 
reconciliation and improved medication safety surveillance through the scheduling and 
follow up of laboratory assessments.17  To ensure fidelity across sites and efficiency for 
providers, the specific day-to-day tasks involved with this intervention, which include 
assessing and intervening on the alerts and appropriate follow up with patient and 
providers, will be delivered using the SOP manual.15-17   
 
4 
SELECTION AND ENROLLMENT OF STUDY SITES AND PATIENTS  
Patients will be included in this study if they are veteran organ transplant recipients that 
receive immunosuppressant medications through the VA system.  Organ transplant 
recipients will be identified using ICD-9/10 codes from the VA electronic health record 
(CPRS), with this data housed in the CDW.  Once identified, active medications will be 
queried and those receiving immunosuppressant medications to prevent rejection 
(tacrolimus, cyclosporine, azathioprine, mycophenolate, sirolimus, everolimus, or 
belatacept) will be included in the study.  There are no exclusion criteria for patients in 
this study as recruitment is at the VA site-level.  As this is a cluster-randomized study 
and randomization will occur at the site level and not the patient-level, participants will 
not be recruited for the study, and data collection will occur through querying and linking 
the VA CDW, the SRTR transplant registry and CMS data repositories.  We will request 
waiver of written documentation of patient informed consent. We will also request 
HIPAA authorization waiver as well for all patients.  The following protected health 
information (PHI) elements will be required to complete the study with the HIPAA 
authorization waiver. 
 
 
PHI Required for Study 
Rationale 
Social Security Number 
Require real SSNs to create finder file 
and send to Scientific Registry of 
Transplant Recipients (SRTR) to acquire 
baseline transplant information and graft 
outcomes 
Date of transplant, date of clinical events 
(including graft loss, rejections, 
Require dates to ensure events occurred 
during the study period and to conduct 

 
14 
 
hospitalizations, emergency department 
visits and death) 
time to event survival analyses for 
primary and secondary aims 
Zip codes 
Require zip code to assess and adjust for 
potential geographic confounding in 
multivariable analyses 
 
4.1 Inclusion Criteria (require both to be included) 
• Veteran organ transplant recipients will be identified using ICD-9/10 codes from 
the VA electronic health record (CPRS).  Patients must have an active code 
stating they are a recipient of an organ transplant.  The following codes will be 
utilized: 
o ICD-9 codes: V42.0, V42.1, V42.6, V42.7, V42.83, V42.84, 996.81, 
996.82, 996.83, 996.84, 996.86, 52.80 
 
                                               OR 
o ICD-10 codes: C80.2, T86.1, T86.10, T86.11, T86.12, T86.13, T86.19, 
T86.2, T86.20, T86.21, T86.22, T86.23, T86.290, T86.298, T86.3, T86.30, 
T86.31, T86.32, T86.33, T86.39, T86.4, T86.40, T86.41, T86.42, T86.43, 
T86.49, T86.810, T86.811, T86.812, T86.818, T86.819, T86.9, Z48.2, 
Z48.21, Z48.22, Z48.23, Z48.24, Z48.280, Z48.288, Z48.298, Z94.0, 
Z94.1, Z94.2, Z94.3, Z94.4, Z94.83 
 AND 
 
• Actively receiving at least one anti-rejection medication dispensed by the VA site.  
These medications include tacrolimus, cyclosporine, azathioprine, 
mycophenolate, sirolimus, everolimus, and belatacept. 
 
4.2 Exclusion Criteria  
• There are no exclusion criteria for patients in this study as recruitment is at the 
level of the VA site. 
• All veterans meeting inclusion criteria will be monitored by the dashboard system 
and will be included in the outcomes assessment.  Patients may enter or exit the 
study in a rolling manner, which will be accounted for during analyses. 
 
4.3 Study Enrollment Procedures  
Once IRB approval is obtained (via central IRB), each potential participating pharmacist 
will undergo informed written consent.  Following this, each site will be randomized to 

 
15 
 
standard care or standard care plus the pharmacist-led intervention.  To ensure a 
roughly equal number of patients between the two comparison groups, randomization 
will be stratified by estimates veteran organ transplant recipients (<125 versus ≥125 
active patients).  After randomization, each participating pharmacist will be informed of 
their assigned group.  Those in the intervention arm will be trained on the dashboard 
system, utilizing the dashboard, and delivering the intervention.  Those in the usual care 
group will continue to provide the same level of care they are currently providing as part 
of their normal day-to-day activities and job functions. 
If a site or participating pharmacist withdraws prior to study initiation, if possible, it will 
be replaced with a suitable alternative site/pharmacist.  If a site pharmacist withdraws 
after initiation, the investigational team will determine if the site needs to be replaced.  
This will depend on how far along the intervention is and the number of veterans 
impacted by the withdrawal.  
Because this is a cluster-randomized trial, and randomization occurs at the site level, 
we will not recruit, consent, or enroll at the subject level.  We will request waiver of 
written documentation of patient informed consent. We request waiver of HIPAA for all 
patients.   
 
5 
STUDY INTERVENTIONS  
 
5.1 Interventions  
The sites randomized to the intervention arm will continue to use current standard of 
care procedures within their sites (described below), while also utilizing the dashboard 
system daily to identify Veteran transplant recipients with potential medication safety 
issues.  Each day, the pharmacists will open the dashboard system, which is updated 
each morning.  The system will identify new patient-specific alerts that will inform the 
pharmacist on potential medication safety issues.  The pharmacist will then serve as a 
patient navigator, intervening to resolve the medication safety issue, by using protocols 
embedded within the SOP manual that the pharmacists will be trained on prior to study 
initiation.  Once the pharmacist validates that the dashboard alert is a relevant issue, 
they will develop a management plan using these protocols, discuss the 
recommendations with the providers (when necessary), agree on a plan, and implement 
the plan.   
The four primary medication safety issues the intervention pharmacists will be alerted to 
and address are laboratory abnormalities, medication non-adherence, drug interactions 
and medication coordination/communication issues.  For laboratory abnormalities, 
common issues encountered within transplant patients include electrolyte irregularities, 
organ dysfunction, drug level outliers and cytopenias.  For each of the laboratory values 
that will be monitored and reported in the dashboard, a detailed algorithm will be 
provided delineating how to address the issue, within the SOP manual (see below).  As 
one example, it is well known that magnesium levels are low in transplant recipients due 
to a side effect of calcineurin inhibitors.50  Strategies to address this include dietary 
interventions and supplementation.  Issues with out-of-range drug levels will also be 

 
16 
 
addressed.  These include ensuring the patient is taking the correct dose, ensuring the 
level is a true trough value, checking for new drug interactions and adjusting the dose 
when necessary.  When medication non-adherence is identified as a potential issue, the 
pharmacist will determine if it is deliberate or unintentional.  Strategies to address 
deliberate non-adherence include removing perceived or actual barriers, using 
motivational interviewing, and addressing side effects and cost issues.  For 
unintentional non-adherence, pharmacists can implement trigger reminder strategies, 
simplify regimens, and re-educate.  Within drug-interaction alerts, the intervention will 
focus on reducing the impact of these through changing of regimens (when 
appropriate), educating patients and/or providers and increased monitoring and 
surveillance.51  Alerts for medication coordination issues will encompass discharges 
from the ER and hospital and missed laboratory assessments.  The intervention 
pharmacist will ensure accurate and safe medication regimens through medication 
reconciliation and improved medication safety surveillance through the scheduling and 
follow up of laboratory assessments.17   
To ensure fidelity across sites and efficiency for providers, the specific day-to-day tasks 
involved with this intervention, which include assessing and intervening on the alerts 
and appropriate follow up with patient and providers, are fully developed and 
incorporated within the SOP manual.  This manual was vetted by transplant clinicians, 
provided to each intervention site pharmacist, with appropriate training.  
  

 
17 
 
   
 
5.2 Training for Study Interventions  
Prior to the initiation of the study, the pharmacists randomized to the intervention will 
participate in a two-day training conference held at the coordinating site.  During this 
training, the dashboards functionality will be reviewed, along with the SOP manual 
which fully guides interventions.  During this training conference, the intervention 
pharmacists will also participate in case-based competencies.   
 

 
18 
 
Sessions that occur during this conference include:   
1. General overview of organ transplantation 
2. Review of immunosuppression pharmacology and pharmacokinetics 
3. Review of transplant immunosuppression dashboard specifics 
4. Running reports to create alerts 
5. Assessing clinical relevance of alerts 
6. Alert categories 
7. Interventions to address alerts 
8. Suppressing alerts 
9. Clinical and research documentation 
10. Motivational interviewing techniques – face-to-face and over the phone 
11. Provider collaboration strategies 
12. Case-reviews   
 
The investigational team and outside consultant experts will lead these sessions.  We 
will also utilize expertise from the local VA and transplant center to conduct training 
sessions when needed.    In addition, during the 24-month intervention period, we will 
have monthly teleconferences with the intervention pharmacists at the five sites to 
discuss and address issues with the dashboard and/or intervention.  As an additional 
safety measure and to further ensure intervention fidelity, we will have experts available 
to discuss patient issues with sites participating in the intervention.  These include the 
PI, transplant nephrology, transplant hepatology and an internal medicine physician.  All 
interventions will be documented in the medical record using an approved note 
template. As another quality assurance step, through querying the notes entered into 
CPRS by intervention site pharmacists, we will capture specific interventions made by 
the study site pharmacists; these will be monitored by the PI and investigational team to 
ensure clinical appropriateness.  If there are concerns with specific interventions made 
by study site pharmacists, these will be addressed on a case-by-case basis.  During the 
course of the study, if an intervention pharmacist has a particular issue that is time 
sensitive and is unsure how to address it, they will have the ability to call and/or page 
the PI or investigational team, which includes expertise in transplant-related care.  For 
non-urgent issues, pharmacists can send secure emails to investigational team 
members or bring up the issue during conference calls.  Thus, there will be ample 
training and resources available to all intervention site pharmacists to ensure the study 
is conducted in a safe clinically appropriate manner with strong fidelity. 
 
 
 

 
19 
 
5.3 Usual Care  
Both the control and intervention groups will continue to receive usual care as part of 
this study.  Patients at the control sites will receive standard of care, and patients at the 
intervention sites will receive standard of care plus the pharmacist-led intervention. 
Usual care for Veteran organ transplant recipients across the 10 study sites varies 
somewhat, but generally includes the following:  at most sites, nurse coordinators 
and/or mid-level practitioners are responsible for general transplant patient oversight, 
including ensuring laboratory assessments are scheduled/reviewed and medication 
regimens are accurate and up to date.  However, large patient numbers and workload 
constraints preclude these health care professionals from prospective detailed daily 
monitoring of patients.  If there is non-adherence to laboratory assessments or 
medications, they are rarely identified concurrently or early after they occur.  Thus, 
medication safety issues that arise usually do not get identified or addressed until they 
have caused a clinically significant issue, leading to a health care encounter (clinic visit, 
ER visit, hospitalization).  In addition, during this long-term ambulatory phase of care for 
transplant patients, pharmacists usually act as consultants and are only involved in 
direct patient care if an issue arises, and the nurse or provider engages the pharmacist 
for assistance.  Within usual follow up care, pharmacists do not conduct routine daily 
surveillance of all transplant patients.  Most transplant patients have multiple physician 
providers, both within and outside the VA system, providing care and medication 
management.  This includes a primary care physician to manage common comorbidities 
(hypertension, diabetes, etc.) and a specialist (nephrology, hepatology) to manage 
allograft function; other specialists, including endocrinologists and cardiologists, are 
commonly involved with this patient population as well.  Thus, care coordination and 
communication across these providers and health care organizations can often be 
fragmented.  This usual care environment is complex and difficult to navigate, even for 
patients with high health literacy.3,18   
 
5.4 Adherence Assessment  
To ensure fidelity across sites and efficiency for intervention providers, the specific day-
to-day tasks involved with this intervention, which include running the dashboard, 
assessing and intervening on the alerts and appropriate follow up with patient and 
providers, are fully developed in the SOP manual.  This manual was vetted by 
transplant clinicians, provided to each intervention site pharmacist with appropriate 
training.  In addition, we will have experts available to discuss patient issues with sites 
participating in the intervention.  These include the PI, transplant nephrology, transplant 
hepatology and an internist.   
To assess adherence to be SOP, all interventions will be documented in the medical 
record using approved note templates, including a specific transplant medication 
monitoring note which has already been developed and is in use by pharmacists in 
VISN 7 and VISN 12.  Through querying the notes entered into CPRS by intervention 

 
20 
 
site pharmacists, we will capture specific interventions made by the study site 
pharmacists; these will be monitored by the PI and investigational team to ensure 
clinical appropriateness.  This is particularly the case for interventions involving changes 
in immunosuppression regimens. If there are concerns with specific interventions made 
by study site pharmacists, these will be addressed on a case-by-case basis.  During the 
course of the study, if an intervention pharmacist has a particular issue that is time 
sensitive and is unsure how to address it, they will have the ability to call and/or page 
the PI or investigational team, which includes expertise in transplant-related care.  For 
non-urgent issues, pharmacists can send secure emails to investigational team 
members or bring up the issue during conference calls.   
Adherence to the intervention and the SOP manual will be monitored by the 
coordinating study site.  Specific interventions will be reviewed in a concurrent fashion.  
All deviations or issues that are identified will be reviewed by the study team and 
addressed.  If a concerning pattern of non-adherence develops at a particular study site, 
the PI, in conjunction with the Co-Is and data safety monitoring team as the right to 
remove that site from the study.  Replacement of a site will depend on criteria discussed 
in the study enrollment procedures section of this protocol. 
 
6 
STUDY PROCEDURES 
Because of the nature of this intervention, all care that is provided, including laboratory 
and clinic visits, will be considered as usual care.  The intervention consists of 
increased review of patients by a pharmacists and increased scrutiny of patients’ 
medication regimens and laboratory values, through the use of a dashboard 
surveillance system.  However, any interventions that are made as a result of this 
increased review will be considered usual care as they are directly related to monitoring 
of the organ transplant and immunosuppression medications.  Thus, there are no study 
procedures that will occur above and beyond usual care procedures, which are 
described in the usual care section of this protocol. 
 
The procedures unique to this study includes randomization of sites to use of the 
dashboard (intervention) or usual patient care review,  training of the five intervention 
site pharmacists, site initiation and monitoring of intervention integrity and fidelity during 
the 24-month intervention phase of the study.  No patient-specific procedures or 
interventions will occur that are above and beyond usual care will occur as part of this 
study. 
 
7 
SAFETY ASSESSMENTS  
The impact of study interventions on patient outcomes will be closely monitored by an 
internal oversight committee (IOC). The IOC will consist of the PI, co-investigators, 
study coordinator, data manager and consultants on the proposal. The functions of the 
IOC will include: 1) providing scientific oversight; 2) reviewing all serious adverse events 
(graft loss and death events, compared across intervention groups) or complications 
related to the study; 3) monitoring site adherence to the intervention; 4) reviewing 

 
21 
 
summary reports relating to compliance with research protocol requirements; and 5) 
providing advice on resource allocation. The IOC will meet bi-annually and as 
necessary by telephone. The recommendations of the IOC will be reviewed and the PI 
will take appropriate corrective actions as needed. As part of this oversight, we will also 
recruit an independent transplant nephrologist that is not affiliated with this study to 
provide their expertise and review of all study activities. This expert will conduct the 
following: 
1) Review the research protocol and plans for data and safety monitoring. 
2) Evaluate the progress of the intervention, including periodic assessments of data 
quality and timeliness, site enrollment and retention, participant risk versus 
benefit, integrity of the intervention, and other factors that can affect study 
outcome. 
3) Consider factors external to the study when interpreting the data, such as 
scientific or clinical developments that may impact the safety of study participants 
or the ethics of the study. 
4) Make recommendations to the IOC, IRB, and VA R&D/VA HSR&D for 
continuation or termination of the study. 
5) Protect the confidentiality of study data and monitoring. 
The independent transplant nephrologist will have the authority to discontinue the trial 
temporarily or permanently if they perceive that harm is occurring due to the 
intervention. They will meet with the IOC to review serious adverse event reports, 
patient complaints if any, and site issues. Data will be provided at these meetings on 
key variables that may indicate harm, including, hospitalizations, ED visits, graft loss 
and death. The IOC biostatistician will evaluate the confidentiality and integrity of the 
database and the procedures for recording and storing confidential files. The 
independent transplant nephrologist will also review the elements of the plan to manage 
emergencies. 
 
7.1 Serious Adverse Events (SAEs) 
The intervention for this trial does not involve the administration of medications or 
investigational agents, but rather supplemental pharmacist-led care provided in addition 
to usual care already delivered to veteran organ transplant recipients.  Thus, the 
intervention is considered minimal risk research.  As part of this study, SAEs will be 
monitored, compared between the intervention and usual care group and reported to 
the IOC and independent transplant nephrologist on a bi-annual basis.   
SAEs are defined as hospitalizations, ED visits, graft loss and death.  Hospitalizations 
and ED visits will be obtained from VA CDW data housed in the VINCI environment.  
Graft loss events will be obtained from SRTR data linked to the VA data, while death will 
be obtained from both SRTR and VA CDW data. Due to the minimal risk of this study 
intervention and the fact that the intervention does not involve the administration of 
investigational agents, adverse events, including abnormalities in laboratory values, 
new syndromes or diseases or new or worsening patient symptoms will not be 
monitored or reported as a part of this study.  These will be monitored and addressed 
under normal usual care provided to the veteran at the study sites. 

 
22 
 
 
7.2 Reporting Procedures 
All reporting of safety concerns will occur through the IOC, as described above.  The 
IOC will meet bi-annually and as necessary by telephone. The IOC will review reports 
created by the study biostatistician which incorporate clinical outcomes and SAEs, 
including rates of hospitalization, ED visits, graft loss and death, compared between the 
two groups.  The recommendations of the IOC will be reviewed and the PI will take 
appropriate corrective actions as needed. As part of this oversight, we will also recruit 
an independent transplant nephrologist that is not affiliated with this study to provide 
their expertise and review of all study activities. This expert will conduct the following: 
1) Review the research protocol and plans for data and safety monitoring. 
2) Evaluate the progress of the intervention, including periodic assessments of data 
quality and timeliness, site enrollment and retention, participant risk versus 
benefit, integrity of the intervention, and other factors that can affect study 
outcome. 
3) Consider factors external to the study when interpreting the data, such as 
scientific or clinical developments that may impact the safety of study participants 
or the ethics of the study. 
4) Make recommendations to the internal IOC, IRB, and VA R&D/VA HSR&D for 
continuation or termination of the study. 
5) Protect the confidentiality of study data and monitoring. 
 
8 
INTERVENTION DISCONTINUATION  
The intervention will be discontinued based on the recommendations of the IOC and/or 
the expert transplant nephrologist opinion.  These decisions will be based on bi-annual 
reports provided by the study biostatistician that will report SAE rates compared 
between the intervention and usual care groups.  Discontinuation will be recommended 
if the intervention demonstrates increased harm or potential harm to veterans at sites 
randomized to the intervention.  This decision will be made based on clinically or 
statistically significant differences that are apparent in SAE rates between intervention 
and usual care sites.  Worrisome trends will also be considered in the IOC decision. 
The study sample size and power analyses were not designed to specifically conduct 
interim analyses for outcomes and factor in early termination due to substantial 
improvements or futility.  Thus, only safety concerns will be used as criteria for early 
stoppage. 
If a particular site discontinues the intervention or usual care, attempts will be made to 
replace the site if possible.  However, if the site discontinues the study after 
considerable intervention time has elapsed, data collection for follow up outcomes will 
be locked at that time point and a replacement site will not be recruited. 
 
9 
STATISTICAL CONSIDERATIONS  
 

 
23 
 
9.1 General Design Issues  
The primary hypothesis for the ISTEP study is that pharmacist-led immunosuppressant 
therapy management, facilitated through the use of innovative technology, will 
significantly improve immunosuppressant safety and clinical outcomes in Veteran 
transplant recipients.   
The study will consist of a 24-month prospective, parallel arm, cluster-randomized, 
controlled multicenter study.  The clinical trial will randomize 10 sites, 5 to standard of 
care and 5 to standard of care plus the below-discussed intervention, using prospective 
methodology to monitor patients for 24-months following site-randomization, comparing 
outcomes between the treatment and control sites. The cluster-randomized design of 
this study was chosen for a number of important reasons.  First, this study design will 
allow investigators to test a promising intervention against a similar control group with 
respect to patient constitution and time.  The use of randomization at the site level will 
allow us to complete the 24-month study within a reasonable period and within the 
budget constraints.  Using the cluster randomized parallel design at 10 sites will provide 
sufficient patient numbers to meet statistical power to detect clinically meaningful 
differences in health care utilization.  Randomization at the patient-level, as opposed to 
the site, would not be feasible, as there would be a high probability of cross-
contamination based on the intervention proposed and the technology component, 
which uses site-specific population surveillance.  Additionally, randomization at the 
patient-level would substantially prolong enrollment, thus making completion within 4 
years infeasible.  Other trial designs were considered and discussed with the 
investigational group, including stepped-wedge, interrupted time series, one group pre-
post test, one group post-test only and a traditional patient-level randomization.  The 
stepped wedge is the best alternative to our proposed study, but would provide limited 
time to evaluate outcomes at some of the sites.  Given the proposed intervention type, 
the use of population-surveillance technology, the number of sites recruited and the 
time and budget constraints, the investigative team felt that the parallel-group, cluster 
randomized controlled trial was the strongest in terms of methodology, internal validity, 
statistical power, feasibility, and likelihood of success. 
 
There are three main hypotheses this study aims to test: 
1. Veterans in the intervention group will have significantly lower rates of 
hospitalizations and emergency room visits, as compared to the control group, 
at the end of the 24-month study. 
2. Veterans in the intervention group will have significantly reduced health care 
costs, as compared to the control group, at the end of the 24-month study. 
3. The immunosuppressant dashboard system will provide near real-time, 
actionable alerts that will allow the pharmacist to conduct timely intervenes on 
potential or ongoing drug-related problems. 
 

 
24 
 
9.2 Sample Size and Randomization 
Based on the projected enrollment numbers, there is ample power to detect a 
statistically significant difference between the intervention and control groups with 
regards to the primary aim of hospitalization and ED visit rates.  To determine this, we 
used data from a recent national study conducted between 2009-2012.53 These results 
demonstrated that the rate of ED visits after transplant was 1.269 per person-year and 
48% of those ED visits resulted in hospitalization. We will enroll 10 sites (5 sites for 
each study arm) and will have a total of just more than 1,600 patients that are roughly 
split between the intervention and control sites.  We used a conservative estimate of an 
intra-cluster correlation of 0.05 and calculated a sample size of 1,350 to allow us to 
detect a 25% relative decrease in ED visit and hospitalization rates with 80% power. 
The 25% relative improvement in rates is a conservative estimate of intervention effect, 
based on previous pharmacist-led initiatives the investigators have conducted.15,16  After 
allowing for 15% loss to follow up, we require a total of 1,600 to meet study power. We 
expect to have 1,689 patients based on site estimates.  These power calculations were 
conducted using a two-sided test for counts with Poisson regression adjusting for intra-
cluster correlation and with alpha set at 0.05. 
 
9.2.1 Treatment Assignment Procedures 
Randomization will occur at the site level (cluster randomized trial).  Ten sites will be 
randomized to either the intervention arm or usual care arm using statistical software 
(SAS, SAS Institute, Cary, NC).  Randomized will occur after all sites have agreed to 
participate and site-level regulatory approval has occurred.  Randomization will be 
stratified by site using estimated sample size (<125 vs. ≥125 veteran organ transplant 
recipients).  This will be done to ensure an approximate even distribution of patients 
across study arms. 
 
9.3 Outcomes  
The outcomes that are assessed for this proposal all relate to evaluating the impact of 
an intervention designed to improve medication safety within Veteran organ transplant 
recipients that receive immunosuppression through the VA system.  This population 
represents a high-risk group that has a high incidence of medication errors, non-
adherence and adverse drug events often leading to health care encounters, including 
ER visits and hospitalizations, which cause significant accrual of health care costs.  
 
     9.3.1 Primary outcome   
The primary outcome for this study will include the overall rate of ER visits and 
hospitalizations, compared between the intervention and control groups.  These 
encounters will be captured electronically throughout the 24-month study through the 
use of the CDW, stored in the VINCI environment.  In addition, to ensure encounters are 
captured in a comprehensive manner, we will also link the VA CDW data to CMS 
(Medicare) claims data and capture non-VA ER and hospitalization encounters (after 
study completion).  As most transplant patients are dual users and have Medicare 

 
25 
 
insurance benefits, this linkage will provide a more accurate assessment of these health 
care utilization.3  We expect ER visits and hospitalizations to significantly decrease 
within the intervention arm, towards the latter half (second 12-months) of the study due 
to a lag effect of the intervention on this outcome.  This expectation is based on our 
previous research in this field and assessment of pharmacist interventions on 
medication safety and reduced adverse drug events requiring hospitalization.  ER visits 
and hospitalizations will be assessed and compared as described in the statistical 
analysis plan.  These will be defined based on encounter criteria within the CDW and 
CMS data:  all-cause ER visits and all-cause hospitalizations leading to at least one 
overnight stay will be included as events.    
 
9.3.2  Secondary outcomes   
Secondary assessment will include a cost benefit analysis.  Overall health care costs 
accrued during the 24-month study, as well as those accrued in the 24-months prior to 
study initiation will be analyzed and compared between the control and intervention 
groups.  Cost data will be standardized using the VA Health Economics Resource 
Center (HERC) definitions, which normalizes regional differences in costs due to 
variation in cost of living indices.  As with the primary outcome, we will also acquire and 
link CMS claims data to gain a comprehensive assessment of costs, including those 
that accrue from non-VA care (after study completion).  Costs will also be sub-analyzed 
into three predominant categories: inpatient costs, outpatient costs and pharmacy costs, 
as described in the statistical analysis plan.  To fully assess if the intervention is has a 
direct monetary benefit, we will measure costs of the intervention will include estimates 
of time and resources to implement the dashboard and train and support the 
intervention site pharmacists.  Similar to primary outcome, we expect that as ER visits 
and hospitalizations decrease within the intervention arm, there will be a corresponding 
decrease in overall costs as well, predominantly driven by inpatient costs.  We expect 
this decrease to occur during the second 12 months of the study period, as the 
intervention is likely to have a lag effect on this outcome as well.   
 
Another secondary outcome is to assess the success of the dashboard systems 
expansions and utilization.  To do so, we will evaluate the dashboard’s functionality by 
measuring and reporting descriptive statistics for the alert numbers, alert relevance, 
time and the actions taken with regards to the alert and the intervention magnitude.  We 
will measure and report the time effort needed to conduct the interventions by study site 
pharmacists.  Using REDCap, we will build an electronic data capture system, which 
allows the intervention pharmacists to efficiently enter alert and intervention information 
through a web-based portal, which will be housed within the VINCI environment.  This 
information will be captured during the first 12 months of the intervention period, which 
will provide ample data without overwhelming the clinical pharmacists throughout the 
24-month study.  These measures will allow us to ascertain if the expanded dashboard 
is meeting expectations, with regards to functionality and efficiency.  These measures 
will be assessed by investigators at quarterly intervals.  If there is strong evidence to 
suggest particular components of the dashboard are not providing clinically relevant 
alerts or if the ratio of alerts to actionable alerts is exceedingly high, then the 
investigational team may decide to modify this component of the system.  Thus, these 

 
26 
 
process measures will serve to both assess the dashboard functionality and allow for 
modification to improve the efficiency of the intervention.  The time assessment data will 
allow us to determine the amount of time and effort implementing this system requires, 
as it relates to embedding it into normal workflow processes.  This information will be 
important if the dashboard is deemed effective and is disseminated for use to across the 
VA, as it will allow administrators to accurately assess the required pharmacist time 
needed for implementation.  Finally, we will assess the type and magnitude of the 
interventions made using this dashboard to ensure study fidelity.  The categorization 
and magnitude of these interventions will be assessed using the Overhage criteria, 
which is a well-established and validated method used in numerous studies, included 
several published by the investigators proposing this study.52 We will also assess the 
number of potential immunosuppression safety issues that occur and compare these 
between the two study arms.  To do so, we will use the dashboard to provide monthly 
measurements of the following:  percent of patients with missing laboratory 
assessments, percent of patients with alarming laboratory values without follow up 
scheduled, mean adherence to immunosuppression, based on refill timeliness and 
estimated using the PDC, percent of patients with a significant drug interaction without a 
immunosuppressant level and percent of patients with hospital discharge or ED visit 
without follow up scheduled.  These will be measured in all patients and compared 
between the intervention and control groups at monthly intervals.   
 
As an exploratory assessment, we will measure and compare graft survival and patient 
mortality between intervention and control sites.  This study is not powered to determine 
if the intervention improves these clinical outcomes.  However, it is well established that 
hospitalizations are a substantial risk factor for graft loss and patient death.53  Thus, it is 
possible that this intervention may produce differences with regards to these clinical 
outcomes, even if the magnitude of the estimates does not produce statistically 
significant differences.  We will also assess graft loss and mortality to ensure patient 
safety and that the intervention is not causing harm.  These data elements (death and 
graft loss) will be captured through both the VA system (VINCI) and linking to the 
transplant registry (UNOS).  The investigational team has experience with linking VA 
data to SRTR registry databases feels confident that this is feasible and will produce 
accurate measures of these important clinical outcomes.   
 
9.4 Data Analyses 
The data analysis will incorporate the intent-to-treat principle; namely, all participants 
enrolled in the intervention group will be included in the analysis and compared to all of 
the patients captured within the control group.  If a patient dies, has graft loss or moves 
out of their study site, their data will be censored at that time point.  For comparative 
statistical assessments within the utilization outcomes, the two groups will be compared 
using a generalized linear mixed models (GLMM) approach.54  This approach allows for 
measurement of participants at different time points, clustering by study site, missing 
data under the assumption of missing at random (MAR), time varying or invariant 
covariates, and can also account for the effect of correlated longitudinal measurements 
within participants. In addition, GLMM accommodates a wide range of distributional 
assumptions such as dichotomous (e.g. binomial), count (e.g. Poisson), continuous 

 
27 
 
(e.g. Gaussian), and categorical or ordinal outcomes.  For comparing the primary 
outcome (ED visit and hospitalization rates), we will use GLMM (with appropriate link 
function) with counts of these events as the outcome variable and intervention group 
(yes/no) as the primary independent variable.  This analysis is equivalent to Poisson 
regression or negative binomial regression (in the case of over-dispersion), both special 
cases of GLMM.   Additional adjustment covariables will be added to the model in a 
second set of analyses.  Covariates will include patient sociodemographics:  age, sex, 
race, comorbidities (using modified Elixhauser definitions), marital status and education.  
In these models, donor information (age, sex, race, deceased or living) and transplant 
characteristics (HLA mismatches, ischemic times, panel reactive antibody levels, time 
since transplant) will also be included.  This will assist in making valid intervention effect 
estimates after accounting for baseline differences, as there may be patient-level 
differences between groups. The magnitude of between intervention differences on 
outcome variables (effect sizes) at each time point will be estimated using appropriate 
contrasts in the corresponding GLMM models. 
 
We anticipate that some of the count outcomes will exhibit excess zeros (e.g. 
hospitalizations). If the counts exhibit excess zeros, we will use zero-inflated (ZI) 
versions of the proposed model (zero inflated Poisson and zero-inflated negative 
binomial [NB]) to account for the excess zeros with random effects to account for 
clustering by site.55 Typically, Poisson regression is used to model count data where 
observations are assumed to be independent and the number of cases has variance 
equal to the mean for each level of the covariates. However, in practice, either the 
independence or equal mean and variance assumption is often violated, which can lead 
to over-dispersion (when the variance is greater than the conditional mean). Thus, we 
consider a NB model that handles the problem of over-dispersion and that does not 
assume an equal mean and variance assumption. In certain cases, over-dispersion may 
not be sufficiently modeled via the extra parameter in NB. In this case we will consider 
including random effects into the NB model. Zero inflated models such as ZINB can be 
used for modeling the excess zeros. The ZINB model is a mixture of NB model for the 
count part and a logit model for the excess zeros. The parameters in the ZINB model 
have conditional or latent class interpretations, which correspond to a susceptible 
subpopulation at risk for the condition (in our case ED visits/hospitalizations) with counts 
generated from a NB distribution and a non-susceptible subpopulation that provides the 
extra or excess zeros. Thus, the ZINB model parameters are not well suited for 
quantifying the effect of an explanatory variable in the overall mixture population. We 
will use a marginalized ZINB model to estimate the population mean count directly, 
allowing straightforward inference for overall covariate effects.56  We will use AIC and 
BIC, which deal with the trade-off between the goodness of fit and complexity of the 
models, to choose the best fitting model among the different models; further 
assessment of the goodness of fit for the final model is made via the Pearson goodness 
of ﬁt statistic. A model with a smaller value of AIC, BIC, and a Pearson statistic close to 
one is considered a better fit of the data. We will use SAS 9.4, housed within the VINCI 
environment, to manage the data and fit all the models. 
 
For the cost analysis, we will also utilize multivariable modelling and propensity score 

 
28 
 
calibration (PSC).57  We will assess the effect of the intervention on different sources of 
cost which include inpatient, outpatient and pharmacy in addition to the total aggregated 
cost. The cost models will be estimated using log-normal or gamma models (special 
cases of GLMM) to examine the association of the intervention with cost, adjusting for 
the aforementioned patient sociodemographics, donor information and transplant 
characteristics. Cost will be obtained from VINCI CDW and HERC as inpatient, 
outpatient, pharmacy and total aggregate for two years prior to the study period and for 
the two years of the intervention. We will also account for non-VA costs through linkage 
with CMS claims data.  We will estimate different models adjusting for the clinical 
outcomes in order to examine the robustness of the results. GLMM allows for 
correlation between cost categories as well as for the likely skewed distribution of cost 
data. The Park test will be performed to check for the appropriate distribution of the cost 
data and AIC and BIC statistics for goodness of fit. Since, inpatient cost could exhibit a 
point mass of zero, we will use marginalized two-part models that are appropriate for 
semi-continuous data to model the relationship between the intervention and inpatient 
cost.58, 59  In general, statistical analyses will use two-sided tests with alpha set at 0.05. 
 
For the assessment of the functionality of the dashboard and the time required to 
complete the intervention, there will be no comparisons or modeling within this analysis; 
we will utilize standard descriptive statistics for these measurements, including mean 
(SD), median (IQR), proportion (%) and 95% confidence interval.  Variables assessed 
for this outcome will be captured through input into the REDCap database by 
intervention pharmacists during the first year of the intervention.  We will compile the 
total number of alerts the dashboard system generates at each site and assess the 
proportion of these that were deemed clinical relevant and actionable by the 
pharmacists.  We will also determine the mean/median time required to complete the 
intervention on a daily basis and estimate the time per 100 patient days.  We will 
conduct these assessments concurrently on a bimonthly basis for the first year of the 
intervention, which will allow us to modify the dashboard system’s reporting thresholds 
and improve functionality if deemed necessary by the investigational team.  We will also 
assess the overall functionality of the dashboard at the end of the first year to determine 
composite metrics.  Based on our experience with the current system, after an initial 
bolus effect, we expect the dashboard will produce two to three new alerts per 100 
patient days with one to two of these being relevant and actionable.  We will utilize 95% 
confidence intervals actual versus expected rates. 
 
We will handle missing data using several techniques including multiple imputation and 
maximum likelihood.60  Missing data mechanisms will be examined using both 
univariate and multivariate methods. We will check for missing at random (MAR) by 
creating a missing indicator for each missing variable and study the predictors of 
missingness using logistic regression. If any of the fully observed covariates or 
outcomes becomes significant in the missing data model then we will use methods for 
MAR if we do not have any reason to believe that the missing data mechanism is not at 
random. The proposed GLMM models, which are mixed effects models for the 
longitudinal data analyses, can handle data that are missing at random (MAR) and 
provide robust estimates.  For the cross-sectional data analysis plan, we will use 

 
29 
 
multiple imputation approaches and for categorical missing data we will use latent class 
based multiple imputation.57,60  
 
10 
DATA COLLECTION AND QUALITY ASSURANCE 
 
10.1 Data Collection and Management 
There will two general types of data used for this proposal, which includes operational 
data used for the dashboard system and research data to assess the impact of the 
intervention on outcomes.  Operational data for the dashboard will be captured through 
querying the national VA CDW operational database using the FRE environment and 
stored in VISN 7 space already approved and allocated for this project.  These data 
elements include diagnoses, laboratory values, medication regimens, refill histories, 
provider types and health care encounters (hospitalizations, ER visits), gathered and 
queried on a daily basis.  The pharmacists at intervention sites will access their site-
specific dashboard information stored on the server through an end-user permission 
access link that will restrict which patients are visible to ensure appropriateness of 
access to the correct PHI.   
 
For the research component of the proposal, data will be captured through the research 
CDW and housed within the VINCI environment.  We will link the VA data to the SRTR 
registry and CMS claims data.  The CDW will provide data to assess outcomes 
including hospitalizations, ER visits and costs, as well as mortality.  CDW data will also 
be used to assess interventions by querying pharmacists’ progress notes.  The CMS 
data will provide non-VA health care utilization, including hospitalizations and ER visits, 
as well as non-VA cost estimates.  The SRTR data will provide all baseline donor, 
recipient and transplant characteristics as well as clinical outcomes, including acute 
allograft rejection, graft loss and death.  We will also capture pharmacist entered data, 
which will be their assessments of the dashboard system.  We will deploy a web-based 
data entry system that the intervention pharmacists will use during the first year of the 
intervention to assess the dashboard functionality.  Queries answered by intervention 
pharmacists will include the number of alerts received, how many were considered 
clinically relevant/actionable, time to conduct the intervention and general intervention 
types.  This data will be captured through the use of the REDCap system.  
 
After data is cleaned and completely ready for analysis, we will remove identifiers from 
the final datasets.  This will minimize the risk of loss of confidentiality.  All data will be 
stored on the VINCI system and only approved study personnel will have access to this 
data. 
 
10.2 Quality Assurance  
 

 
30 
 
10.3.1  Training 
Pharmacists randomized to the intervention will participate in a two-day intensive 
training conference held at the coordinating site (Charleston, SC).  During this training, 
the dashboards functionality will be reviewed, along with the standard operating 
procedure manual which fully guides interventions.  During this training conference, the 
intervention pharmacists will also complete competencies, using case-based questions.  
Pivotal sessions that occur during this conference will include:  review of dashboard 
specifics, running the dashboard to create alerts, assessing clinical relevance of alerts, 
alert categories, interventions to address alerts, clinical and research documentation, 
provider collaboration mechanisms, motivational interviewing training and case-review 
sessions.  The investigational team will lead these sessions and utilize local experts to 
deliver the presentations and lead the case-based sessions.  The full training program 
will be developed after sites are randomized and the final dates of the training 
conference are agreed upon by study sites.   
 
10.3.2 Quality Control  
During the 24-month intervention period, we will have monthly teleconferences with the 
intervention pharmacists at the five sites to discuss and address issues with the 
dashboard and intervention.  As an additional safety measure and to further ensure 
intervention fidelity, we will have experts available to discuss patient issues with sites 
participating in the intervention.  These will include the PI, transplant nephrology, 
transplant hepatology and an internist.  These experts are members of the study team 
and have full knowledge of the study.  All interventions will be documented in the 
medical record using an approved note template, which has already been developed 
and is in use by pharmacists in VISN 7 and VISN 12. All interventions will be made 
under the approved scope of practice for the pharmacist or will be approved by the 
patient’s VA primary care provider or other provider as appropriate.  Thus, all 
interventions delivered as part of this study will be within usual care standards and 
practices. 
As another quality assurance step, the coordinating site will oversee and review study 
interventions through querying the REDCap database system; these will be monitored 
by the PI and investigational team to ensure clinical appropriateness.  This is 
particularly the case for interventions involving changes in immunosuppression 
regimens. If there are concerns with specific interventions made by study site 
pharmacists, these will be addressed on a case-by-case basis.  During the course of the 
study, if an intervention pharmacist has a particular issue that is time sensitive and is 
unsure how to address it, they will have the ability to call and/or page the PI or 
investigational team, which includes expertise in transplant-related care.  For non-
urgent issues, pharmacists can send secure emails to investigational team members or 
bring up the issue during conference calls.  Thus, there will be ample training and 
resources available to all intervention site pharmacists to ensure the study is conducted 
in a safe clinically appropriate manner with strong fidelity. 
. 

 
31 
 
10.3.3 Metrics 
The IOC will meet and review clinical events that are related to harmful issues, which 
may indicate harm, including hospitalizations, ED visits, graft loss and death.   The PI 
and study team will also review the appropriateness of study interventions, which 
include medication changes, discontinuations and initiations.  This includes 
immunosuppression medications.   
 
10.3.4 Protocol Deviations 
The SOP manual is provided to study site pharmacists with detailed instructions on how 
to deliver the interventions, including specific protocols to use once medication safety 
issues have been identified.  These study site interventions will be monitored by the 
study site and PI.  If any of these interventions are deemed to be inappropriate or 
deviations of the SOP manual and protocols, they will be further reviewed by the study 
site physicians and adjudicated.  Following this, if the intervention is still considered as a 
protocol deviation, the study site pharmacist will be informed and corrective clinical 
action will be taken to address the deviation.  This deviation will be documented.  If 
patterns develop in which multiple protocol deviations are occurring at a particular study 
site, the IOC will be informed.  They will review the issues and determine a corrective 
plan, which may include re-education and training, replacement of the study site 
pharmacist or discontinuation of the study at that particular site. 
 
10.3.5 Monitoring 
During the 24-month intervention period, we will have monthly teleconferences with the 
intervention pharmacists at the five sites to discuss and address issues with the 
dashboard and intervention.  As an additional safety measure and to further ensure 
intervention fidelity, we will have experts available to discuss patient issues with sites 
participating in the intervention.  These will include the PI, transplant nephrology, 
transplant hepatology and an internist.  These experts are members of the study team 
and have full knowledge of the study.  All interventions will be documented in the 
medical record using an approved note template, which has already been developed 
and is in use by pharmacists in VISN 7 and VISN 12. All interventions will be made 
under the approved scope of practice for the pharmacist or will be approved by the 
patient’s VA primary care provider or other provider as appropriate.  Thus, all 
interventions delivered as part of this study will be within usual care standards and 
practices. 
As another quality assurance step, the coordinating site will oversee and review study 
interventions through querying the REDCap database system; these will be monitored 
by the PI and investigational team to ensure clinical appropriateness.  This is 
particularly the case for interventions involving changes in immunosuppression 
regimens. If there are concerns with specific interventions made by study site 
pharmacists, these will be addressed on a case-by-case basis.  During the course of the 
study, if an intervention pharmacist has a particular issue that is time sensitive and is 
unsure how to address it, they will have the ability to call and/or page the PI or 

 
32 
 
investigational team, which includes expertise in transplant-related care.  For non-
urgent issues, pharmacists can send secure emails to investigational team members or 
bring up the issue during conference calls.  Thus, there will be ample training and 
resources available to all intervention site pharmacists to ensure the study is conducted 
in a safe clinically appropriate manner with strong fidelity. 
 
11 
PARTICIPANT RIGHTS AND CONFIDENTIALITY 
 
11.1 Institutional Review Board (IRB) Review 
The VA Central IRB and VA R&Ds boards will review and approve the funded protocol, 
ensure protection of patient privacy and safety, and monitor the study on an ongoing 
basis. Serious adverse events will be reported to the VA CIRB and VA R&D as they 
occur. Annual reports to the VA CIRB and R&D will indicate serious adverse events, 
new findings that may influence continuation of the study, and reports of the IOC. 
 
11.2 Informed Consent Forms and Procedure for Obtaining Consent 
Because the randomization will occur at the site level, the participants will be the 
pharmacists at each of the sites.  Therefore, each site pharmacist, prior to 
randomization, will provide written informed consent (see Appendix II for informed 
consent document).  The informed consent process of each pharmacist will occur over 
the phone, between the principle investigator and each pharmacist.  The informed 
consent document will be provided to each pharmacist prior to the call, to give them 
ample time to read and comprehend each component of the consent.  During the 
consent call, the PI will review the rationale of the study, the voluntary nature of the 
study for each site pharmacist, the expected role, activities and duration of activities for 
each pharmacist and that there will be no punitive actions taken should the pharmacist 
not wish to continue with the study.  We will ensure each pharmacist of this by also 
having their direct supervisor sign a letter (see Appendix III) stating this explicitly.  The 
informed consent will also state that there are no incentives or promises provided to 
each pharmacist by participating in the study, including, but not limited to: promotion, 
salary support, increase in grade or step or co-authorship on publications that arise from 
this research.  After the informed consent document is read and reviewed by each site 
pharmacist, they will be given the opportunity to ask questions about the expectations of 
the study.  Following this, each site pharmacist willing to participate in the study will 
electronically sign and date the form (using their PIV-enabled signature code) and email 
the form back to the principle investigator using the internal VA email system and 
encrypting the email. The PI will then also electronically sign and date the same 
document using their PIV-enabled signature code, finalizing the informed consent 
process for that participating pharmacist. 
We are requesting waiver of patient informed consent.  We are also requesting waiver 
of HIPAA authorization for all patients.  Per Federal guidelines, this research meets the 
requirements of minimal risk and thus waiver of informed consent may be granted 
through regulatory oversight if this research fulfills the following four criteria:63,64 

 
33 
 
 
1) the research involves no more than minimal risk to the subjects 
2) the waiver or alteration will not adversely affect the rights or welfare of the 
subjects 
3) the research could not be practicably carried out without the waiver or alteration 
4) whenever appropriate, the subjects will be provided with additional pertinent 
information after participation 
At the control group centers, there will be no study-related contact with the patients and 
thus informed consent and HIPAA authorization will not occur.  Within the intervention 
group, those patients that are not contacted by pharmacists (no medication safety 
issues identified by the dashboard system) also will not provide informed consent or 
HIPAA authorization  as no study related contact will occur with these patients.  For 
those patients that are contacted by pharmacists as a result of a dashboard alert, the 
care provided is usual care that should be provided to this patient population, that is 
prompted by the dashboard tool. The care prompted may have also been provided in 
the control group in the same situation. The contact will be by phone only and 
pharmacists are only operationalizing the dashboard as a tool to assist in their day-to-
day workflow. Given these facts, it is not practical or feasible to obtain informed consent 
in patients.  Thus, we are requesting waiver of patient informed consent. The study is 
minimal risk and meets these criteria. 
Ethical considerations for conducting cluster randomized studies have been thoroughly 
discussed and vetted within the research community.  Precedent is well-established that 
obtaining individual-level informed consent to conduct such studies is not an absolute; 
and, in fact, there are numerous examples within the biomedical literature of conducting 
such trials with a waiver of documentation of written informed consent.  Based on this 
literature and a review of the interventions and specifics regarding site-level randomized 
trials, the investigative team feels strongly that this proposal meets the four 
aforementioned criteria, is clearly a minimal risk study and is be capable of being safely 
completed with a waiver of written consent.  Specifically, the research does meet 
Federal guidelines for minimal risk, which is “research in which the probability and 
magnitude of possible harms implied by participation in the research is no greater than 
those encountered by participants in the aspects of their everyday life that relate to the 
research.”  Since the intervention is being conducted at the site-level as a clinical 
program, it qualifies under this definition.  By having strong and thorough safeguards for 
protection of information and oversight of clinical care, this research will not adversely 
affect the rights or welfare of subjects.  Because the intervention is being initiated at the 
site-level, in a parallel arm design, it could not practically be carried out without this 
waiver.  Finally, we have a detailed and in-depth plan for dissemination of the results 
and further roll out of the intervention, which will allow patients to gain information of the 
potential effectiveness of this intervention after the completion of the study.  Thus, this 
proposal meets the requirements of waiver of documentation of written informed 
consent and there is strong precedent from other cluster randomized trials that waiver is 
appropriate for this type of study design.63-66 
 

 
34 
 
There are minimal risks to patient safety during the completion of this study within the 
intervention arm and no risks with regards to study interventions within the control arm: 
no investigational medications will be used and all changes to patients’ current 
medication regimens will be made in accordance with VA standards of care. The 
pharmacists providing the intervention will be thoroughly trained on use of the 
dashboard and delivery of the intervention.  They will be guided by the use of protocols 
and a detailed standard operating procedure manual.  In addition, they will have access 
to expert level advice from the coordinating center, which includes a transplant 
hepatologist, transplant nephrologist, internal medicine physician and transplant clinical 
pharmacy specialist with over 15 years of experience caring for transplant recipients.  
The interventions made during this study will be documented within the medical record 
and will be communicated in appropriate fashion to the Veterans’ other providers. The 
pharmacists will work closely in a collaborative manner, with the Veteran’s physicians to 
facilitate the management of significant medication safety issues or concerns that are 
identified. 
 
11.3 Participant Confidentiality  
Protection of participant confidentiality is an important component of human subject’s 
protection. We will take careful precautions to maintain confidentiality for all participants, 
including both the pharmacists and patients. All research data will be electronically 
stored on the VINCI server, included data that is manually entered into the system via 
REDCap. There will be no paper data collection forms and no entry of data onto 
portable electronic devices or local storage devices. We will minimize the collection of 
PHI that which is necessary to complete the study Aims.  Paper documents pertaining 
to this study, as required by regulatory oversight, will be stored in locked file cabinets at 
the Charleston VAMC Research Suite (where Dr. Taber’s VA office is located). When 
study results are published or presented, only aggregate reports of the results will be 
used and Veterans’ identities will not be revealed. All investigators and project 
personnel will also complete a certified program of instruction in the protection of human 
subjects in research, such as the VA website tutorial, NIH website tutorial, or the 
University of Miami CITI course. These courses in the responsible conduct of research 
and the protection of human research participants will be completed on an annual basis 
in compliance with institutional, PHS, and NIH regulations. 
 
11.4 Study Discontinuation  
The study will be discontinued based on the recommendations of the IOC and/or the 
expert transplant nephrologist opinion.  These decisions will be based on bi-annual 
reports provided by the study biostatistician that will report SAE rates compared 
between the intervention and usual care groups.  Discontinuation will be recommended 
if the intervention demonstrates increased harm or potential harm to veterans at sites 
randomized to the intervention.  This decision will be made based on clinically or 
statistically significant differences that are apparent in SAE rates between intervention 
and usual care sites.  Worrisome trends will also be considered in the IOC decision. 

 
35 
 
  
12 
ETHICAL CONSIDERATIONS 
Pharmacist participants are completely voluntary for this study.  Each participating 
pharmacist will undergo informed written consent, with the process described in detail 
within section 11.2.  Participating pharmacists have all the same rights as usual 
research participants, including the right to withdraw at any time without punitive action, 
the right of confidentiality of information and the right to be informed of any updated 
information that may impact the study or their participating in the study. 
Ethical considerations for conducting cluster randomized studies have been thoroughly 
discussed and vetted within the research community.  Precedent is well-established that 
obtaining individual patient-level informed consent to conduct such studies is not an 
absolute; and, in fact, there are numerous examples within the biomedical literature of 
conducting such trials with a waiver of documentation of written informed consent.  
Based on this literature and a review of the interventions and specifics regarding site-
level randomized trials, the investigative team feels strongly that this proposal meets the 
four aforementioned criteria, is clearly a minimal risk study and is be capable of being 
safely completed with a waiver of documentation of written informed consent from each 
patient.  Specifically, the research does meet Federal guidelines for minimal risk, which 
is “research in which the probability and magnitude of possible harms implied by 
participation in the research is no greater than those encountered by participants in the 
aspects of their everyday life that relate to the research.”  Since the intervention is being 
conducted at the site-level as a clinical program, it qualifies under this definition.  By 
having strong and thorough safeguards for protection of information and oversight of 
clinical care, this research will not adversely affect the rights or welfare of subjects.  
Because the intervention is being initiated at the site-level, in a parallel arm design, it 
could not practically be carried out without waiver of documentation of written informed 
consent.  Finally, we have a detailed and in-depth plan for dissemination of the results 
and further roll out of the intervention, which will allow patients to gain information of the 
potential effectiveness of this intervention after the completion of the study.  Thus, this 
proposal meets the requirements of waiver of documentation of written informed 
consent and there is strong precedent from other cluster randomized trials that waiver is 
appropriate for this type of study design.63-66    
13 
COMMITTEES 
There are two groups formed specifically for the conduct of this study, which include the 
IOC and the pharmacists within the study sites assigned to the intervention group.  The 
IOC will consist of the PI, co-investigators, study coordinator, data manager and 
consultants on the proposal. The functions of the IOC will include: 1) providing scientific 
oversight; 2) reviewing all serious adverse events (graft loss and death events, 
compared across intervention groups) or complications related to the study; 3) 
monitoring site adherence to the intervention; 4) reviewing summary reports relating to 
compliance with research protocol requirements; and 5) providing advice on resource 
allocation. The IOC will meet bi-annually and as necessary by telephone. The 
recommendations of the IOC will be reviewed, and the PI will take appropriate 
corrective actions as needed. As part of this oversight, we will also recruit an 

 
36 
 
independent transplant nephrologist that is not affiliated with this study to provide their 
expertise and review of all study activities.  The intervention group clinical pharmacist 
group will include the five study site clinical pharmacists that were randomly assigned to 
the intervention arm of the study.  During the study, this group will have monthly 
meetings to discuss the intervention, identify and resolve study related issues and 
discuss pertinent study-related materials. 
 
14 
PUBLICATION AND DISSEMINATION OF RESEARCH FINDINGS 
We plan to present the findings of these research efforts at national meetings via 
submission of research abstracts and also plan to publish the results of this research in 
several manuscripts.  There are six planned manuscripts, which may increase or 
decrease depending on study operations and findings.  The six manuscripts include the 
following: 
1. Study design, rationale, barriers, pitfalls and solutions 
2. Technical dashboard development and validation 
3. Methodology paper on developing and validating a transplant cohort within the 
VA system 
4. Primary outcomes of the clinical trial – utilization across study arms 
5. Secondary costs outcomes of the clinical trial, including cost-benefit analysis 
6. Descriptive paper detailing the interventions made by the clinical pharmacists 
and impact on process measures, including medication adherence and safety 
events 
Authorship and author order will be discussed during conference calls.  Lead authors 
will be identified and author panels will be decided upon during these calls.  There is no 
promise or expectation of authorship simply from participating as a site pharmacist for 
this study.  Disputes with authorship and order will be resolved prior to the drafting of 
papers and will be determined through majority consensus of the panel.  We plan to 
disseminate the results of the research to the transplant and VA communities at large. 
We will work with operations partners, which include those within VISN 7 and other 
VISNs that provide care to significant numbers of organ transplant recipients, to develop 
a plan to implement the monitoring dashboard across the VA system.  We will also 
develop a technical report for dissemination to VA leadership, which will include detailed 
aspects of the dashboard system, along with training and educational requirements for 
pharmacists and other health care professionals to utilize the system to improve the 
monitoring and management of Veteran organ transplant recipients.  We will seek to 
present this information at cyber seminars and regional and national meetings and 
conferences, both within the formal context of research conferences and informally, 
through administrative and operational meetings. 
 

 
37 
 
15 
REFERENCES 
1. Veterans Health Administration. Department of Veterans Affairs Blueprint for 
Excellence. va.gov/HEALTH/docs/VHA_Blueprint_for_Excellence.pdf.  Published 
9/21/2014.  Accessed 2/10/2015. 
2. Veterans Health Administration. VA Informatics and Computing Structure (VINCI) 
personal correspondence. April 2016. 
3. Thrall S, Taber DJ. Ambulatory care coordination issues of dual use solid organ 
transplant recipients in a Veterans Affairs medical center. 45th Annual 
Southeastern Residency Conference. 2015;45(1). 
4. Matas A, Smith J, Skeans M, et al. OPTN/SRTR 2011 annual data report: Kidney. 
Am J Transplant 2012;13(s1):11-46. 
5. Organ Procurement and Transplantation Network (OPTN) and Scientific Registry 
of Transplant Recipients (SRTR). OPTN / SRTR 2010 annual data report. 
Rockville, MD. Published 2011. 
6. Scientific Registry of Transplant Recipients (SRTR) website. srtr.org. Updated 
2012. Accessed 09/06/2012. 
7. Meier-Kriesche HU, Schold JD, Srinivas TR, Kaplan B. Lack of improvement in 
renal allograft survival despite a marked decrease in acute rejection rates over 
the most recent era. Am J Transplant 2004;4(3):378-383. 
8. Meier-Kriesche HU, Schold JD, Kaplan B. Long-term renal allograft survival: Have 
we made significant progress or is it time to rethink our analytic and therapeutic 
strategies? Am J Transplant 2004;4(8):1289-1295.  
9. Parasuraman R, Abouljoud M, Jacobsen G, Reddy G, Koffron A, Venkat K. 
Increasing trend in infection-related death-censored graft failure in renal 
transplantation. Transplantation 2011;91(1):94-99. 
10. Samal L, Bates DW. To err is human: lessons from patient safety research for 
transplant care. Clin J Am Soc Nephrol 2014;9:845-7. 
11. Friedman AL, Geoghegan SR, Sowers NM, Kulkarni S, N. FR,Jr. Medication 
errors in the outpatient setting: classification and root cause analysis. Arch Surg. 
2007;142(3):278-83. 
12. Irani S, Seba P, Speich R, Boehler A. Medication errors: an important component 
of nonadherence to medication in an outpatient population of lung transplant 
recipients. Am J Transplant 2007;7(11):2561-6. 
13. Taber DJ, Pilch NA, Bratton CF, McGillicuddy JW, Chavin KD, and Baliga PK. 
Medication errors and adverse drug events in kidney transplantation: incidence, 
risk factors, and clinical outcomes. Pharmacotherapy 2012;32:1053-60. 
14. Taber DJ, Spivey JR, Tsurutis VM, et al. Clinical and economic outcomes 
associated with medication errors in kidney transplantation. Clin J Am Soc 
Nephrol. 2014;9:960-6. 

 
38 
 
15. Taber DJ, Pilch NA, McGillicuddy JW, et al. Improving the perioperative value of 
care for vulnerable kidney transplant recipients. J Am Coll Surg. 
2013;216(4):668-76. 
16. Taber DJ, Pilch NA, McGillicuddy JW, Bratton CF, Chavin KD, Baliga PK. 
Improved patient safety and outcomes with a comprehensive interdisciplinary 
improvement initiative in kidney transplant recipients. Am J Med Qual 
2013;28:103-12. 
17. Musgrave C, Pilch N, Taber D, et al. Improving transplant patient safety through 
pharmacist discharge medication reconciliation. Am J Transplant 2013;13(3):796-
801. 
18.  Thrall S, Taber DJ. A technology-enabled, pharmacist-led intervention to improve 
immunosuppression therapy monitoring in veteran organ transplant recipients. 
46th Annual Southeastern Residency Conference. 2016;46(1). 
19. Kidney Disease: Improving Global Outcomes (KDIGO) Transplant Work Group. 
KDIGO clinical practice guideline for the care of kidney transplant recipients. Am 
J Transplant. 2009;9 Suppl 3:S1-155. 
20. Machnicki G, Lentine KL, Salvalaggio PR, Burroughs TE, Brennan DC, Schnitzler 
MA. Kidney transplant Medicare payments and length of stay: Associations with 
comorbidities and organ quality. Arch Med Sci 2011;7(2):278. 
21. Schnitzler MA, Gheorghian A, Axelrod D, L'Italien G, Lentine KL. The cost 
implications of first anniversary renal function after living, standard criteria 
deceased and expanded criteria deceased donor kidney transplantation. J Med 
Ec 2012(0):1-41. 
22. Sellares J, De Freitas D, Mengel M, et al. Understanding the causes of kidney 
transplant failure: The dominant role of Antibody‐Mediated rejection and 
nonadherence. Am J Transplant 2012;12(2):388-399. 
23. Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and 
potential adverse drug events: Implications for prevention. J Am Med Assoc 
1995;274(1):29-34. 
24. Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L. Relationship 
between medication errors and adverse drug events. J Gen Int Med 
1995;10(4):199-205. 
25. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug 
events in pediatric inpatients. J Am Med Assoc 2001;285(16):2114-2120. 
26. Morimoto T, Gandhi TK, Seger AC, Hsieh TC, Bates DW. Adverse drug events 
and medication errors: detection and classification methods. Qual Saf Health 
Care. 2004;13(4):306-314. 
27. Leape LL, Cullen DJ, Clapp MD, et al. Pharmacist participation on physician 
rounds and adverse drug events in the intensive care unit. J Am Med Assoc 
1999;282(3):267-270. 

 
39 
 
28. Bond C, Raehl CL, Franke T. Clinical pharmacy services, hospital pharmacy 
staffing, and medication errors in United States hospitals. Pharmacotherapy 
2002;22(2):134-147. 
29. Folli HL, Poole RL, Benitz WE, Russo JC. Medication error prevention by clinical 
pharmacists in two children's hospitals. Pediatrics 1987;79(5):718-722. 
30. Kucukarslan SN, Peters M, Mlynarek M, Nafziger DA. Pharmacists on rounding 
teams reduce preventable adverse drug events in hospital general medicine 
units. Arch Intern Med. 2003;163(17):2014-2018. 
31. Schnipper JL, Kirwin JL, Cotugno MC, et al. Role of pharmacist counseling in 
preventing adverse drug events after hospitalization. Arch Intern Med. 
2006;166(5):565-571. 
32. Campana DC, Regazzi MB, Buggia I, Molinaro M. Clinically significant drug 
interactions with cyclosporin an update. Clin Pharmacokinet 1996;30(2):141-179. 
33. Christians U, Jacobsen W, Benet LZ, Lampen A. Mechanisms of clinically 
relevant drug interactions associated with tacrolimus. Clin Pharmacokinet 
2002;41(11):813-851. 
34. Kaplan B, Meier‐Kriesche H. Death after graft loss: An important late study 
endpoint in kidney transplantation. Am J Transplant 2002;2(10):970-974. 
35. Ojo AO. Cardiovascular complications after renal transplantation and their 
prevention. Transplantation 2006;82(5):603-611. 
36. Pilmore H, Dent H, Chang S, McDonald SP, Chadban SJ. Reduction in 
cardiovascular death after kidney transplantation. Transplantation. 
2010;89(7):851-857. 
37. Weiner DE, Carpenter MA, Levey AS, et al. Kidney function and risk of 
cardiovascular disease and mortality in kidney transplant recipients: The 
FAVORIT trial. Am J Transplant 2012;12(9):2437-2445. 
38. Su X, Zenios SA. Recipient choice can address the efficiency-equity trade-off in 
kidney transplantation: a mechanism design model. Management Science. 
2006;52(11):1647-60. 
39. Whiting JF, Woodward RS, Zavala EY, et al. Economic cost of expanded criteria 
donors in cadaveric renal transplantation: Analysis of Medicare Payments. 
Transplantation 2000;70(5):755-760. 
40. Eggers P. Comparison of treatment costs between dialysis and transplantation. 
Semin Nephrol 1992;12(3):284. 
41. Laupacis A, Keown P, Pus N, et al. A study of the quality of life and cost-utility of 
renal transplantation. Kidney Int 1996;50:235-42. 
42. Chisholm‐Burns M, Spivey C, Graff Zivin J, Lee J, Sredzinski E, Tolley E. 
Improving outcomes of renal transplant recipients with behavioral adherence 
contracts: a randomized controlled trial. Am J Transplant 2013;13(9):2364-73. 

 
40 
 
43. Chisholm MA, Mulloy LL, Jagadeesan M, DiPiro JT. Impact of clinical pharmacy 
services on renal transplant patients’ compliance with immunosuppressive 
medications. Clin Transplant 2001;15(5):330-6. 
44. Chisholm MA, Mulloy LL, Jagadeesan M, Martin BC, DiPiro JT. Effect of clinical 
pharmacy services on the blood pressure of african-american renal transplant 
patients. Ethn Dis 2002;12(3):392-7. 
45. Maldonado AQ, Weeks DL, Bitterman AN, et al. Changing transplant recipient 
education and inpatient transplant pharmacy practices: a single-center 
perspective. Am J Health Syst Pharm 2013;70:900-4. 
46.  Spencer JC.  Organizational knowledge, learning and memory:  three concepts 
in search of theory.  J Organ Change Manage 2006;9:63-78. 
47.  Bates DW, Cullen DJ, Laird N, et al.  Incidence of adverse drug events and 
potential adverse drug events.  Implications for prevention.  J Am Med Assoc 
1995;274:29-34. 
48. Hoff T, Jameson L, Hannan E, Flink E. A review of the literature examining 
linkages between organizational factors, medical errors, and patient safety. Med 
Care Res Rev. 2004;61(1):3-37. 
49. Holden RJ, Karsh B. A theoretical model of health information technology usage 
behaviour with implications for patient safety. Behaviour & Information 
Technology 2009;28(1):21-38. 
50.  Navaneethan SD, Sankarasubbaiyan S, Gross MD, Jeevanantham V, Monk RD.  
Tacrolimus-associated hypomagnesemia in renal transplant recipients.  
Transplant Proc 2006;38:1320-22. 
51.  Dobbels F, Damme‐Lombaert R V, Vanhaecke J,  Geest SD.  Growing pains: 
Non‐adherence with the immunosuppressive regimen in adolescent transplant 
recipients. Ped Transplant 2005;9:381-390. 
52. Overhage JM, Lukes A. Practical, reliable, comprehensive method for 
characterizing pharmacists' clinical activities. Am J Health Syst Pharm 
1999;56(23):2444-50. 
53. Schold JD, Elfadawy N, Buccini LD, Goldfarb DA, Flechner SM, et al.  Emergency 
department visits after kidney transplantation.  Clin J Am Soc Nephrol 
2016;11:674-83. 
54. Fitzmaurice GM, Laird NM and Ware JH. (2004). Applied Longitudinal Analysis. 
New York: John Wiley and Sons. 
55. Lambert D. Zero-inflated Poisson regression, with an application to defects in 
manufacturing. Technometrics. 1992;34(1):1-14. 
56. Long DL, Preisser JS, Herring AH, Golin CE. A marginalized zero-inflated 
Poisson regression model with overall exposure effects. Statist Med. 2014 
2014/09/14;33(29):5151-65. 

 
41 
 
57. Gebregziabher M.  DeSantis S. A latent class based multiple imputation approach 
for missing categorical data. J Stat Planning Inference 2010;140:3252-62. 
58. Voronca DC, Gebregziabher M, Durkalski-Mauldin V, Liu L, Egede LE.   
Marginalized two part models for generalized gamma family of distributions.  
arXiv preprint arXiv:1511.05629 (2015). 
59. Gebregziabher M, Zhao Y, Dismuke C, Axon N, Hunt K, Egede LE. Joint 
modeling of longitudinal multiple source cost data using multivariate generalized 
linear mixed models. Health Services and Outcomes Research Methodology 
2013;13:39-57. 
60. Little RJ, Rubin DB. Statistical Analysis with Missing Data. Second edition. New 
York: John Wiley and Sons, 2002. 
61. Grimshaw J, Thomas R, MacLennan G, et al. Effectiveness and efficiency of 
guideline dissemination and implementation strategies.  Health Tech Assess 
2004;8:1-72. 
62. McFarlane WR, Dunne E, Lukens E, et al. From research to clinical practice: 
Dissemination of New York State's family psychoeducation project. Psychiatric 
Services. 1993;44(3):265-70. 
63. McRae AD, Weijer C, Binik A, Grimshaw JM, Boruch R, et al.  When is informed 
consent required in cluster randomized trials in health care?  Trials 2011;12:202-
15. 
64. Anderson ML, Califf Rm, Sugarman J.  Ethical and regulatory issues of pragmatic 
cluster randomized trial in contemporary health systems.  Clin Trials 
2015;12:276-86. 
65.  Gallo A, Weijer C, White A, Grimshaw JM, Boruch R, et al.  What is the role and 
authority of gatekeepers in cluster randomized trial in health research?  Trials 
2012;13:116-30. 
66.  McKinney RE, Beskow LM, Ford DE, Lantos JD, McCall J, et al.  Use of altered 
informed consent in pragmatic clinical research.  Clin Trials 2015;12:494-502. 
 
 
16 
SUPPLEMENTS/APPENDICES 
I. 
Appendix I - Standard Operating Procedure Manual  
II. 
Appendix II - Informed Consent document for participating pharmacists 
III. 
Appendix III - Letter for supervisors of participating pharmacists to sign