A Health System Wide Evaluation of Mandated Use and Clinical Decision Support Tools to Improve PDMP Utilization and Patient Outcomes
Conditions: Medication Abuse
Sponsor: University of Colorado, Denver

Protocol Template 
Page 1 
CF-146, Effective 7/10/11 
 
Study Title: A Health System Wide Evaluation of Mandated Use and Clinical Decision 
Support Tools to Improve PDMP Utilization and Patient Outcomes 
NCT Number: NCT04918355 
Document Date: 3 December 2021 
 

Protocol Template 
Page 2 
CF-146, Effective 7/10/11 
 
COMIRB Protocol 
 
COLORADO MULTIPLE INSTITUTIONAL REVIEW BOARD 
CAMPUS BOX F-490     
TELEPHONE:  303-724-1055  Fax:  303-724-0990 
 
 
Protocol #: 
19-1130 
Project Title: 
A Health System Wide Evaluation of Mandated Use and Clinical Decision Support 
Tools to Improve PDMP Utilization and Patient Outcomes 
 
 
 
Principal Investigator:   Jason Hoppe, DO 
Version Date: 3 December 2021 
 
 
 
 
 
  
 
I. Hypotheses and Specific Aims:  
Prior research by this team has shown that when emergency department (ED) prescribers are less 
likely to prescribe an opioid analgesic to patients at high-risk of overdose or misuse when they 
consult the Colorado Prescription Drug Monitoring Program (PDMP). Despite extensive work to 
reduce workflow and accessibility barriers to use of the PDMP, the majority of providers still do 
not use the PDMP when making prescription decisions. Clinical decision support (CDS) that is 
embedded in the Electronic Health Record (EHR) has promise as a tool to increase PDMP use, 
advance presentation of important PDMP data to providers and improve opioid safety without 
interrupting provider workflow.   
 
This cluster-randomized study is designed to determine if: 
• 
CDS tools that deliver focused, timely clinical information to prescribers can improve use of 
the PDMP   
• 
The use of focused CDS tools can decrease high-risk opioid prescribing 
• 
The use of focused CDS tools is associated with better patient outcomes: decreasing high-risk, 
long-term, and aberrant opioid use  
 
Specifically, we will test the hypotheses that: 
1. Use of a focused CDS tool will increase the percentage of high-risk patients who are checked 
in the PDMP in the target populations, compared to those who have not implemented the 
tool. 
2. Use of a focused CDS tool will decrease the percentage of high-risk patients who receive a 
controlled medication prescription in the target populations, compared to those who have 
not implemented the tool. 
 
II. Background and Significance:  
The opioid crisis is a public health emergency that threatens the wellbeing of our communities. 
Despite an overall decrease in opioid prescribing[1], 11.4 million US citizens suffer from opioid 
misuse[2] and opioid deaths continue to rise in the US.[3] Opioid abuse and deaths represent a 
significant cost to the US economy, with an estimated $78.5 billion spent annually.[4]  
Colorado suffered from 1,010 drug-related deaths in 2017, a 9% increase from the prior year and 
the most ever.[5] Prescription opioid deaths reached 357 in 2017.[6] In the 2016-2017 National 
Survey on Drug Use and Health Colorado ranked 7th in number of individuals >12 years old 

Protocol Template 
Page 3 
CF-146, Effective 7/10/11 
 
reporting past year 
non-medical opioid use 
(5.03%), with more 
than 300,000 
Coloradans misusing 
prescription pain 
medications.[7] Many 
of those reporting use 
of illegal opioids such as 
heroin or illicit fentanyl 
report inappropriate 
use of prescription 
opioids prior to a shift 
to illegal drugs. Hence, improved prescribing of opioids is an essential intervention in addressing 
the opioid epidemic. 
Ten years ago, the Colorado PDMP was established in order to track controlled medication 
prescribing in an effort to improve patient safety and control misuse and abuse of opioids and 
other controlled medications. By consulting the PDMP, a prescriber can identify patients with 
prescription histories that indicate high-risk or inappropriate use. Originally, use of the PDMP 
required log on, transcription and searching steps on the state web portal which prescribers 
found the cumbersome and disruptive to their workflow, especially in fast-paced environments 
such as EDs. Our prior project incorporated PDMP access directly into the UCHealth Electronic 
Health Record (EHR), which resulted in incremental improvements in PDMP utilization but failed 
to achieve PDMP usage in a majority of opioid prescriptions.   
As a growing part of improving the digital environment in EHRs, CDS is focused on providing the 
right information to right person in the right format at the right time in the right channel to 
improve patient care. The overall goal is facilitate decision-making and improve delivery of best 
practices. CDS tools provide information and alternatives to clinicians to supplement but not 
replace good clinical judgment. The proposed CDS tools suggest but neither require nor prohibit 
any particular course of treatment and are not designed to gather information on compliance 
with state and federal laws governing opioid prescribing. The final decision is always at the 
providers’ discretion.  
Given the small progress made by incorporating “one click” PDMP access into the EHR, a logical 
next step is to investigate if we can improve PDMP use and patient outcomes by designing 
focused CDS tools for ED, hospital discharge and in primary care clinics. The proposed tool would 
search the PDMP automatically (saving provider time) and then present providers the PDMP 
information only when the patient is determined to be high risk. This will improve workflow by 
increasing the signal to noise ratio for PDMPs.  
Our initial studies in incorporating the PDMP into the EHR were carried out in 5 UCHealth EDs 
across the Front Range. Primary care clinics work under different time constraints and practice 
patterns, so it is possible that tools which are beneficial in the ED are not easily transferrable to 
primary care workflows. As such, a key first step in designing effective CDS tools to improve opioid 
prescribing has been to consult non-ED physicians and prescribers to determine gaps in their 
knowledge and processes, and learn at what time points additional information and decision 
support tools would be most effective in their practices. This step was described separately in 
Figure 1:  Number of Drug Poisoning Deaths in Colorado by Drug 
Type, 2001-2016* 

Protocol Template 
Page 4 
CF-146, Effective 7/10/11 
 
study 19-3063, “Prescriber preferences in development of Clinical Decision Support tools to 
improve PDMP utilization.” 
In the current form used by the state, PDMP data is available as a list of prescriptions, but is 
generally presented in UCHealth system in the form of a NarxCare report.  This report includes a 
proprietary calculated risk score that incorporates aspects of a patient’s controlled medication 
use history and has been shown to correlate with overdose risk.[8] Individual risk scores are 
provided for opioids, stimulants, and sedatives (controlled substances tracked by the CO PDMP). 
These scores will be used to drive CDS tool activations.   
 
III. Preliminary Studies/Progress Report: 
In our prior work at UCHealth EDs, we sought to reduce barriers to PDMP use and investigate the 
effectiveness of improved access to the PDMP and “mandated” consultation of the PDMP prior to 
signing an opioid e-prescription. We determined PDMP usage rates, prescription rates when the 
PDMP is used (or not), and patient outcomes. In the setting of already decreasing opioid 
prescribing, we found that one-click PMDP EHR integration is associated with an increase in the 
frequency of PDMP searches for ED patients who are prescribed an opioid. Unfortunately, still 
only a minority of ED patients receiving an opioid prescription were searched even after 
mandating use through an interruptive “best practice alert”. “Mandated” use for our studies, is a 
relatively loose term. In order to not disrupt clinical workflows, information is presented as a 
“best practice alert” (BPA) in a pop-up window but providers can act on or bypass the alert. This 
design is required in order to ensure the tool does not impede patient care or the exercise of 
clinical judgment. Additionally, use across different ED sites and providers was highly variable, 
suggesting inconsistent care and the need for improvement.   
After adjusting for site, provider and the patient’s race/ethnicity and age, the predicted 
probability that a patient who received an opioid prescription would have been checked in the 
PDMP was only 0.218 (95% CI: 0.202 – 0.234) under the most stringent “mandatory” use criteria 
in our previous study.  
Notably, there was a significant reduction in the probability that an opioid was prescribed as the 
number of opioids filled in the six 
months prior to the ED visit increased if 
the patient was searched in the PDMP 
(OR: 0.92 [95%CI:0.91-0.94] for every 
additional prescription). There was no 
such reduction for patients who were 
not searched (OR: 1.01 [95%CI:1.00-
1.01] for every additional prescription). 
This suggests that reviewing the PDMP 
did have an effect on prescribing 
behaviors for high-risk patients.  
We conclude that providing effective 
information on prior controlled 
medication use and risk assessment improves patient safety by decreasing high-risk prescribing. 
However, given poor compliance with “mandatory” consultation of the PDMP, it is clear that 
improvements are needed in information presentation to facilitate PDMP use. To address this 

Protocol Template 
Page 5 
CF-146, Effective 7/10/11 
 
issue, the current study will focus on designing, testing and implementing CDS tools to ensure 
delivery of the right PDMP information to prescribers at the right time within their workflow to 
increase compliance with the recommended best practice of PDMP review.  
IV. Research Methods 
A. Outcome Measure(s):   
Primary outcome: 
• 
Changes in the percentage of high-risk patients (patients who are flagged by the CDS 
logic) who are checked in the PDMP  
Secondary outcomes: 
• 
Changes in the percentage of high-risk patients (patients who are flagged by the CDS 
logic) who receive a controlled medication prescription 
• 
Overall utilization of the PDMP prior to prescribing for low risk/naïve patients vs. high risk 
patients 
• 
Percentage of patients who receive a controlled medication prescription who go on to 
long- term or aberrant use of controlled medication when CDS is used vs. controls  
 
B. Description of Population to be Enrolled:   
Two distinct populations will be involved in this study.   
1. Health care providers licensed to prescribe opioid analgesics in the UCHealth system. 
These individuals will use CDS tools provided in the course of normal clinical care and 
have the opportunity for qualitative feedback on tools. We will collect de-identified data 
on CDS use and prescribing practices of providers. There will be no attempts to link 
qualitative responses with measures of prescribing practices, which will be measured 
system wide as described elsewhere. Oncology, hospice/palliative care, and pediatric 
practices will be excluded 
2. Patients seeking care in the UCHealth system, who may receive an opioid prescription. 
The population will be limited to non-prisoners 12-89 years of age. Individuals with an 
active cancer diagnosis, sickle cell disease, or receiving specific end of life care are 
excluded.  Pregnant women and decisionally challenged people will be included but are 
not considered a target population. No efforts will be made to include these individuals. 
Patients with limited literacy or limited English fluency will be included but are not 
considered a target population. Juveniles seen at providers included in the specialties of 
interest will be included (ages 12-17 inclusive) due to the inclusion of their health care 
provider. On the advice of pediatric toxicology experts, a cut point of 12 years of age is a 
clinically relevant age at which misuse of opioid analgesics and other controlled 
medications becomes a notable risk. However, no efforts will be made to target or 
enhance inclusion of juveniles. Pediatrics-only specialist offices will not be included in 
rollout of the CDS tools. Outcomes measures from juveniles will be examined separately 
from adults but may be included in the overall population if statistically appropriate.  
 
C. Study Design and Research Methods   
This will be a 1) cluster-randomized (by practice, as identified by a group of providers who 
work in the same department or set of departments within UCHealth) study for 
ambulatory/outpatient providers and 2) a randomized study (by provider) for emergency 
medicine and inpatient providers, with four study arms: 
 

Protocol Template 
Page 6 
CF-146, Effective 7/10/11 
 
1. Arm 1:  No change in practice 
• 
Providers are free to access the PDMP when prescribing via the existing integrated 
PDMP button.  
• 
This control group will enable monitoring of the secular changes in rates of controlled 
medication prescription within the UCHealth system. 
• 
In order to facilitate analysis of patient outcomes, CDS logic as in group 4 will run 
“silently” in this group such that risks are identified and recorded but not seen by the 
provider. 
2. Arm 2:  “Mandatory” use of the PDMP for all opioid prescriptions.   
• 
A best practice alert (BPA) will be triggered and fire when ordering an opioid 
prescription. The BPA will not appear (i.e. it will be suppressed) if the provider already 
reviewed the PDMP within the encounter. The BPA is shown in Figure 2 below.   
• 
Although this condition is termed “mandatory” use, it is not a hard stop. The provider 
can easily close the window without consulting the PDMP and the prescription can be 
signed whether the PDMP has been consulted or not. The BPA will be built to track 
utilization of the BPA vs. bypass. Note: This has been deemed too interruptive for 
ambulatory clinics and they will not be included in this arm.   
3. Arm 3:  CDS for high risk patients based on PDMP data 
• 
Based on initial interviews with providers (see 19-3063) CDS support will be designed 
to be supportive of provider workflows via interruptive alerts (BPAs) for high-risk 
patients only. No alerts will fire for patients without high risk indicators based on data 
in the PDMP or prescription history as shown in Epic.  An example is given below. 
Exact phrasing may be modified during final testing in order to streamline appearance 
of the alert. 
• 
Again, BPA will be triggered upon entry of an opioid or benzodiazepine prescription 
and will fire only for patients determined to be at risk for controlled medication 
associated adverse events, misuse or overdose, based on logic built to identify high-
risk characteristics. Definition of these risk characteristics is based on CDC guidelines 
and include: NarxCare opioid or overall overdose risk score (PDMP based) indicating 
the patient is in the highest ~5% of the population for controlled medication risk, 
initiating opioid treatment with an extended or long acting release formulation with 
no opioid usage in the previous 100 days, or information in the PDMP or EHR 
indicating overlapping opioid use or co-prescribing of a sedative with an opioid.   
• 
One-click access to the PDMP for all patients will be maintained as in other arms. If 
the provider reviews the PDMP during the encounter before prescribing, the BPA will 
be suppressed.   
Figure 2:  Example BPA used in prior study. 

Protocol Template 
Page 7 
CF-146, Effective 7/10/11 
 
• 
As in Group 2, this is not a truly mandatory check and can be bypassed. BPA 
messaging was determined based on work with providers to deliver the most useful 
and clear information.  
• 
The text displayed in the alert will be specific to the patient. BPA logic and phrasing is 
included in Appendix A. 
 
Figure 3: Sample alert.   
 
4. Arm 4:  CDS for high risk patients based on PDMP and/or clinical characteristics   
• 
The CDS will fire based on the same characteristics as in Group 3 OR if clinical history 
characteristics recorded in the EHR indicate increased high-risk. Again, these risk 
factors have been determined based on work with providers and include risk factors 
based on prior published work[9; 10]. Factors included are: history of overdose, an 
active mental health diagnosis, a history of positive tox screen for illicit drugs, an 
active or prior diagnosis of a substance use disorder, or multiple recent ED visits 
which included administration of an opioid. 
• 
One-click access to the PDMP for all patients will be available as in other arms. If the 
EHR encounter includes records of a PDMP check, the BPA will be suppressed and not 
fire.   
• 
As in Group 2 and 3, this is not a truly mandatory check and can be bypassed.  
The exact messaging included in and format of the BPAs were determined based on the 
results of study 19-3063 and technical feasibility. For high-risk prescribing, each BPA will 
provide an alert and embedded link to action (“check the PDMP”), a reason (“your patient is 
at high risk due to XXX”) and a link to a website providing more information about the study, 
the logic used to fire the CDS and resources for prescribers such as the CDC Guidelines. No 
treatment recommendations will be made as the goal of the CDS is to encourage providers to 
make effective use of existing resources to inform their prescribing.  BPAs will be designed to 
track provider use vs. bypass.  

Protocol Template 
Page 8 
CF-146, Effective 7/10/11 
 
A series of silent alerts will be used to capture information about patient risks and provider 
PDMP usage over time. For Arms 2 and 3 as described above, an alert nearly identical to the 
study alert will fire but not be visible to the provider (2nd level in model). The silent alert will 
have the same PDMP and EHR criteria but will not filter out patients for whom the provider 
has already checked the PDMP (suppressed in provider alert). This will allow tracking of 
provider PDMP behavior change over time since we will have PDMP use due to BPA and 
PDMP by provider which suppresses BPAs. This is important to identify behavior change 
regarding PDMP use by providers.   
In addition, a second silent alert (3rd level in model) will fire for all eligible patients seen by 
study providers, using the same criteria as the PDMP + EHR alert described above. As with the 
previously described alert, PDMP use will not suppress this alert so that all patients can be 
captured. This alert will allow us to document the same risk information for all study patients 
so we can describe patient risk information across intervention groups. Much of this 
information tends to be transient and variable over time. Having a snapshot of the data at 
that moment will be valuable in the final analysis. 
 
Figure 4: Visible (top row) and silent alerts used in the study CDS 
Due to primary care not allowing interruptive BPAs, there will be two units of randomization 
corresponding to the ED/hospital discharge and the outpatient care setting, respectively. 
Prescribers working in UCHealth Emergency Departments and inpatient facilities  
• 
Each ED/inpatient prescriber will be randomized to one of the four study arms 
described above. Dentists will be excluded.  
• 
Prescribers will be stratified based on primary work site.  As a number of facilities 
within the UCHealth system share staff, “work site” is being defined based on shared 
staffing as follows: 
Emergency Department(s) 
Hospital/Inpatient facilities 
UCHospital at AMC 
UCHospital at AMC 

Protocol Template 
Page 9 
CF-146, Effective 7/10/11 
 
Broomfield Hospital; Commerce City, Green 
Valley, Littleton, Mississippi, Parker, 
Ralston freestanding EDs 
Broomfield Hospital 
Highlands Ranch Hospital 
Highlands Ranch Hospital 
Memorial Hospital North, Memorial 
Hospital Central, Pikes Peak Regional, 
Grandview Hospital; Fountain, Meadows, 
Powers, Woodmen freestanding EDs 
Memorial Hospital North, 
Memorial Hospital Central, Pikes 
Peak Regional, Grandview 
Hospital 
Greeley Hospital, Longs Peak Hospital, 
Medical Center of the Rockies, Poudre 
Valley Hospital, Harmony freestanding 
Greeley Hospital, Longs Peak 
Hospital, Medical Center of the 
Rockies, Poudre Valley Hospital 
Estes Park Hospital 
Estes Park Hospital 
Yampa Valley Medical Center 
Yampa Valley Medical Center 
• 
Prescribers will be randomized at the time of activation of the CDS using permuted 
block randomization base upon their primary work site and will remain in study for 12 
months. 
• 
CDS will only fire for discharge prescriptions (i.e. not for inpatient orders) for 
prescribers working in hospital-based facilities. 
• 
Providers who work in multiple locations will be assigned to the location where they 
do the majority of their prescribing. 
Primary care and selected specialty practices  
• 
Practices will be randomized into study arms 1, 3, or 4. Due to workflow concerns and 
the longitudinal physician/patient relationship in primary care, and based on 
feedback from primary care leadership, inclusion of an interruptive, non-specific alert 
was considered to be counterproductive in this population. 
• 
Practices will be stratified as small, medium, or large based on the number of opioid 
and benzodiazepine prescriptions written in the year prior to randomization. Small 
practices are those with a number of prescriptions in the lowest third, large practices 
in the highest third, and medium size practice all others. 
• 
Practices with less than 52 total opioid plus benzodiazepine prescriptions in the year 
prior to randomization will be excluded. With less than one prescription per week, the 
impact of the CDS on prescribing is anticipated to be minimal and excluding such low 
providers will reduce the risk of inactive prescribers biasing results. 
• 
Practices will be randomized at the time of activation of the CDS and will remain on 
study for 12 months. 
 
Practices may be added during the course of the study. Follow up for patient outcomes will 
continue for 6 months following deactivation of the alerts.   
CDS tools will be activated first in EDs, then for inpatient providers, and finally in ambulatory 
settings.  This staged roll out will allow for identification and modification of system bugs and 
unanticipated problems, as well as allow for increased socialization of the tools and greater 
buy in by providers. 

Protocol Template 
Page 10 
CF-146, Effective 7/10/11 
 
This study design is chosen for a combination of scientific and pragmatic concerns. This study 
is effectively an implementation of a novel CDS augmentation, such as would happen 
routinely in a non-research environment on a practice-by-practice basis. Therefore, to better 
mimic this real world situation, the most appropriate unit of randomization would be the 
practice/site.  However, pragmatic concerns also inform the design, specifically: 
1. Primary care: Randomizing by practice will optimize the opportunities for provider 
education and reduce the risk of cross-group contamination if a single provider or 
practice treated multiple patients with similar risk patterns but received different CDS 
alerts. It also ensures that all members of a treatment team working with a single 
patient or single encounter receive the same information about that patient’s risk 
stratification.   
2. Hospital based:  The small number of EDs and hospitals, especially large EDs, relative 
to the number of randomization groups translates to a large risk of unbalanced groups. 
To address this issue, a smaller unit of randomization is appropriate specifically for EDs 
and hospital discharges. The authorizing provider is ultimately responsible for opioid 
prescribing and as such is most appropriate as a smaller unit. As residents work under 
a number of attending physicians, this will also prevent the development of 
overreliance on automated alerts in a physician in training, in order to minimize the 
chance of risk to clinical care. 
Prior to CDS activation and at the time of activation, training materials will be shared with 
relevant sites and practices. These materials are included in Appendix #.  A link to a provider 
feedback survey in REDCap will be included in order to collect standardized feedback from 
system users. 
Activation of the alert will be monitored using automated tracking within Epic which identifies 
providers and patients associated with alert firings. A key feature of the CDS is that the PDMP 
check must be “attested” to in order to suppress the CDS and prevent re-firing of the alerts, 
which is interruptive to the provider. This attestation is a workflow change for providers and 
is essential for CDS success. It can happen prior to prescription entry, in which case the alerts 
will not activate (suppression), or after activation of the alert.  
In the event that providers are being “hit” with CDS alerts multiple times within the same 
encounter for a single patient because they miss the attestation step, an educational message 
with a link to the education website will be sent to the provider via email or EHR internal 
messaging systems to remind the provider to click the attestation button in order to prevent 
multiple firings. A weekly tally of messages sent will be retained, but no patient information 
will be retained. No provider will receive more than three educational reminders. The 
reminder will not include any patient information. This check is intended only to provide an 
additional personalized education step.  
This is important because (1) dissemination of education about new CDS across all of 
UCHealth is challenging so we will miss providers and (2) we have concerns that if providers 
find the CDS to be a nuisance (due to a simple knowledge gap of not understanding the 
attestation requirement) they will not use the CDS.   
Similarly, if a provider receives multiple alerts but does not access the PDMP in response, a 
reminder will be sent that checking the PDMP requires the same number of “clicks” as 
dismissing the alert and is considered best practice prior to prescribing. As with the contact 

Protocol Template 
Page 11 
CF-146, Effective 7/10/11 
 
described above, these reminders will not contain any patient specific information and are 
designed as a personalized education step. No provider will receive more than three contacts 
in this way. 
From initiation of educational messages until 9/20/2021, providers who receive one of the 
above educational messages will receive the message via email. Educational messages sent on 
or after 9/21/2021 will sent via internal messaging within the EHR. This change in educational 
message delivery will allow for examination of best practices with providers. 
Provider identifying information will be retained in order to determine the effectiveness of 
the educational outreach. Information retained will include alert firings (including multiple 
firings/patient) and provider actions over time. In order to better understand the 
environments in which educational intervention is most effective, provider specialty, primary 
work site, and basic demographic information (age, gender, race/ethnicity), will be obtained 
from within the EHR. Method of notification (email or internal messaging within the EHR) will 
also be tracked to compare provider behavioral changes between educational notifications. 
Provider work environment (Emergency Department, inpatient or ambulatory) will be 
determined based on study randomization lists. Data will be deidentified for analysis. Data on 
CDS and site performance will be monitored continuously during the course of the study (see 
attached file “Baseline data” for details of data monitored) via automated data extract built 
by ED Epic analytics team and monitored by study team. No identifiable data will be collected 
in order to protect confidentiality. De-identified extracts of these data may be exported from 
Epic to the UCDenver OneDrive system for CDS monitoring and analysis.  Baseline data will be 
gathered retrospectively for the 12 months prior to first deployment of the CDS for each 
randomization unit. Providers who are not in the system for the baseline will be excluded 
from analysis.  
Validation of the alerts 
In order to ensure that the alerts are functioning properly prior to activation, up to 30 
individual firings will be identified by study staff and EHR records of the patient will be 
checked to confirm that a controlled medication was prescribed and the risk criteria identified 
by the alert system are reflected in the medical record. Records of this validation will be 
maintained only by alert firing number.  Identifying information associated with the alert will 
not be maintained once the alert firing is validated. 
The addition of routine collection of opioid and overall risk scores from the Appriss PDMP 
interface is a new addition to the UCHealth electronic health record.  As such, while the 
developer can supply data on distribution of scores in their development cohort, no data are 
available on the distribution of scores in the UCHealth population.  Therefore risk scores for 
patients across the UCHealth system will be collected in order to verify the distribution of 
scores and number of concurrent controlled medication prescriptions, as reflected in the 
score. Only risk scores and the department specialty where the patient is being seen will be 
collected.  As most of these data will be from patients whose visit is not relevant to study 
procedures or study participation, no identifying information will be collected at this step and 
no attempts will be made to link any given score to a patient, encounter, or specific 
prescription. 
Collection of patient specific data and outcomes:   

Protocol Template 
Page 12 
CF-146, Effective 7/10/11 
 
Data on patient outcomes will be collected retrospectively. These data are crucial to 
determine the downstream effectiveness of the CDS tools.   
Pilot data from selected sites will be pulled by Compass prior to rollout in order to test the 
data query system. Once the CDS have been activated, data will be pulled shortly after initial 
deployment, plus up to 2-4 times per year to monitor progress and data integrity. Clinical and 
demographic data will be extracted from Epic by Compass, based on pre-defined criteria. The 
list of patients identified will be sent to CDPHE, who will provide PDMP data for all individuals 
for whom there is a record. PDMP data will provide additional information on controlled 
substance use by patients from non-UCHealth sites and may be useful to identify aberrant or 
high-risk use of controlled medications by the patient. Data will be merged with All Payer 
Claims Data for selected outcomes (ED utilization, hospitalization, death) then secondarily de-
identified by Compass acting as an “honest broker” and returned to the study team as a 
limited data set for analysis. 
D.   Description, Risks and Justification of Procedures and Data Collection Tools: 
Procedures: This 
study does not 
include any 
procedures which 
mandate a 
particular course of 
patient care or 
prescriber 
behavior. All PDMP 
information shared 
is already available 
through the state 
portal or integrated Epic PDMP button. It is designed to improve access and data presentation 
of patient-related information to facilitate safety through risk stratification. Use of the PDMP 
prior to prescribing an opioid is considered to be a CDC clinical best practice and is required 
by law in multiple states and in Colorado under certain circumstances[11]. This study is 
designed to make PDMP access easier and more efficient (increase signal to noise) and 
measure the impact through both healthcare system and patient outcomes. As such, it 
represents no greater risk to patients or providers than normal clinical care.  
Data collection:  CDS activation and provider behavior. Most healthcare system data 
collection will be automated within the EHR. This is the lowest risk and most accurate method 
of collecting information on the function of EHR tools and clinical decisions. Collection of 
feedback from providers will be collected through secure REDCap-based surveys and 
information collection tools. Providers have the option of answering all questions 
anonymously. However, there is a small risk of release of information considered private on 
clinical practices and decision-making processes. Provision of input is fully voluntary, we will 
not ask for any information on specific patients or any proprietary information from the 
practice. No information will be provided about statutory compliance to regulatory agencies 
so there is no risk to the provider around employability or reputation. 

Protocol Template 
Page 13 
CF-146, Effective 7/10/11 
 
Data collection:  Patient specific outcomes. This is the only stage of the study which will 
access PHI, however, there is no direct patient contact by the research team and the study is 
designed to improve information availability and flow, not dictate a course of treatment. As 
such, the primary risk of the study is release of private information or a breach of 
confidentiality around PHI but does not change the patient’s risk beyond that of normal 
medical care. In order to mitigate this risk, UCDenver Compass for Health will act as an 
“Honest Broker” and provide only a de-identified data set to the study team as they have on 3 
prior collaborations with our team. More information on data protections and processes are 
provided in the COMIRB application document. 
Data collection: Provider educational outreach.  Provider identifying information will be 
collected, unassociated with specific patient information. All data will be deidentified by the 
research prior to analysis. Each provider will be assigned a record number as part of the 
RedCap database. At the time of analysis, provider name and email will be deleted from the 
records and the record number will be used to identify the provider. The only risk to the 
provider from the educational outreach is the possibility of a small breach of privacy around 
prescribing patterns. All messaging and data storage will be kept within the 
UCHealth/UCDenver systems in order to maintain privacy. Records including provider names 
will be stored in RedCap. 
    
E.   Potential Scientific Problems:   
In our previous work, we noted a temporal trend toward system-wide decrease in opioid 
prescribing across the EDs examined. If this is ongoing and includes outpatient clinics, it will 
complicate analysis of the impact of the CDS tools. As such, it is important both to include a 
control group with no visible CDS and to apply the CDS across as large a range of providers as 
possible in order to collect enough data to be able to separate temporal and study related 
differences. To avoid potential data-distorting geographic differences in opioid prescribing, 
we will randomize clinics by site rather than geographic groupings as was done in our 
previous study. As we are working only in Colorado, generalizability of the study may be 
limited if there are unique conditions in this state which cannot be controlled for or 
statistically managed. 
We cannot truly mandate consultation of the PDMP at any point in patient care, therefore we 
will not be able to definitively determine if prescribing is wholly dependent on the 
information in the PDMP. However, it is important that clinical care not be disrupted in order 
to increase acceptance, and therefore impact, of the CDS.   
There is a risk that patients will be seen by multiple providers and so be represented in 
multiple arms of the study. For primary care, this is a small risk as most patients remain with 
the same clinics for primary care. This is a greater concern for ED or hospital visits where 
patients are likely to see different providers for subsequent or specialty visits. For provider 
level outcomes (Aim1) the multiple patient visits can be accounted for with random effects 
and from a provider prospective the visits can essentially be treated as unique. For patient 
follow-up outcomes, patients can be censored at the time of a subsequent visit and then 
followed from the new time point.  
At the time of this submission, design and building of the CDS tools is nearly complete and it 
appears that the required system-level process monitoring data can be collected in an 

Protocol Template 
Page 14 
CF-146, Effective 7/10/11 
 
automated or semi-automated manner, ensuring quality data collection and alleviating 
previous concerns about potential technical problems with system data collection. 
The greatest scientific risk associated with this study is incomplete data. At each stage of the 
study, it will be possible to bypass study associated BPAs. When this happens, automated 
data collection fields will remain blank and it will be impossible to determine linkages 
between the BPAs, use of the PDMP, and prescribing behavior. Epic CDS team have designed 
CDS and BPAs so bypasses and utilization can be tracked.  
 
F.   Data Analysis Plan:   
Two categories of data will be collected in conjunction with this study. Hospital system 
measures will be collected and monitored both at baseline and during conduct of the study. 
Technical or wording issues in the CDS tools which are identified by this monitoring may be 
addressed, but the alert logic and conduct of the study will not be changed. Patient level 
measures will be measured retrospectively. 
Hospital system measures will include: 
1. Description of practices to include: 
• 
# of licensed providers  
• 
# of patients 
• 
# of visits 
• 
Specialty/Service 
• 
Study group 
2. Description of prescribing to include the number of opioid and other controlled 
medication prescriptions written by: 
• 
Site/practice/prescriber 
• 
Study arm 
• 
Month 
• 
Specialty 
• 
PDMP button annotation (reviewed/unable to review) 
• 
PDMP check not recorded 
• 
To patient with existing sedative prescription  
• 
Formulation – immediate vs sustained/extended release 
3. Description of use of the PDMP and study-associated BPAs, specifically monitoring  
• 
# of times PDMP activity/website is checked by site/practice/service, monthly, 
and specialty 
• 
Frequency of use of the PDMP one-click button per visit and per opioid 
prescription 
• 
Frequency of CDS and BPA trigger overall and per opioid prescription 
• 
Frequency of prescriptions for which the BPA would have fired but the prescriber 
had already checked the PDMP (look for change over time – do the BPAs lead to 
learning & change in prescriber behavior)  
• 
Number of times CDS alert fired, total and per visit (the alert is designed to only 
fire once during any encounter, thus the per visit rate will be <1) 
• 
Reviewed (via radio button or via alert)/unable to review/blank 

Protocol Template 
Page 15 
CF-146, Effective 7/10/11 
 
• 
Resulted in signed Rx (reviewed/unable checked, signed prescription) 
• 
Use of training tools and website. Identifying information on individuals accessing 
tools will not be recorded. 
• 
Qualitative/semi-quantitative feedback from users as the CDS alerts are 
implemented 
 
Patient specific measures will include: 
o Number of controlled medication prescriptions, both within UCHealth and 
outside, based on PDMP data (including prescription details to include generic 
name, dosage, MMEs, days prescribed) 
o Use of the PDMP by the prescriber prior to each controlled medication 
prescription written 
o NarxCare risk score at the time of each prescription 
o Number of providers/practices used for controlled substance prescriptions 
o Number of pharmacies used for controlled substance prescriptions  
o Diagnosis of any substance use disorder after the index visit date  
o Accidental overdose of controlled substance after the index date (if data 
available) 
o Patient demographics and clinical history characteristics  
 
Educational outreach analysis: Alert firings for providers part of the educational outreach group 
will be analyzed for percent of non-compliant firings before outreach vs after as well as a 
comparison between the two different educational outreach efforts (email or EHR messaging 
system). Non-compliance is defined as 3 or more alerts for a single patient/encounter or >80% of 
firings closed using the comment/close option rather than using the PDMP link. Firings over time 
will be tracked as will changes in compliance before and after each outreach attempt. These data 
will be de-identified prior to reporting. Data will be examined for patterns based on demographics 
or work environment. 
 
Calculation of sample size: 
The number of providers who choose to provide input on CDS design and implementation can 
only be very roughly estimated. The UCHealth system includes approximately 18,000 individual 
prescribers, not all of whom will see the alerts. Assuming 5% of individual prescribers choose to 
provide feedback through the survey available on the website,  approximately 900 providers 
could provide feedback. No minimal response rate is required for significance as no hypotheses 
will be tested with these data. Survey data will be summarized via descriptive statistics only. 
The driving factor in total enrollment is the patient outcome phase. As the tools will be active 
over the course of approximately 20 months, total patient counts over 20 months, rather than a 
power calculation, inform our assumptions about enrollment. Of the approximately 2.3 million 
individuals in the UCHealth system between the ages of 12 – 89, approximately 290,000 received 
at least one opioid prescription in the last 20 months. Approximately 906,000 individual patients 
were seen in the UCHealth system in the same time period.  
The CDS tools used in this study will automatically query each patient record in order to obtain 
risk data, but data will only be seen by a provider in a fraction of cases.  Based on the advice of 
privacy officials and COMIRB staff during the design of this study, all individuals queried are 
included as study participants, even if a provider does not see a study associated CDS alert for all 

Protocol Template 
Page 16 
CF-146, Effective 7/10/11 
 
patients.  As such, it is estimated that 950,000 patients will be included in this study. The number 
of patients queried will be monitored during the course of the study and updated as needed. 
 
Data to be collected: 
 
Variable 
Definition 
Importance 
Prescriber/system data 
Overall controlled medication 
prescribing rate 
Total number of 
opioid/sedative/stimulant prescriptions 
divided by total number of visits at each 
site and the hospital system 
Measures the overall 
impact for each CDS 
tool 
Individual provider controlled 
medication prescribing rates 
Total number of 
opioid/sedative/stimulant prescriptions 
divided by total number of encounters for 
each provider 
Measures the behavior 
and practice variation 
of individual prescribers 
for each CDS tool 
Overall rate of PDMP use 
Total number of times PDMP is accessed 
when opioid/sedative/stimulant 
prescriptions are written 
Measures how readily 
providers obtain 
information for each 
CDS tool 
Provider specific rate of 
PDMP use 
Total number PDMP searches for each 
provider, changes over time for each 
provider 
Measures provider 
search behavior and 
variability changes for 
each CDS tool 
Overall prescription rate after 
accessing PDMP 
Number of times 
opioid/sedative/stimulant were 
prescribed after accessing the PDMP  
Evaluate the impact of 
PDMP review on the 
decision to prescribe 
controlled mediations 
Provider demographics 
Age, sex (M/F), years of employment 
(when available, data saved in EHR) 
Evaluate variability in 
provider acceptance 
based on information 
stored in the EHR 
system. 
Provider specific prescription 
rate after accessing PDMP 
Total number of 
opioid/sedative/stimulant prescriptions 
divided by the number of times the 
provider accessed the PDMP for each 
provider 
Evaluate provider 
variability in 
interpretation with and 
without decision 
support  
Patient specific outcomes 
Patient 
demographics 
Age, gender, ethnicity, insurance type 
Description of patient 
population  
Visit 
characteristics 
Date of service#, service location, provider 
specialty, zip code 
Evaluate visit for 
possible influences on 
patient and provider 
outcomes 

Protocol Template 
Page 17 
CF-146, Effective 7/10/11 
 
Opioid 
prescription 
Type, name, strength, number of pills, 
prescriber 
Evaluate the 
effectiveness of the 
CDS tools 
PDMP search 
completed 
Yes/no 
Evaluate the 
effectiveness of the 
CDS tools 
Patient 
characteristics 
Discharge diagnosis, past medical history, 
current co-morbid diagnoses, social 
history, other medications 
Evaluate patient 
variability and other 
influences on 
prescribing/risk levels 
CDS activation 
Activation of CDS alert, which alert 
Identify risk factors for 
patient as presented to 
prescriber  
Prescriptions 
from PDMP 
Type of medication (opioid, sedative, 
stimulant), date written#, date filled, # 
medication name, strength, formulation, 
# of pills prescribed, # of days supply, 
milligram morphine equivalents (MME), 
insurance type, pharmacy number 
Evaluate patient 
outcome, needed for 
complete picture of 
prescription use 
Non-UCHealth 
visits from All 
Payers Claims 
Data 
ED visits, hospitalizations 
Evaluate overall patient 
outcome, including 
unintended 
consequences of 
limitation of 
prescription pain 
medications 
Provider Feedback 
Use of training 
materials and 
website 
Usage (file access) over time 
Measure of 
effectiveness of 
educational efforts 
Survey 
responses 
See draft included with submission 
Provide ideas of aspects 
of CDS tool considered 
worthwhile by users.  
Descriptive statistics 
only, non-quantitative 
analysis. 
#All data will be deidentified prior to release of data to the research team.  
 
G.  Summarize Knowledge to be Gained:   
As PDMP use improves prescribing decisions, this study will provide immediate, direct benefit to 
study participants. Further, it will provide greater understanding of the reasons why clinicians use 
or do not use the PDMP as recommended, and explore several options for improving utilization 
and maximizing appropriate clinician use of the information provided.  Finally, it will explore 
patient focused outcomes when the PDMP is used (or not), and if patient outcomes are better 
(defined as less aberrant or long term opioid use) when PDMP CDS tools are better integrated 
into the provider’s workflow and better focused on providing high impact, risk based information 

Protocol Template 
Page 18 
CF-146, Effective 7/10/11 
 
to the provider at the appropriate time. Given the number of patients and providers covered by 
the UCHealth system, and the geographic dispersion of UCHealth facilities around the state, it is 
likely that results will be generalizable through the state and other states through the region. The 
degree to which our findings will be generalizable beyond Colorado and the mountain west 
regions is uncertain.     
 
 H. References: 
 
[1] Garcia MC, Heilig CM, Lee SH, Faul M, Guy G, Iademarco MF, Hempstead K, Raymond D, 
Gray J. Opioid Prescribing Rates in Nonmetropolitan and Metropolitan Counties Among 
Primary Care Providers Using an Electronic Health Record System - United States, 2014-
2017. MMWR Morb Mortal Wkly Rep 2019;68(2):25-30. 
[2] Elinore F. McCance-Katz MD P. The National Survey on Drug Use and Health: 2017. In: 
UDoHaH Services editor, 2019. 
[3] Hedegaard H, Minino AM, Warner M. Urban-rural Differences in Drug Overdose Death Rates, 
by Sex, Age, and Type of Drugs Involved, 2017. NCHS Data Brief 2019(345):1-8. 
[4] Schuchat A, Houry D, Guy GP, Jr. New Data on Opioid Use and Prescribing in the United 
States. Jama 2017;318(5):425-426. 
[5] Vestal C. Overdose Deaths Fall in 14 States, Vol. 2019: Pew Trust, 2018. 
[6] Ingalls J. More than 1,000 Coloradans Lost to Drug Overdoses in 2017, Vol. 2019: Colorado 
Health Institute, 2018. 
[7] Rachel N. Lipari PD, Struther L. Van Horn, M.A., Arthur Hughes, M.S. and, Matthew Williams, 
Ph.D. STATE AND SUBSTATE ESTIMATES OF NONMEDICAL USE OF PRESCRIPTION 
PAIN RELIEVERS. SAMHSA Short Report, 2017. 
[8] Huizenga JE BB, Patel VR, Raz A, Speights. NarxCheck Scores as a Predictor of Unintentional 
Over. In: I Appriss editor, 2016. 
[9] Seymour RB, Leas D, Wally MK, Hsu JR. Erratum to: Prescription reporting with immediate 
medication utilization mapping (PRIMUM): development of an alert to improve narcotic 
prescribing. BMC Med Inform Decis Mak 2016;16(1):125. 
[10] Seymour RB, Leas D, Wally MK, Hsu JR. Prescription reporting with immediate medication 
utilization mapping (PRIMUM): development of an alert to improve narcotic prescribing. 
BMC Med Inform Decis Mak 2016;16:111. 
[11] Colorado Senate Bill 18-022: Concerning clinical practice measures for safer opioid 
prescribing, 2018. 
 
 
 
 

Protocol Template 
Page 19 
CF-146, Effective 7/10/11 
 
 
ATTACHMENT A 
Provider/practice specialties included in data 
 
Anesthesiology 
Behavioral Health 
Brain and Spine 
BREAST 
Breast Surgery 
Burn Surgery 
Cardiac Intensive Care 
Cardiac Surgery  
Cardiothoracic Surgery 
Dentistry 
Dialysis 
EXEC HEALTH 
Family Medicine 
General Surgery 
Gerontology 
Gynecology 
Hand Surgery 
INTEGRATED MEDICINE 
Internal Medicine 
INTERVENTIONAL CARDIOLOGY 
Maternal and Fetal Medicine 
Medical Surgical 
Neurology 
Neurosurgery 
Obstetrics and Gynecology 
Occupational Medicine 
Ophthalmology 
Oral Surgery 
Orthopedic Surgery 
Otolaryngology 
Pain Medicine 
Physical Medicine and Rehabilitation 
Plastic Surgery 
Primary Care 
Psychiatry 
Rehabilitation 
Rheumatology 
Spine and Rehab Medicine 
Sports Medicine 
Surgical Intensive Care 
Surgical/Trauma 
Thoracic Surgery 
Transplant 
Urgent Care 
Urology 
VASCULAR 
Vascular Surgery 
Womens Health 
Workers Compensation 
Wound Care