Application of Economics & Social Psychology to Improve Opioid Prescribing Safety Trial 1: Electronic Health Record Nudges
Phase: NA
Conditions: Opioid Abuse, Unspecified
Sponsor: University of Southern California

1 
 
 
 
STUDY PROTOCOL 
Application of Economics & Social psychology to improve Opioid Prescribing Safety Trial 1 (AESOPS-T1): 
A cluster randomized trial 
  
Principal Investigator:   
  
Jason N. Doctor, Ph.D., Associate Professor, University of Southern California 
Mark Sullivan, Professor, University of Washington 
  
  
Supported by: 
  
The National Institute on Aging 
R33AG057395 
 
Version date: March 18, 2020 
 
  
 
  
  
 
 
 
 
 

2 
 
 
TABLE OF CONTENTS  
STUDY TEAM ROSTER 
6 
PARTICIPATING STUDY SITES 
7 
PRÉCIS 
8 
1. Study objectives 
10 
1.1 Primary Objective 
10 
1.2 Secondary Objectives 
10 
2. BACKGROUND AND RATIONALE 
10 
2.1 Background on Condition, Disease, or Other Primary Study Focus 
10 
2.2 Study Rationale 
11 
3. STUDY DESIGN 
12 
4. SELECTION AND ENROLLMENT OF PARTICIPANTS 
13 
4.1 Inclusion Criteria 
13 
4.2 Exclusion Criteria 
13 
4.3 Study Enrollment Procedures 
13 
5. STUDY INTERVENTIONS 
14 
5.1 Interventions, Administration, and Duration 
14 
5.2 Handling of Study Interventions 
14 
5.3 Adherence Assessment 
14 
6. STUDY PROCEDURES 
15 
6.1 Schedule of Evaluations 
15 

3 
 
 
6.2 Description of Evaluations 
16 
6.2.1 
 16 
6.2.2 
 16 
7. SAFETY ASSESSMENTS 
17 
7.1 Specification of Safety Parameters 
17 
7.2 Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
17 
7.3 Adverse Events 
17 
7.4 Reporting Procedures 
17 
7.5 Safety Monitoring 
17 
8. INTERVENTION DISCONTINUATION 
18 
9. STATISTICAL CONSIDERATIONS 
18 
9.1 General Design Issues 
18 
9.2 Sample Size and Randomization 
19 
9.2.1 
Error! Bookmark not defined. 
9.3 Interim analyses and Stopping Rules 
20 
9.4 Outcomes 
21 
9.4.1 
20 
9.4.2 
21 
9.5 Data Analyses 
21 
10. DATA COLLECTION AND QUALITY ASSURANCE 
23 
10.1 Data Collection Forms 
22 
10.2 Data Management 
22 
10.3 Quality Assurance 
22 
10.3.1 
22 

4 
 
 
10.3.2 
22 
10.3.3 
22 
10.3.4 
22 
10.3.5 
23 
11. PARTICIPANT RIGHTS AND CONFIDENTIALITY 
23 
11.1 Institutional Review Board (IRB) Review 
23 
11.2 Informed Consent Forms 
23 
11.3 Participant Confidentiality 
24 
11.4 Study Discontinuation 
24 
12. COMMITTEES 
24 
13. PUBLICATION OF RESEARCH FINDINGS 
24 
14. REFERENCES  
24 
15. SUPPLEMENTS/APPENDICES 
26 
Appendix A. Opioid Prescribing Diagnoses Exclusions and Medications for Outcome Assessments 
26 
Appendix B:  Study Entrance Survey 
27 
Appendix C: Post-study survey 
32 
Appendix D : Opioid Decision Nodes 
36 
Appendix E: Education Modules 
37 
Appendix F: Epic Build Technical Specifications 
38 
Appendix G: AESOPS Nudge trigger logic 
51 

5 
 
 
Appendix H: Informed Consent for Provider Survey 
55 
Appendix I: Death Notification Letter 
57 
Appendix J: Variable List 
58 
Appendix K: Coded Identifier List 
61 
 
 
 

6 
 
 
STUDY TEAM ROSTER  
  
Jason N. Doctor, Ph.D. (Principal Investigator), Leonard D. Schaeffer Center for Health Policy and Economics, University of 
Southern California, 3335 S. Figueroa Street, Unit A, Los Angeles, CA 90089-7273, Office: 213.821.7943, Fax: 213.740.3460, 
jdoctor@usc.edu  
 
Mark Sullivan, M.D., Ph.D. (Co-Principal Investigator) University of Washington, Department of Psychiatry & Behavioral Sciences, 
1959 NE Pacific Street, Box 356560, Room BB1644, Seattle, WA 98195-6560 
sullimar@uw.edu   
 
Daniella Meeker, Ph.D (Co-Investigator), University of Southern California, 2250 Alcazar Street Health Sciences Campus 
Los Angeles 90033, 
dmeeker@usc.edu  
 
Stephen D. Persell, MD, MPH, .(Co-Investigator), Division of General Internal Medicine and Geriatrics, Institute for Healthcare 
Studies, Feinberg School of Medicine, Northwestern University, 750 N. Lake Shore Drive, 10th Floor, Chicago, IL 60611, 
spersell@nmff.org  
 
Jeffrey A. Linder, MD, MPH, FACP (Co-Investigator), Division of General Internal Medicine and Geriatrics, Institute for Healthcare 
Studies, Feinberg School of Medicine, Northwestern University, 750 N. Lake Shore Drive, 10th Floor, Chicago, IL 60611, 
jlinder@partners.org 
 
Mark Friedberg, MD, MPP (Co-Investigator), RAND, 20 Park Plaza, Suite 920, Boston, MA  02116,  mfriedbe@rand.org  
 
Craig R. Fox, PhD (Co-Investigator), UCLA Anderson School of Management, 110 Westwood Plaza D-511, Los Angeles, CA 90095, 
craig.fox@anderson.ucla.edu  
 
Noah J. Goldstein, Ph.D. (Co-Investigator), UCLA Anderson School of Management, 110 Westwood Plaza, A-412, Los Angeles, CA 
90095, noah.goldstein@anderson.ucla.edu  
 
Tara K. Knight, Ph.D. (Project Manager) Leonard D. Schaeffer Center for Health Policy and Economics, University of Southern 
California, 3335 S. Figueroa Street, Unit A, Los Angeles, CA 90089-7273, Office: 213.821.7943, Fax: 213.740.3460, knight@usc.edu  

7 
 
 
PARTICIPATING STUDY SITES  
  
Northwestern University.  Jeffrey Linder, M.D., M.P.H., FACP and Stephen D. Persell, M.D., M.P.H.,  Division of General Internal 
Medicine and Geriatrics, Institute for Healthcare Studies, Feinberg School of Medicine, Northwestern University, 750 N. Lake Shore 
Drive, 10th Floor, Chicago, IL 60611, jlinder@nmff.org  spersell@nmff.org  
  
Palo Alto Medical Foundation Research Institute, Sutter Health. Robert J. Romanelli, PhD MPH, Associate Scientist , 
Assistant Clinical Professor,  School of Pharmacy, University of California San Francisco. 795 El Camino Real, Ames Building, Palo 
Alto, CA 94301, Office: 650.330.5865 Mobile: 415.205.9429 
Email: romanerj1@sutterhealth.org  
  
AltaMed Health Services.  Gilberto Hernandez, M.D., Ph.D., Medical Director for Innovation, 240 Camfield Ave, Los Angeles, CA 
90040, 323 720 5676, gihernandez@la.altamed.org  
  
The Children’s Clinic.  Maria Chandler, MD, MBA, Chief Medical Officer, 2790 Atlantic Avenue, Long Beach, CA, 90806, 562-933-
0446; mchandler@thechildrensclinic.org   
  
Asante Medical Group. Brandon J. Lynch, M.D. 701 Black Oak Drive, Medford, OR 97504 (541) 789-6850 ext 16850 
Brandon.lynch@asante.org  
 
 
 
 
 

8 
 
 
PRÉCIS  
Study Title 
Application of Economics & Social psychology to improve Opioid Prescribing Safety (AESOPS): A cluster randomized trial 
Objectives 
There is a lack of evidence that long-term opioid use offers benefit for noncancer pain and an abundance of evidence of harm. 
The objective of the Application of Economics & Social psychology to improve Opioid Prescribing Safety (AESOPS) trial is 
to develop and test novel behavioral nudges to encourage adherence to Oregon Pain Guidance and CDC guidelines for opioid 
prescribing for persons with noncancer pain. We have successfully developed and piloted different nudges in the R21 phase, 
then we compare these nudge interventions across 374 primary care clinics nationally in a cluster randomized trial. 
Design and Outcomes  
  
We will conduct a multi-site cluster randomized trial of 2 behavioral interventions to encourage safer opioid prescribing during 
ambulatory visits when pain is a symptom. The design is longitudinal with respect to clinician participants and "repeated cross-
sectional" with respect to patient observations because the visit observations used for analyses may come from different sets of 
patients over time. We randomized at the clinic level to avoid intra-clinic contamination of the intervention. 
  
Interventions and Duration. The intervention period will be 18-months in length for all participants, with a 6-month follow-up 
period to measure persistence of effects after interventions end.  The following intervention conditions will be compared:  
Condition 1: Education, justifications, and precommitments. Guideline education consists of receiving the CDC guidelines and 
completing a brief online educational module at the start of the study period. This will include educational clinical content 
related to the CDC guidelines, the Oregon Pain Guidance document, tapering training and other resources tapering training and 
other resources such as SAMHSA Medication-Assisted Treatment Physician Locator and the Naloxone Provider Guide. When 
prescribing an opioid, clinicians are prompted to record an explicit justification that appears in the patient electronic health 
record. Justifications are tailored to first time prescriptions, greater than first time prescriptions, but less than 91 days; and high 
dose chronic opioid therapy > 90 days of use. 
Condition 2: Guideline Education Control. Control group clinicians will receive the CDC guideline and will complete a brief 
online educational module at the start of the study period. This will include educational clinical content related to the CDC 
guidelines, the Oregon Pain Guidance document, tapering training and other resources tapering training and other resources 

9 
 
 
such as SAMHSA Medication-Assisted Treatment Physician Locator and the Naloxone Provider Guide. 
Outcome Measures: The primary outcome is clinician aggregate monthly morphine milligram equivalent (MME) dosing. 
This will be measured for two groups: (i) 50 MME and above, and (ii) below 50 MME daily dose visits. For an observation to 
qualify for an above 50 MME measure, a clinician has to have treated a patient with at least one visit where > 50 MME daily 
dose triggered high dose decision support. Otherwise, observations are placed in the < 50 MME condition. We will estimate 
daily milligram morphine equivalent for each clinician by summing the total number of daily morphine equivalents written 
within a monthly observation period divided by the number of 30 days. For example, suppose a clinician had three qualifying 
visits over three days in one month and prescribed: Patient #1 is prescribed 200mg ME/day x 30 days, Patient #2 has tapered 
from 200 mg ME/day to now 100mg ME/day x 30 days and Patient # 3 is on a tapering plan from 60 to 30 mg ME/day and is 
receiving 50 mg ME/day x 15 days.  The clinician’s outcome for that month then equals: 
   [(200 mg/day x 30 days) + 100 mg/day x 30 days) + (50 mg/day x 15 days)]  x [1 Month/30 days] 
= (300 mg x 1 + 100 mg x 1  + 50 mg x (½) ) = 425 mg MED/[Month]. 
This measures can capture the influence of all relevant CDC guideline prescribing endpoints that aim to reduce reliance on 
opioids in primary care including: 1) Use of alternatives to opioids (CDC recommendation 1), 2) use of lowest effective 
starting dose (CDC recommendation 5), 3) lower duration acute pain prescriptions (CDC recommendation 6), 4) Tapering 
(CDC recommendation 7), 5) avoidance of co-prescribing (CDC recommendation 11) and 6) referral for medication assisted 
therapy to substitute buprenorphine (CDC recommendation 12).  The measures excludes visits below > 50mg ME/day as these 
are low risk for adverse events. Our secondary outcome though captures transitions from low to high dose prescribing.  
Prescribing data on opioids will be captured using appropriate opioids listed in the Food and Drug Administration’s National 
Drug Code files for scheduled drugs. Oral buprenorphine will be excluded from the calculations as it relates to opioid use 
disorder treatment.  Morphine equivalent dose will be computed by standard means described elsewhere.(Vieweg, Carlyle 
Lipps, and Fernandez 2005) Qualifying pain visits will be captured by including all ICD10 pain diagnostic codes and 
excluding active cancer diagnoses. 
One of the secondary outcomes is the proportion of dosages that escalated to > 50 MME/day over time, which is a computable 
clinical quality measure from the electronic health record.  This is widely used in medicine to evaluate quality improvement 
and reliability and validity are generally supported.(Persell 2006) Visits are excluded from the denominator when patients have 
an active cancer or acute pain diagnosis that makes CDC guidelines not apply. Numerator visits are guideline discordant 
inappropriate actions that occurred at a denominator visit: co-prescription and transition from prior prescription above the 50 
mg/day threshold. Daily milligram morphine equivalent for each clinician will be estimated by summing the total number of 
daily morphine equivalents written within a monthly observation period divided by 30 days. Additional secondary outcomes 
include the change in opioid prescriptions from any source according to state-level prescription drug monitoring program 

10 
 
 
(PDMP) data. 
STUDY OBJECTIVES 
1.1 
Primary Objective  
Our primary hypothesis is that practices randomized to receive behavioral economic interventions will have lower opioid prescribing 
rates compared to control practices. 
1.2 
Secondary Objectives 
Secondary outcomes will examine opioid prescribing once interventions stop over 6 months of follow-up to investigate persistence of 
effects. 
2. 
BACKGROUND AND RATIONALE  
2.1 
Background on Condition, Disease, or Other Primary Study Focus 
Over the last two decades, prescription opioids have grown to become a public health crisis. Today, on average 1 in 5 persons with 
chronic noncancer pain receives an opioid prescription in the U.S.(Dowell, Haegerich, and Chou 2016) Yet, despite this record level of 
prescribing, reports of pain in America have not gone down.(Daubresse et al. 2013; Chang et al. 2014)  The greater availability of 
prescription opioids has been accompanied by an alarming rise in the negative consequences related to their use.  In 2015, there were 
over 365,000 emergency department visits for misuse and 20,101 prescription overdose deaths, more than have ever been recorded in 
U.S. history.(“Website” n.d.)  The costs of prescription opioids are staggering.  Aggregate costs for prescription opioid harms are 
estimated at over $78.5 billion (in 2013 dollars).  One-fourth of the aggregate economic burden is publicly funded (i.e., Medicaid, 
Medicare, and veterans' programs).(“[No Title]” n.d.; Florence et al. 2016) 
In 2016, the CDC issued the “CDC Guideline for Prescribing Opioids for Chronic Pain” which gives 12 patient care recommendations: 
(1) use alternatives to opioids, (2) set realistic goals for pain and function, (3) discuss opioid risks upon therapy initiation, (4) use 
immediate release agents instead of long-acting ones, (5) use lowest effective starting dose, avoid escalation above 50 milligram (mg) 
morphine equivalent (ME) daily dose and abide by a 90 mg ME/day dose prescribing limit, (6) prescribe lowest dose at lowest quantity 
needed for acute pain—usually 3 days with a maximum of 7 days in rare cases  (7) evaluate for and discuss tapering within 1 - 4 weeks 
and after 90 days, (8) prescribe naloxone and evaluate substance use history with opioid prescription, (9) review the state prescription 
drug monitoring program data (10)  conduct urine drug tests to provide information about drug use that is not reported by the patient, 
(11) avoid co-prescribing with benzodiazepines, and (12) refer patients with opioid use disorder to medication-assisted treatment in 

11 
 
 
combination with behavioral therapies.  Our objective is to increase adherence to the CDC guideline and Oregon Pain Guidance pain 
management guideline. To evaluate the effects of doing so, we will measure as a primary outcomes clinician aggregate monthly mg ME 
for patient visits with >50 mg ME daily dose. Secondary outcomes include rate of dose escalation to >50 mg ME/day. 
2.2 
Study Rationale 
Rationale for Accountable Justifications.  In the Accountable Justifications intervention, clinicians will be prompted to record an 
explicit justification for why they are prescribing an opioid that appears in the patient’s EHR. Accountable justifications incorporate 
several behavioral principles.  First, they signal an injunctive norm (a norm, often provided by an authoritative source, that strongly 
indicates how people should behave) indicating that prescribing an opioid is not recommended. This may make the clinician more 
likely to believe both that not prescribing an opioid is the best medical decision and that prescribing when it is not indicated violates 
professional standards.  Second, it incorporates social accountability.  A clinician justification become an explicit, separate part of the 
medical record, so a clinician’s decision to prescribe is subject to the review and judgment of the provider’s peers. Third, the 
justification alert implicitly designates guideline-concordant prescribing as the default action. Defaults are options that are exercised if 
the decision maker takes no special action to opt in or out of a given choice. Prior to our intervention, choosing to deviate from 
guidelines did not carry a special requirement to document a clinical rationale in the EHR.  Accountable justifications, therefore, reset 
the default action. Guideline-concordant treatment choices will not require special justification, but a clinician must now “opt-in” to 
prescribing an opioid by providing a justification for which they are accountable. 
 
Rationale for Precommitments.  Precommitment asks the decision-maker to commit to a future expectation. This intervention targets 
patients with problematic opioid use that do not meet criteria for opioid use disorder. For these patients, taper discussions may be 
perceived as difficult to initiate at the present time, but perceived as easier in the future because of present bias.(Loewenstein and 
Prelec 1992) Our hypothesis is that the choice to discuss a taper with the patient may benefit from a precommitment.(Goldstein, 
Cialdini, and Griskevicius, n.d.; Cialdini et al. 1978)  
In applied work, use of precommitment to a behavior has had notable success.  For example, people often state they want to save for 
retirement but fail to follow through with their intent.  The “Save More Tomorrow” retirement savings study elicited a precommitment 
of future raises to retirement funds; this study increased savings from 3.5 percent to 13.6 percent over the course of 40 months.(Thaler 
and Benartzi 2004)   In our own work in health, we found that physician public precommitment to judicious antibiotic use reduced 
inappropriate antibiotic prescribing by 19.7 percentage points.(Meeker et al. 2014)  
 

12 
 
 
3. 
STUDY DESIGN  
The Application of Economics & Social psychology to improve Opioid Prescribing Safety (AESOPS) trial is a two arm multi-site 
cluster randomized trial. The primary aim is to test the ability of two interventions based on behavioral economic principles to reduce 
the reliance on opioids and encourage safe and effective pain management. We will randomize practices in one of 2 conditions: 1) 
Education, justifications, and precommitments: When prescribing an opioid, clinicians are prompted to record an explicit justification 
that appears in the patient electronic health record. Justifications are tailored to first time prescriptions, greater than first time 
prescriptions, but less than 91 days; and high dose chronic opioid therapy > 90 days of use. (2) Guideline Education Control. We will 
have a sufficient sample size (374 clinics) to detect small effects on the primary outcomes. A null finding will indicate evidence of 
negligible or no effect. Northwestern includes 55 primary care clinics that contain 289 clinicians with 12,552 patients on chronic 
opioid therapy for noncancer pain—opioids greater than 3 months. Altamed Medical Group has 30 clinics with 134 clinicians with 
17,674 such patients.  The primary outcome is milligram morphine equivalents tiered to two levels: (i) above, and (ii) below 50 mg 
ME daily dose visits. For an observation to qualify for an above 50 mg ME measure, a clinician has to have treated a patient with at 
least one visit where > 50 mg ME daily dose triggered high dose decision support. Otherwise, observations are placed in the < 50 mg 
ME condition. Secondary, outcomes are benzodiazepine co-prescribing and rate of dose escalation to > 50 mg ME/day. We will 
estimate daily milligram morphine equivalent for each clinician by summing the total number of daily morphine equivalents written 
within a monthly observation period divided by the number of 30 days. Data from electronic medical records for participating 
practices are transferred to the Data Coordinating Center on a weekly basis.   
4. 
SELECTION AND ENROLLMENT OF PARTICIPANTS  
4.1 
Inclusion Criteria  
The subjects involved in this trial are clinicians who will be recruited from multiple clinical sites in Illinois and California.  The target 
group of physicians (and the patients that they treat) is fully inclusive and representative.  Clinicians will be eligible if they treat adult 
patients. We will request a waiver of consent for physician participation, but will enroll for any survey assessments. 
 
Each study clinic is required to have an electronic health record (EHR) system in place and have its own physical building (as opposed 
to multiple clinics sharing the same space, such as the floor of a hospital, where interactions between providers assigned to different 
intervention groups would be more likely).  Clinicians must meet the following inclusion criteria to participate in this study:  1) treat 
adult patients and 2) practice at one of the study clinics. 
 
An office visit is eligible for inclusion in the outcome denominator if: 1) the patient was 18 years old or older, 2) the provider and 
practice site were enrolled in the study, and 3) the visit occurred during the 18-month intervention period. 

13 
 
 
4.2 
Exclusion Criteria  
Visits will be excluded from the primary analysis when they have active cancer. Cancer exclusions (ICD-10 codes) are listed here. 
4.3 
Study Enrollment Procedures  
We will seek a waiver of consent for intervention, but will enroll for survey assessments. All clinicians with adult patients in 
participating practices will be contacted by email and in-person meetings. Enrollment and consent will be conducted using an online 
survey administration application.  
The email includes a description of the broad goals of the study, a general description of the intervention, compensation providers 
would receive for participation, and a link to the electronic consent form and baseline survey. 
The baseline survey includes an educational module. After providing consent, providers are asked to complete a 15 to 20 minute 
online survey and educational module. The educational module contains information about pain management (Oregon Pain Guidance) 
and safe opioid prescribing for chronic pain (CDC guidelines). The educational module also describes the interventions to which a 
clinician’s site was assigned, including changes they would observe in their electronic health record. 
5. 
STUDY INTERVENTIONS  
5.1 
Interventions, Administration, and Duration  
The intervention period will be 18-months in length for all participants, with a 6-month follow-up period to measure persistence of 
effects after interventions end. The pre-intervention baseline period will be 6 months in length.   
5.2 
Handling of Study Interventions  
The two conditions to be compared will be referred to as Nudges (Condition 1) and Control (Condition 2). Both groups will receive 
guideline education. Guideline education consists of receiving the CDC guidelines and completing a brief online educational module 
at the start of the study period. This will include educational clinical content related to the CDC guidelines, the Oregon Pain Guidance 
document, tapering training and other resources tapering training and other resources such as SAMHSA Medication-Assisted 
Treatment Physician Locator and the Naloxone Provider Guide. 
Physicians in clinics randomized to the Nudges Condition will be prompted with an EHR nudge if the prescribing history for the 
patient falls into one of the following three categories: opioid naïve or no recent opioid use, opioid prescription within 90 days, or 
chronic, high-dose opioids. The criteria for an opioid naïve patient include no recent opioid use or an order for an included opioid and 

14 
 
 
do not have a start date less than 91 days. The criteria for an opioid prescription within 90 days include: (1) an order for an included 
opioid with a start date greater than 1 day and less than 91 days and (2) not an opioid prescription with a start date greater than 90 
days. The criteria for chronic, high dose opioids are an order for an included opioid that’s dose is greater than 50 MME, two or more 
opioid prescriptions with two different start date both greater than 1 day and less than 91 days, and an opioid prescription with a start 
date greater than 90 days.  
The nudges included in Condition 1 are Accountable Justification (AJ) alerts, Precommitments (PR) to tapering and the PainTracker 
tool. Accountable Justification alerts pertain to tapering, high dose prescribing, co-prescribing and short duration prescribing for acute 
pain diagnoses. AJ is an EHR-based intervention that will prompt the clinician to justify, in a free text response, the decision to 
prescribe an opioid or to not reduce high risk/high dose prescribing. The prompt is designed to inform the clinician that others will see 
the justification in the patient’s medical record as a “High risk prescribing justification” note, and that if no justification is entered, the 
phrase “no justification given” will appear in the medical record. The behavioral economic principles underlying accountable 
justification include injunctive norms, social accountability and defaults. These AJ nudges have proven effective in a previous trial set 
in different prescribing contexts (Madrian and Shea 2001).  
Precommitments (PR) is an EHR-based intervention tailored to high dose opioid prescribing that encourages the clinician to commit to 
a taper discussion with the patient. Precommitments to taper discussions with patients may prompt physicians to initiate these 
conversations that may seem easier to tackle in the future due to present bias. PainTracker is a pain assessment tool aimed at 
broadening the understanding of pain and reframing the patient visit around reaching functional goals. It consists of the patient’s self-
reported pain and prescription opioid dosage over time. AJ, PR and PainTracker are developed within the EPIC electronic health 
record system.  
5.3 
Adherence Assessment  
In order to ensure that the study interventions are being reliably delivered we will create testing scripts that cover logical and coding 
variation in EHR-based interventions. Study staff will conduct site visits regularly during the intervention to ensure that tests do not 
fail.  
 

15 
 
 
6. 
STUDY PROCEDURES   
6.1 
Schedule of Evaluations 
 
 
Assessment 
Screening: 
Baseline 
prescribing 
(Month -17 
to Month 0) 
Baseline, 
Enrollment,  
Randomizatio
n: (Day 1) 
Intervention 
start (Month 
1) 
Continuously 
Measured or 
monitored 
Intervention 
end: (Month 
18) 
Follow-up 
period: 
(Month 19 to 
Month 30) 
Clinician-level 
Assessments 
 
 
 
 
 
 
Informed Consent Form   
 
X 
 
 
 
 
Demographics 
 
X 
 
 
 
 
Inclusion/Exclusion 
Criteria  
X 
X 
 
 
 
 
Provider Attitudes Survey 
 
X 
 
 
X 
 
Visit-level assessments 
 
 
 
 
 
 
ICD-10 codes 
X 
X 
X 
X 
X 
X 
Ordering Data 
X 
X 
X 
X 
X 
X 
Adverse Events 
 
 
X 
X 
X 
 
 
 

16 
 
 
6.2 
Description of Evaluations  
6.2.1 Screening Evaluation 
Consenting Procedure 
Consenting will only apply for survey assessments. The main study will seek a waiver of consent. With the assistance of each 
site’s medical director, we will send providers at participating sites an introductory email that includes a description of the 
broad goals of the study, a general description of the intervention, and a link to the electronic consent form and baseline 
survey.  The consent document will indicate that participation is voluntary and that decisions to participate (or not) will have 
no bearing on any provider’s status at his or her clinic. Providers who provide consent to participate will be asked to complete 
an online survey and will be reimbursed for their time. We will send up to 6 follow up emails to providers who do not respond, 
and study personnel will contact them in person when feasible.   
6.2.2 Enrollment, Baseline, and/or Randomization 
Enrollment 
We have a waiver of consent for the trial.  The intervention will be turned on for all clinics/clinicians randomized to receive 
the intervention. Start date of the trial will be recorded for each clinic. 
 
Baseline Assessments 
 
● Baseline prescribing monthly morphine equivalents 
● Baseline survey to assess provider characteristics and provider attitudes toward practice guidelines, clinical decision support, 
electronic health records, and practice environment. 
 
Randomization 
We will implement a cluster-randomized design at the clinic level to avoid contamination that might occur if individual 
providers in close proximity are randomized to different interventions. Providers who practice at multiple clinics will be 
assigned to the intervention of the clinic for which they spend at least 85% of their time.  Geographically distinct individual 
clinics will be the unit of randomization. We will conduct a block randomization of clinics by clinic organization.15   

17 
 
 
7. 
SAFETY ASSESSMENTS  
Data for patients who were noted to have been abruptly stopped of opioid prescriptions, emergency room eligible study visit with a 
diagnosis that could represent a serious complication of untreated pain will be extracted from study site EHRs and reported to the Data 
Safety and Monitoring Board. Relative rate of ED visits between study conditions will also be evaluated. 
7.1 
Specification of Safety Parameters 
Data elements from qualifying visits for providers enrolled in the study will be collected from the electronic health record.  Clinics 
will incorporate exclusions (ICD 10 cancer suppressor codes found here) used in the decision to trigger the clinical decision support.  
Aggregate counts of total visits across sites for which the intervention was triggered, for high dose opioid patients, if abrupt changes to 
dose (> 20% morphine equivalent daily dose) were made.  Such cases will be examined closely to determine if unsafe drops in opioids 
occurred. 
7.2 
Methods and Timing for Assessing, Recording, and Analyzing Safety Parameters 
Annual reports of our safety measures will be delivered to our Data Safety Monitoring Board. 
7.3 
Adverse Events  
Per CDC guideline clarification, adverse events are defined as an abrupt discontinuation of opioids for persons whose most recent 
prescription exceeds > 49 morphine equivalent daily dose; or as reported to study staff.(Dowell, Haegerich, and Chou 2019) 
Emergency department visits will also be evaluated as well as increases in prescribing > 20% presumably in response to reports of 
worsening pain. 
7.4 
 Reporting Procedures 
The Principal Investigator will report any unanticipated events to the IRB as well as the Data Safety and Monitoring Board (DSMB) 
assembled for this study. When notified of an unanticipated event, the DSMB will convene and make a decision as to whether the 
study should continue. The IRB will also be notified of the DSMB's decision. 
7.5 
Safety Monitoring  
A Data Safety and Monitoring Board (DSMB) will be established by NIH.  The following individuals were recommended to NIH with 
expertise in either opioids, overprescribing or biostatistics/research methods: Joe Frank, Jeanmarie Perrone, and Carl Peiper. 

18 
 
 
8. 
INTERVENTION DISCONTINUATION  
Following each DSMB meeting, the board will make recommendations to the local IRBs as to whether the study should continue or if 
changes to the protocol are necessary for continuation. 
9. 
STATISTICAL CONSIDERATIONS  
9.1 
General Design Issues  
Hypotheses 
Our primary hypothesis is that practices randomized to receive behavioral economic interventions will have lower opioid 
prescribing rates compared to control practices. This hypothesis will be evaluated in an intent-to-treat difference-in-differences 
framework using a mixed-effects regression model on clinician morphine equivalent daily dose. Fixed effects will include the 
effects of interventions over time (i.e., interactions between randomization assignment and time), using a 6-months prior to the 
intervention baseline period. Providers and randomization unit (clinic) will be modeled as random effects. 
Design 
The Application of Economics & Social psychology to improve Opioid Prescribing Safety (AESOPS) trial is a two arm multi-site 
cluster randomized trial. The primary aim is to test the ability of three interventions based on behavioral economic principles to 
reduce the reliance on opioids and encourage safe and effective pain management. We will randomize practices in one of two 
conditions: 1) Education, justifications, and precommitments: When prescribing an opioid, clinicians are prompted to record an 
explicit justification that appears in the patient electronic health record. Justifications are tailored to first time prescriptions, greater 
than first time prescriptions, but less than 91 days; and high dose chronic opioid therapy > 90 days of use. (2) Guideline education 
control. We will have a sufficient sample size (374 clinics) to detect small effects on the primary outcomes. 
 
Outcome Measures  
The primary outcomes is clinician aggregate monthly mg morphine equivalent. This will be evaluated for two groups: (i) above, 
and (ii) below 50 mg ME daily dose visits. For an observation to qualify for an above 50 mg ME measure, a clinician has to have 
treated a patient with at least one visit where > 50 mg ME daily dose triggered high dose decision support. Otherwise, observations 
are placed in the < 50 mg ME condition. One of the secondary outcomes is the proportion of dosages that escalated to > 50 
MME/day over time. Daily milligram morphine equivalent for each clinician will be estimated by summing the total number of 
daily morphine equivalents written within a monthly observation period divided by 30 days. Additional secondary outcomes 
include the change in opioid prescriptions from any source according to state-level prescription drug monitoring program (PDMP) 
data. 

19 
 
 
 
9.2 
Sample Size and Randomization 
Our proposed trial will have a sufficient sample size (374 clinics) to detect small effects on the primary outcomes. A null finding 
will indicate evidence of negligible or no effect. Northwestern includes 55 primary care clinics that contain 289 clinicians with 
12,552 patients on chronic opioid therapy for noncancer pain—opioids greater than 3 months. Sutter includes 273 primary care 
clinics containing 1,289 clinicians with 53,401 patients with noncancer pain who receive chronic opioid therapy. Altamed Medical 
Group 
has 
30 
clinics 
with 
134 
clinicians 
with 
17,674 
such 
patients. 
 
. 
 
Assuming an average of 5 clinicians participating per clinic, clinic-level ICC of 0.055 for clinics and 0.03 for clinicians, and 
Bonferroni multiple-comparison corrections for 3 comparisons (each active treatment arm to control and with each other), we 
calculate the following number of clinics needed to achieve 0.8 and 0.85 statistical power for a one-tailed tests at the 0.05 level of 
significance for 3-, 5-, 7- and 12-percentage point reductions in the primary outcome.  Based on Weimer et al.,(Weimer et al. 2016) 
we assume baseline high dose (> 50mg ME/day) opioid use is 260 (+ 35) milligram morphine equivalents daily dose per high-dose 
patient.  With 374 clinics among our participating organizations we will have greater than an 85% chance to detect a 3% change in 
high-dose opioid use dosage. 
 
Table 1.  Number of total  number of clinics needed to detect a reduction from 260(+35) MME daily long-term opioid use  
 
MME Reduction 
Statistical Power 
3% 
5% 
7% 
12% 
0.80 
120 clinics 
45 clinics 
27 clinics 
9 clinics 
0.85 
138 clinics 
51 clinics 
39 clinics 
18 clinics 
For our secondary outcome, transition to high dose use from low dose use, assuming conservatively a baseline concordance rate of 70%, 
we can detect a 3% change with the use of only 260 of our 374 clinics at 80% statistical power. Randomization is described next in 
Section 9.2.1. 
 
9.2.1 Treatment Assignment Procedures 
Randomization of study sites 

20 
 
 
We have chosen a cluster-randomized design at the clinic level to avoid contamination that might occur if individual providers in 
close proximity are randomized to different interventions. Providers who practice at multiple clinics are assigned to the 
intervention of the clinic for which they spend at least 85% of their time.  
Geographically distinct individual clinics will be treated as the unit of randomization.  These are clinics belonging to one of four 
larger clinical organizations covering a connected geographic area in either Illinois and California. We will carry out a block 
randomization of clinics by clinic organization using the statistical computing language R.  For each clinic organization, we will 
construct ordered collections of clinics.  We then will employ the sample function in R to return a random permutation of each 
ordered collection.  For each collection of clinic organizations we will draw a sample that represents the largest number of clinics 
within each clinic organization that was divisible by 2, the number of study arms.  We then will use the list function, a function 
that ties together related data that do not share the same structure, to assign each randomly permuted clinic to a study arm, 
repeating this process until clinics have filled the two arms of the study in equal measure.  Because the number of clinics at each 
organization is not always divisible by 2, we will treat “remainder” clinics across all organizations differently. These remainder 
clinics will be randomized to conditions separately. This will be accomplished in a procedure similar to the one described above.  
Allocation of the sequence will be concealed until after the interventions were assigned.  
9.3 
Interim analyses and Stopping Rules 
No interim analysis will be conducted on primary or secondary outcomes.  The Data Safety and Monitoring Board is granted the 
power to recommend discontinuation of the study to each study IRB, if safety concerns are found. The board will meet biannually 
throughout the duration of the study to review patient safety and adverse events. Following each meeting, the board will make 
recommendations to the local IRBs as to whether the study should continue or if changes to the protocol are needed. The Board 
will compare between study conditions the frequency of emergency department visits, frequency of patients receiving a 20% 
increase in opioid prescriptions and the proportion abruptly cut-off of opioids (>49 MMED to zero) between visits. We will be 
extracted from study site EHRs and reported to the Board.   
9.4 
Outcomes  
9.4.1 Primary outcome   
The primary outcome is clinician aggregate monthly morphine milligram equivalent (MME) dosing. This will be measured for 
two groups: (i) 50 MME and above, and (ii) below 50 MME daily dose visits. For an observation to qualify for an above 50 MME 
measure, a clinician has to have treated a patient with at least one visit where > 50 MME daily dose triggered high dose decision 
support. Otherwise, observations are placed in the < 50 MME condition.  
 

21 
 
 
9.4.2 Secondary outcomes   
 
One of the secondary outcomes is the proportion of dosages that escalated to > 50 MME/day over time. Daily milligram morphine 
equivalent for each clinician will be estimated by summing the total number of daily morphine equivalents written within a 
monthly observation period divided by 30 days. Additional secondary outcomes include the change in opioid prescriptions from 
any source according to state-level prescription drug monitoring program (PDMP) data. 
9.5 
Data Analyses  
We will use means and medians for continuous measures, frequencies for count data, standard deviations and interquartile ranges for 
variance to describe the sample characteristics.  For inferential analysis, our primary hypothesis is that milligram morphine equivalent 
dose will decrease for persons on high-doses of opioid therapy.  We assume a linear mixed effects hierarchical knotted spline 
regression model which offers a flexible way to accommodate non-linear trends before and after the introduction of the intervention.  
This model places a knot at the intervention start date allowing slopes before and during treatment to vary for each intervention and 
control.  For our two-group study evaluating the primary outcome, mean milligram morphine daily dose prescribed for intervention 
and control, we will place a knot at t*, the start of the intervention, and evaluate for each comparison between groups: 
 
Y = β1 + β2Time + β3(Time  - t*)+ + β4Group + β5Time x Group + β6(Time - t*)+ x Group + ζ + η + error    [1] 
 
where (z)+ is a truncated line function that equals z when z is positive and is equal to zero otherwise, ζ is the clinician random effect 
and η  is the clinic random effect.  
In addition to the knotted spline regression model, we will also conduct a linear mixed effects hierarchical difference-in-differences 
regression. This model has fewer interaction terms, and mitigates convergence and collinearity errors: 
 
Y = β1 + β2Group + β3Time + β4Time x Group + ζ + η + error    [2] 
where time is categorical, consisting of: 1) baseline (reference), 2) the intervention period and, 2) the post-intervention period.  
The secondary analysis is logistic and replaces Y in Eq.1 with ln[(1 - pi)/pi] where pi is the probability of a guideline concordant 
decision at visit i.  

22 
 
 
10. 
DATA COLLECTION AND QUALITY ASSURANCE 
10.1 
Data Collection Forms  
Two types of data will be collected – data from electronic medical and billing records and data from self-administered online surveys 
at the beginning and end of the study.  
10.2 
Data Management  
Each of the participating sites will create an extract from their Electronic Medical or Billing Records of the Data Elements. These 
records will be transferred to the coordinating center on a weekly basis.   
The CC has created programs and quality control queries for transforming all of the data into a standard model (Observational Medical 
Outcomes Partnership Common Data Model, version 5.1).  
The data collection forms will be online surveys. The electronic data system, Epic, will have native data capture formats.  
10.3 
Quality Assurance  
10.3.1 Training 
Staff will be trained on the permissible values present in Electronic Records, frequency of update, and expected volumes of 
data. 
10.3.2  Quality Control Committee  
The quality control committee will consist of practicing clinicians from each participating clinical organization. They will 
review automatically refreshing dashboards for potential deviations in coding systems and appropriate values for codes for 
inclusion in the outcome measures. These dashboards will be reviewed prior to each email distribution. 
10.3.3  Metrics 
Quality control metrics will be based on reports verifying visits were not for cancer exclusions. All drugs prescribed at these 
visits will be categorized as “opioid” or “non-opioid”.  Incorrect categorizations will be corrected and outcome computations 
recomputed before each email is delivered.    
10.3.4 Protocol Deviations 
Our task tracking system, JIRA will be used to track and document issues. Each issue will include both an assignee and a 

23 
 
 
reviewer. 
10.3.5 Monitoring 
In addition to data quality reviews, we will also review the integrity of the interventions. On an approximately quarterly basis, 
staff will visit headquarters of participating sites and verify functionality of decision support tools.  
Additionally, practicing clinicians on our study team will have the ability to monitor electronic medical record interventions in 
their own health systems.   
   
11. 
PARTICIPANT RIGHTS AND CONFIDENTIALITY  
11.1 
Institutional Review Board (IRB) Review  
The study protocol for all clinic sites will be reviewed and approved by the University of Southern California’s Institutional Review 
Board (IRB). Other sites will rely on the USC IRB through the SMART IRB online reliance system. 
11.2 
Informed Consent Forms 
We will seek a waiver of consent for the main study, but will consent for survey assessments. An electronically signed consent form 
will be obtained from each participating clinician.  The consent form will describe the purpose of the study, the procedures to be 
followed, the risks and benefits of participation, and compensation for participation.  
11.3 
Participant Confidentiality  
Data will be recorded with SSL protected web sites to a data warehouse and transferred over secure network protocol. Data will be 
kept in encrypted files on computers in locked offices at USC Schaeffer Center facilities. Only study investigators will have access to 
a list of study ID codes that will be traceable back to actual subject contact identifiers for clinicians. These codes will be kept in 
locked offices at USC Schaeffer Center facilities.  
 
11.4 
Study Discontinuation  
Following each DSMB meeting, the board will make recommendations to the local IRBs as to whether the study should continue or if 
changes to the protocol are necessary for continuation. 

24 
 
 
12. 
COMMITTEES 
Data Safety Monitoring Board: Joe Frank, Jeanmarie Perrone, Carl Peiper. 
13. 
PUBLICATION OF RESEARCH FINDINGS  
Publication of results from our research will follow the NIH Public Access Policy, which requires that we submit to the National 
Library of Medicine’s PubMed Central an electronic version of final, peer-reviewed manuscripts upon acceptance for publication, to 
be made publicly available no later than 12 months after the official date of publication.  
14. 
REFERENCES  
1. 
Grijalva C, Nuorti J, Griffin M. Antibiotic prescription rates for acute respiratory tract infections in US ambulatory settings. 
JAMA. 2009;302(7):758-766. 
2. 
Gonzales R, Malone D, Maselli J, Sande M. Excessive antibiotic use for acute respiratory infections in the United States. Clin 
Infect Dis. 2001;33:757-762. 
3. 
Shehab N, Patel P, Srinivasan A, Budnitz D. Emergency department visits for antibiotic-associated adverse events. Clin Infect 
Dis. 2008;47(735-743). 
4. 
Arias C, Murray B. Antibiotic-resistant bugs in the 21st century—a clinical super-challenge. N Engl J Med. 2009;360:439-443. 
5. 
Klevens R, Morrison M, Nadle J, al e. Invasive methicillin-resistant Staphylococcus aureus infections in the United States. 
JAMA. 2007;298:1763-1771. 
6. 
Lerner J, Tetlock P. Accounting for the effects of accountability. Psych Bull. 1999;125(2):255. 
7. 
Johnson EJ, & Goldstein, D. G. (2003). Do defaults save lives? Science, 302, 1338-1339. 
8. 
Madrian B, & Shea, D. (2001). The power of suggestion. Quarterly Journal of Economics, 18-116. 
9. 
Thaler R, Sunstein C. Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.; 2008. 
10. 
Macfarlane J, Holmes, W., Macfarlane, R., & Britten, N. (1997). Influence of patients’ expectations on antibiotic management 
of acute lower respiratory tract illness in general practice: Questionnaire study. British Medical Journal, 315, 1211-1214. 
11. 
Cialdini R, Reno R, Kallgren C. A focus theory of normative conduct: recycling the concept of norms to reduce littering in 
public places. J Pers Soc Psych. 1990;58(6):1015. 
12. 
Cialdini RB. Influence: Science and Practice.  5th Edition. Boston: Allyn & Bacon; 2010. 
13. 
Goldstein N, Cialdini R, Griskevicius V. A room with a viewpoint: Using social norms to motivate environmental conservation 
in hotels. J Consum Res. 2008;35:472-482. 
14. 
Kiefe C, Allison J, Williams O, Person S, Weaver M, Weissman N. Improving quality improvement using achievable 
benchmarks for physician feedback: a randomized controlled trial. JAMA. 2001;285(22):2871-2879. 

25 
 
 
15. 
Ihaka R GR. A language for data analysis and graphics. J Comput Graph Stat 1996;5:299-314. 
16. 
Chan KS, Fowles, J. B., & Weiner, J. P. (2010). Electronic health records and reliability and validity of quality measures: a 
review of the literature.Medical Care Research and Review. 
17. 
Kish L. Survey Sampling. New York, NY: John Wiley and Sons; 1965. 
 
 
 
 

26 
 
 
15. 
SUPPLEMENTS/APPENDICES 
APPENDIX A. OPIOID PRESCRIBING DIAGNOSES EXCLUSIONS AND MEDICATIONS FOR OUTCOMES ASSESSMENTS 
Opioids Trigger List  
 
Opioids Outcome List 
 
Diagnosis of Exclusion 
 
Non-Opioid Dosing for Alternatives 
Link  
 
Link 
 
Link 
 
Link 
 
("Outcome" Tab) 
 
("Grouper" Tab) 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

27 
 
 
APPENDIX B:  STUDY ENTRANCE SURVEY 
Online Survey 
THE ONLINE SURVEY IS INTENDED TO (1) ELICIT INFORMATION FROM PROVIDERS (2) MONITOR 
IF “EDUCATION” INFLUENCES RESPONSES TO QUESTIONS ABOUT TREATMENT PREFERENCES. 
RESPONDENTS WILL HAVE THE OPPORTUNITY TO CHANGE THEIR ‘FINAL’ ANSWERS AT ANY 
TIME IN THE SURVEY.  WE WILL RECORD ALL ANSWERS AND LOG CLICKS ON INFORMATIONAL 
LINKS PROVIDED. 
 
Basic information about your clinical background. 
1. When did you start working at [name of clinic]? (<1 year ago, 1-2 years ago, 3-5 years ago, 5-10 years ago, >10 years ago) 
2. When did you finish your clinical training as a physician (i.e., your internship, residency, or fellowship—the one you most recently 
completed)? (<2 years ago, 2-5 years ago, 5-10 years ago, 10-20 years ago, >20 years ago) 
3. What is your clinical specialty? (internal medicine, family practice, general practice, pediatrics, other) 
 
Information about the electronic health record (EHR) used at your clinic. 
4. How would you rate your overall level of satisfaction with the electronic health record (EHR) used at your clinic? 
(1= Very unsatisfied , 5=Very satisfied) 
5. Thinking about your workflow during an office visit with a patient, how often do you enter at least 1 diagnosis for the visit into the EHR while 
you are still seeing the patient? 
a. Always 
b. Usually 
c. Sometimes 
d. Rarely 
e. Never 
f. Not applicable: The EHR does not offer a way to enter a diagnosis (or diagnoses) that correspond to the visit. 

28 
 
 
Quality improvement efforts. 
6. Within the past year, have you received any feedback—positive or negative—from your clinic about the quality of care you provide to 
patients (for any kind of care)? 
a. Yes, positive feedback only 
b. Yes, both positive and negative feedback 
c. Yes, negative feedback only 
d. No, did not receive any feedback at all 
e. Unsure / Can’t Remember 
7. [If yes to previous] Based on the feedback you received, did you make any changes to the way you deliver medical care? 
a. Yes, made 1 or more changes 
b. No, made no changes 
c. Unsure / Can’t Remember 
 
8. In the past year, did you attend any medical educational sessions?  Note: “Medical education sessions” include sessions that yielded credit 
towards maintenance of certification (e.g., CME) and less formal sessions that did not yield such credit. 
a. Yes 
b. No 
c. Unsure / Can’t Remember 
 
9. [If yes to question 8] Based on the information you received in any of these educational sessions, did you make any changes to the way you 
deliver medical care? 
a. Yes, made 1 or more changes 
b. No, made no changes 
c. Unsure / Can’t Remember 
10. [If yes to question 8] Did any of the educational sessions you attended cover opioid prescribing safety? 

29 
 
 
a. Yes 
b. No 
c. Unsure / Can’t Remember 
11. [If yes to question 8] Did any of the educational sessions you attended cover opioid prescribing safety? 
a. Yes 
b. No 
c. Unsure / Can’t Remember 
12. Based on your general experience as a clinician, please indicate how much you agree or disagree with the following statements: 
i. Continuing education is an effective way to improve the quality of care  (1 = Strongly agree, 2 = Agree, 3 = Neither Agree nor 
Disagree, 4 = Disagree, 5 = Strongly Disagree) 
ii. Auditing physicians’ clinical performance and providing performance feedback is an effective way to improve the quality of care  (1 = 
Strongly agree, 2 = Agree, 3 = Neither Agree nor Disagree, 4 = Disagree, 5 = Strongly Disagree) 
iii. Electronic decision support tools (e.g., “pop up” reminders in your EHR) are an effective way to improve the quality of care  (1 = 
Strongly agree, 2 = Agree, 3 = Neither Agree nor Disagree, 4 = Disagree, 5 = Strongly Disagree) 
iv. Condition-specific, streamlined electronic order sets are an effective way to improve the quality of care  (1 = Strongly agree, 2 = 
Agree, 3 = Neither Agree nor Disagree, 4 = Disagree, 5 = Strongly Disagree) 
Your assessment of clinical guidelines. 
13. Please indicate your level of knowledge about the following clinical 
guidelines.  
[Know this guideline in detail / Know this 
guideline in general, but not every detail / 
Not familiar with this guideline] 
Screening 
 
Guidelines for colorectal cancer screening (USPSTF guideline: 
https://www.uspreventiveservicestaskforce.org/Page/Document/Upd
ateSummaryFinal/colorectal-cancer-screening2) 
 
Guidelines for breast cancer screening (USPSTF guideline: 
https://www.uspreventiveservicestaskforce.org/Page/Document/Upd
ateSummaryFinal/breast-cancer-screening1) 
 

30 
 
 
Guidelines for cervical cancer screening (USPSTF guideline: 
https://www.uspreventiveservicestaskforce.org/Page/Document/Upd
ateSummaryFinal/cervical-cancer-screening2) 
 
Chronic disease care 
 
Guidelines for the care of diabetes mellitus (ADA guidelines: 
https://care.diabetesjournals.org/content/diacare/suppl/2018/12/17/
42.Supplement_1.DC1/DC_42_S1_2019_UPDATED.pdf) 
 
Guidelines CDC guidelines for opioid prescribing in chronic pain: 
https://www.cdc.gov/drugoverdose/pdf/Guidelines_Factsheet-a.pdf) 
 
Acute care 
 
Guidelines for antibiotic use in non-specific upper respiratory infections 
(CDC guidelines: 
https://www.cdc.gov/getsmart/community/materials-
references/print-materials/hcp/adult-tract-infection.pdf) 
 
Guidelines for imaging in acute low back pain (ACP/APS guidelines: 
https://annals.org/aim/fullarticle/2603228/noninvasive-treatments-
acute-subacute-chronic-low-back-pain-clinical-
practice?_ga=2.171090592.1829219229.1562111963-
458883333.1562111963) 
 
 

31 
 
 
  
a. In the grid below, please estimate the AVERAGE time allocated to you and the amount of time you 
feel would be needed to provide high quality care for your patients. (please check one box) 
Visit type 
Time allocated 
Time needed 
i. 
Complete Physical/Consultation 
_______minutes 
_______minutes 
ii. 
Routine Follow-up Visits 
_______minutes 
_______minutes 
iii. Urgent Care Visits (in general) 
_______minutes 
_______minutes 
iv. Urgent Care Visits for pain 
_______minutes 
_______minutes 
 
b. Which best describes the atmosphere 
in your office? (please check one box) 
Calm, 
orderly 
 
Busy, but 
reasonabl
e 
 
Hectic, 
chaotic 
 
□1 
□2 
□3 
□4 
□5 
 
c. Please indicate how much you agree or 
disagree with the following statement. (please 
check one box) 
Strongly 
disagree Disagree 
Neither 
agree nor 
disagree 
Agree 
Strongly 
Agree 
Overall, I am satisfied with my current job 
□1 
□2 
□3 
□4 
□5 
 
 
 

32 
 
 
APPENDIX C: POST-STUDY SURVEY 
 
STUDY EXIT SURVEY  
1) How would you rate your overall level of satisfaction with the electronic health record (EHR) used at your clinic? 
(1= Very unsatisfied, 2=Unsatisfied, 3=Neither satisfied nor satisfied 4=Satisfied  5=Very satisfied) 
a) How would you rate your overall satisfaction with the alerts and clinical decision support you received relating to opioid 
prescribing?  
(1= Very unsatisfied, 2=Unsatisfied, 3=Neither satisfied nor satisfied 4=Satisfied  5=Very satisfied, 0= I didn’t receive alerts for 
opioids. 
b) How would you rate your overall level of satisfaction with the how these alerts affected visits with your patients?” 
(1= Very unsatisfied, 2=Unsatisfied, 3=Neither satisfied nor satisfied 4=Satisfied  5=Very satisfied) 
2) 
d. In the grid below, please estimate the AVERAGE time allocated to you and the amount of time you 
feel would be needed to provide high quality care for your patients. (please check one box) 
Visit type 
Time allocated 
Time needed 
v. 
Complete Physical/Consultation 
_______minutes 
_______minutes 
vi. Routine Follow-up Visits 
_______minutes 
_______minutes 
vii. Urgent Care Visits (in general) 
_______minutes 
_______minutes 
viii. Urgent Care Visits for pain 
_______minutes 
_______minutes 
 
e. Which best describes the atmosphere 
in your office? (please check one box) 
Calm, 
orderly 
 
Busy, but 
reasonable 
 
Hectic, 
chaotic 

33 
 
 
 
□1 
□2 
□3 
□4 
□5 
 
f. Please indicate how much you agree or 
disagree with the following statement. (please 
check one box) 
Strongly 
disagree 
Disagre
e 
Neither 
agree nor 
disagree 
Agree 
Strongly 
Agree 
Overall, I am satisfied with my current job 
□1 
□2 
□3 
□4 
□5 
 
4)  Please indicate how much you agree 
or disagree with the following 
statements. 
Strongly 
Disagree 
Disagree 
Neither 
Agree 
nor 
Disagree 
Agree 
Strongly 
Agree 
a. I receive useful information about the 
quality of care I deliver 
□1 
□2 
□3 
□4 
□5 
b. When I receive a new report about the 
quality of care, it just makes me feel 
helpless 
□1 
□2 
□3 
□4 
□5 
c. My practice evaluates me in a way that is 
fair 
□1 
□2 
□3 
□4 
□5 
 
i. 
 
 
13) Are you more frequently engaging patients in opioid discussions since the study started? 
Yes ☐ 
No ☐ 
 
14) Please estimate the percentage of your patients in each of these categories:  
 
 
 
b.   Have complex or numerous medical problems  _____%  

34 
 
 
 
 
  c.    Have complex or numerous psycho-social 
problems  
_____%  
 
 
d.   Are generally frustrating to deal with  
_____%  
a.   Suffer from chronic pain  
_____%  
e.   Have alcohol or other substance abuse 
disorders  
_____%  
 
15) Please indicate how much you agree or 
disagree with the following statements. 
Strongly 
disagree 
Disagre
e 
Neither 
agree nor 
disagree 
Agree 
Strongly 
Agree 
a. Many patients demand potentially 
unnecessary treatments 
□1 
□2 
□3 
□4 
□5 
b. Time pressures keep me from developing 
good patient relationships 
□1 
□2 
□3 
□4 
□5 
c. I am overwhelmed by the needs of my 
patients 
□1 
□2 
□3 
□4 
□5 
 
 
  
17) Opioid prescribing is caused by patients' “demand” for opioids. 
 
Please rate your agreement from 1 (Low) to 10 (High). * 
 
  
1 2 3 4 5 6 7 8 910 
  
18)  Reliance on opioids to treat pain is caused by doctors not having enough time with patients? 
Please rate your agreement from 1 (Low) to 10 (High). * 
1 2 3 4 5 6 7 8 9 10 
  
19) Do you generally support performance measurement and quality improvement for 

35 
 
 
doctors’ practices? 
 
Please rate your agreement from 1 (Low) to 10 (High). * 
 
You have completed the exit survey. Thank you very much for your participation! 

36 
 
 
APPENDIX D: OPIOID DECISION NODES
 
 

37 
 
 
APPENDIX E: EDUCATION MODULES 
 
All nudges (1A, 1B, and 1C): https://usc.qualtrics.com/jfe/form/SV_74Bay03kAvWkqGx  
  
 
 

38 
 
 
APPENDIX F: EPIC BUILD TECHNICAL SPECIFICATIONS 
AESOPS R21 NU Site 
EPIC Build 
Lead Site-USC 
R21 pilot site-Northwestern 
PI: Jason Doctor (jdoctor@usc.edu) 
Site PI: Jeff Linder (jlinder@northwestern.edu) 
Project Manager-Tara Knight 
(knight@healthpolicy.usc.edu) 
Site project manager: Tiffany Brown (t-
brown@northwestern.edu) 
1A: Justification alert upon initial opioid order 
1B: Justification alert upon ‘refill’ opioid order 
1C: Active Choice (SmartSet) alert for ‘refill’ >90 days opioid order  
2: Integrate new patient assessment ‘PainTracker’ at point of care when conditions are met 
Notes:  For all triggers, the triggering medication does not have to have been prescribed by study clinician 
AESOPS Nudge 1A 
Title 
Justification alert for initial opioid order 
Slides 
https://docs.google.com/presentation/d/1R5IQKGGD1kjaifRb7NE8GmbQZrz-
8GCH3ZzisPe_33o/edit#slide=id.g4d413a3b8d_0_0  
Version 
February 15, 2019 
Display 
Medication order entry display as pop up 
 
[Appears when the clinician is going to prescribe] 
Trigger 
Criteria 
Visits where order is for an included opioid 
AND 
NOT an opioid prescription with a start date < 91 days 
 
 
BPA Language 
Screen 1: DRAFT LANGUAGE: 
   “Your patient, [Name], can become dependent on opioids after 
being treated for acute pain. Safer alternatives are often just 

39 
 
 
as effective. Please either remove the order or justify the order 
at signing. This justification will be made part of the patient’s 
record.”  
 
 
Non-opioid Analgesic Alternatives 
[ ] Acetaminophen {pull-down dosing} 
[ ] Naproxen {pull-down dosing} 
[ ] Ibuprofen {pull-down dosing} 
[ ] Diclofenac Gel {pull-down dosing} 
 
Non-drug Alternative 
[ ] Physical Therapy (External) 
[ ] Physical Therapy (NM Internal) 
[ ] Physical Therapy (Athletico) 
 

40 
 
 
 
Opioid alternatives: 
https://docs.google.com/spreadsheets/d/17s6kwLwGLMEK-
uTzy6hJdWljzNInBRDXvoYR6Ou6HMM/edit?usp=sharing  
 
[ ] If you still want to prescribe, then click “Order and enter justification 
below” and note your reason for doing so. The reason(s) that you write in this box 
will appear in the Encounter Report on High Risk Prescribing. If you do not write 
a reason, this note will say “No justification was given for an opioid.”  
 
DIRECTIONS ON RESPONSE: 
1. 
free text box-[pre-populated with “no justification given”] 
2. 
Include a button to close out/acknowledge [OK} or [Cancel order] 

41 
 
 
Button 
Capture 
Buttons: 
"Remove the following orders?" 
 
"Remove" {default} 
 
"Justify at Signing" 
 
"Accept" 
 
Event upon "Accept" 
 
"Remove" removes order activates smart set alternative. 
 
"Justify at Signing" disposes Screen 2 to fire at signing orders if medication is 
still in the order list 
 
Intended 
Audience 
MDs, RNs, PAs 
Notes: 
- 
Alert should fire during any outpatient encounter type 
- 
justification text appears in the encounter report for MyChart, Telephone 
and Refill encounters 
- 
pt- reported medications added to Epic to make pt med list more accurate 
should NOT fire BPA - will not impact nudge 1A  
 
 
AESOPS Nudge 1B 
Title 
Justification alert for ‘refill’ opioid order 
Version 
February 15, 2019 
Display 
Medication order entry display as pop up 
 
[Appears when the clinician is going to prescribe] 
Trigger 
Criteria 
Visits where order is for an included opioid 
AND 
Opioid prescription with a start date > 1 day and < 91 
days 
AND 
NOT an opioid prescription with a start date > 90 days 
 
 

42 
 
 
BPA Language 
DRAFT LANGUAGE: 
Screen 1: [Patient name] has an opioid on their medication 
list within the past 30 days. Opioid refills can lead to 
long-term opioid use and dependence. This is a critical 
time to avoid formation of drug dependence by removing the 
order and choosing an alternative pain management strategy.  
 
 
  
Consider alternatives and cancel order:  
Non-opioid Analgesic Alternatives 
[ ] Acetaminophen {pull-down dosing} 
[ ] Naproxen {pull-down dosing} 
[ ] Ibuprofen {pull-down dosing} 
[ ] Diclofenac Gel {pull-down dosing} 
 
Non-drug Alternative 
[ ] Physical Therapy (External) 
[ ] Physical Therapy (NM Internal) 
[ ] Physical Therapy (Athletico) 
 

43 
 
 
 
 
Opioid alternatives: 
https://docs.google.com/spreadsheets/d/17s6kwLwGLMEK-
uTzy6hJdWljzNInBRDXvoYR6Ou6HMM/edit?usp=sharing  
 
 
 
 
DIRECTIONS ON RESPONSE: 
3. 
free text box-[pre-populated with “no justification given”] 

44 
 
 
4. 
Include a button to close out/acknowledge [OK} or [Cancel order] 
Button 
Capture 
Buttons: 
"Remove the following orders?" 
 
"Remove" {default} 
 
"Justify at Signing" 
 
"Accept" 
 
Event upon "Accept" 
 
"Remove" removes order activates smart set alternative. 
 
"Justify at Signing" disposes Screen 2 to fire at signing orders if medication is 
still in the order list 
 
Intended 
Audience 
MDs, RNs, PAs 
Notes: 
Alert should fire during any outpatient encounter type  
 
AESOPS Nudge 1C 
Title 
Enhanced Active Choice Taper >90 days Opioid Order  
Version 
February 15, 2019 
Display 
SmartSet BPA Med Order Entry with Justification* & Patient Question 
 
[Appears when the clinician is going to prescribe] 
 
* Enhanced Active Choice: Taper vs. Justification/status quo; Taper is ‘advantaged’ 
choice. 
Trigger 
Criteria 
Total opioid doses is for > 49 MME 
AND 
Two or more opioid prescriptions with two different start 
dates both > 
 1 day and < 91 days 
AND 
Opioid prescription with a start date > 90 days 
 
 

45 
 
 
Cancer Exclusions:  https://docs.google.com/spreadsheets/d/1U203LWNZ5RUY-DemA-
HZoy-uUp9GaQuGjXm03sPiosI/edit?usp=sharing  
BPA Language 
Taper Activation BPA 
[Mr. / Ms.] [Patient Last Name] is at high risk of opioid overdose with total MMED of [total 
MMED]. To facilitate gradual tapering to a safer dose (< 50 MMED), here are [Mr./Ms.] [Patient 
Last Name]’s short- and long-acting opioids. 
1.[He/she] is prescribed the following short-acting opioids: 
1.[Short-Acting AESOPS  drug #1 generic_name] 
2.[Short-Acting AESOPS  drug #1 generic_name] 
3.[… Short-Acting AESOPS  drug #N generic_name] 
Taper Suggestion: Discuss 1 pill fewer per day or < 10% MMED reduction in short-acting opioids. 
2.[He/she] is prescribed the following long-acting opioids: 
1.[Long-Acting AESOPS  drug #1 generic_name] 
2.[Long-Acting AESOPS  drug #1 generic_name] 
3.[… Long-Acting AESOPS  drug #N generic_name] 
Taper Suggestion: If there are no short-acting opioids in #1, discuss moving half MMED to short-
acting. This will give the patient more control over dose. 
I will take action toward safer prescribing: 
  [  ] YES and I oversee [Mr./Ms] [Patient Last Name]’s opioid prescribing 
  [  ] YES and I do not oversee [Mr./Ms] [Patient Last Name]’s opioid prescribing 
  [  ] NO  (requires justification at signing for continued high risk prescribing) 

46 
 
 
 
 
Ongoing Taper BPA 
[Mr. / Ms.] [Patient Last Name] is at high risk of opioid overdose with total MMED of [total 
MMED] and is on an ongoing taper to a safer dose (< 50 MMED). To facilitate this ongoing taper, 
here are [Mr./Ms.] [Patient Last Name]’s short- and long-acting opioids. 
1.[He/she] is prescribed the following short-acting opioids: 
1.[Short-Acting AESOPS  drug #1 generic_name] 
2.[Short-Acting AESOPS  drug #1 generic_name] 
3.[… Short-Acting AESOPS  drug #N generic_name] 
Taper Suggestion: Discuss 1 pill fewer per day or < 10% MMED reduction in short-acting opioids. 
2.[He/she] is prescribed the following long-acting opioids: 
1.[Long-Acting AESOPS  drug #1 generic_name] 
2.[Long-Acting AESOPS  drug #1 generic_name] 

47 
 
 
3.[… Long-Acting AESOPS  drug #N generic_name] 
Taper Suggestion: If there are no short-acting opioids in #1, discuss moving half MMED to short-
acting. This will give the patient more control over dose. 
I will take action toward safer prescribing: 
  [  ] YES 
  [  ] NO  (requires justification at signing for continued high risk prescribing) 
 
 
BPA Deferring Opioid Rx 
[Mr. / Ms.] [Patient Last Name] is on an ongoing opioid taper with [Tapering Physician Name] 
(total MMED is [TOTAL|_MMED] a safe dose is < 50 MMED). 

48 
 
 
Please do not increase this patient’s opioids. Defer to [Tapering Physician name]. 
I will defer increased opioid decisions to Dr. [Tapering Physician name] 
  [  ] YES 
  [  ] NO (requires justification at signing for continued high risk prescribing) 
 
 
Justification at Signing (failure to initiate or continue taper) 
Your opioid orders for [Patient name] indicate that you have failed to initiate or continue 
a taper of opioid medications. If you still want to prescribe, then click “Order and enter 
justification below” and note your reason for doing so. The reason(s) that you write in 

49 
 
 
this box will appear in the Encounter Report on High Risk Prescribing. If you do not write 
a reason, this note will say “No justification was given for interfering with taper.”  
 
 
 
Justification at Signing (non-deferral) 
You have prescribed an opioid to a patient being tapered off opioids by another clinician. 
If you still want to prescribe, then click “Order and enter justification below” and note 
your reason for doing so. The reason(s) that you write in this box will appear in the 
Encounter Report on High Risk Prescribing. If you do not write a reason, this note will say 
“No justification as given for not proceeding with taper.”  
 
 
 

50 
 
 
 
 
 
 
 
 
 
 
Button 
Capture 
 
Intended 
Audience 
 
Notes: 
3 BPAs depending on stage of taper; registry data collects if patient is on taper and also 
tapering clinician ID. Milligram Morphine Equivalent Daily (MMED) is given by EPIC 
calculator. 
 
AESOPS Nudge 2 
Title 
Presentation of patient assessment ‘PainTracker’ at point of care 
Version 
February 15, 2019 
Display 
Either for MA at vitals, during office visit and/or mychart (TBD) 
Trigger 
Criteria 
Align with 1C 
If “Yes” from Nudge 1C, Nudge 2 should fire  
 
 
visits where order is for an included opioid 
AND 
Total opioid doses is for > 49 MME 
AND 
Two or more opioid prescriptions with two different start 
dates both > 
 1 day and < 91 days 
AND 
Opioid prescription with a start date > 90 days 

51 
 
 
  
 
Fires every 30 days at scheduled taper follow-up visit  +/- 1 week (If they come in for 
some other problem within a week of their scheduled appointment). 
- 
not more than once every 30 days  
- 
tied to ordering prescriber scheduled encounter 
Stopping rule: 2 years (or some fixed date in the future[(e.g., for R21 stop 1/1/2020)) 
 
Assessment 
See document for all items ‘AESOPS NU Pain Tracker build’ 
BPA Language 
assessment opens in separate browser and MA logs out of Epic  
 
Pain Tracker Scoring 
 
PEG Scoring: To compute the PEG score, add the three responses to the questions 
above, then divide by three to get a final score out of 10. The final PEG score can 
mean very different things to different patients. The PEG score, like most other 
screening instruments, is most useful in tracking changes over time. 
 
PHQ Scoring: Total score is determined by adding together the scores of each of the 
4 items.Scores are rated as normal (0-2), mild (3-5), moderate (6-8), and severe (9-
12). Total score ≥3 for first 2 questions suggests anxiety. Total score ≥3 for last 2 
questions suggests depression. 
 
Button 
Capture 
 
Intended 
Audience 
 
Notes: 
Scoring is presented in Epic with graphs showing change over time and shared with 
patient via portal (mock up attached). 
 
 
 
 
Pre-text 

52 
 
 
Because you are managing your pain with Dr. XXX, we are asking you to complete a short 
PainTrackerTM survey to help you and your physician understand how pain is affecting your 
life, including everyday activities, mood and sleep. Your answers can be used to track your 
progress over time, which will help you set and manage your treatment goals. 
Please fill out PainTrackerTM 1 week or less before your next visit with Dr. XXX.  You may be 
asked to complete PainTrackerTM again for future visits.  
Thank you for taking the time to complete this survey. 
If you need help logging in or are having problems with PainTrackerTM, please plan to 
complete PainTrackerTM in the clinic at the time of your next visit.  
 
 
 
 

53 
 
 
APPENDIX G. Nudge Trigger Logic for AESOPs, Opioid Nudges 
 
1A: Opioid naïve or no recent opioid use 
Order is for an included opioid 
AND 
NOT an opioid prescription with a start date < 91 days 
  
 1B: Opioid prescription within 90 days 
Order is for an included opioid 
AND 
Opioid prescription with a start date > 1 day and < 91 days 
AND 
NOT an opioid prescription with a start date > 90 days 
   
1C: Chronic, high-dose opioids 
Order is for an included opioid 
AND 
Total opioid doses is for > 49 MME  
AND 
Two or more opioid prescriptions with two different start dates both > 1 day and < 91 days 
AND 
Opioid prescription with a start date > 90 days 
 
 
 

54 
 
 
 APPENDIX H. Informed Consent for Provider Survey  
 
Permission to Take Part in a Human Research Study 
 
Title of research study: Application of Economics & Social psychology to improve Opioid Prescribing Safety (AESOPS): baseline 
survey 
 
Investigator: Dr. Jason Doctor 
 
Why am I being invited to take part in a research study? 
We invite you to take part in a research study because you are a clinical provider who treats adult patients in [site name]. 
 
Who can I talk to? 
If you have questions, concerns, or complaints, or think the research has hurt you, talk to the research team at 213.821.7943. 
 
This research has been reviewed and approved by the USC Institutional Review Board (“IRB”). You may talk to a IRB staff member 
at 3720 South Flower Street, Third Floor, Los Angeles, CA 90089, 213.821.1154 or oprs@usc.edu for any of the following: 
 
• Your questions, concerns, or complaints are not being answered by the research team. 
• You cannot reach the research team. 
• You want to talk to someone besides the research team. 
• You have questions about your rights as a research subject. 
• You want to get information or provide input about this research. 
 
Why is this research being done? 
This research is being done to gauge your current practice and opinions regarding prescribing of opioids for the treatment of 
noncancer pain. 
How long will the research last? 
We expect that the survey will take less than 20 minutes to complete. 
 
How many people will be studied? 
We expect about XXX physicians will be in this research study at [site name] 
 

55 
 
 
What happens if I say yes, I want to be in this research? 
If you agree to participate, you may complete the survey electronically. You will be able to provide feedback about your current 
practices and opinions. Your responses will be coded using your email address and the document linking your randomly assigned code 
to your email will be kept separate from the data for analysis. 
 
Are there any risks in participating? 
No more than minimal risks are anticipated from taking part in this study. If you feel uncomfortable with a question, you can skip that 
question, or withdraw from the study altogether. If you decide to quit at any time before you have finished the questionnaire, your 
answers will NOT be recorded. 
 
What happens if I do not want to be in this research? 
You may decide not to take part in the research and it will not be held against you. 
 
What happens if I say yes, but I change my mind later? 
You can leave the research at any time and it will not be held against you. Since the data are being collected anonymously, once the 
survey is submitted the research team will not be able to delete your responses if you later decide you do not want your responses 
included in the study. 
 
Will being in this study help me in any way? 
We cannot promise any benefits to you or others from your taking part in this research. However, possible benefits include providing 
new knowledge about how to improve adherence to guidelines for opioid prescribing, which has the potential to help prevent future 
incidents of opioid use disorder and opioid poisoning death by lowering unnecessary population exposure to these drugs. 
 
What else do I need to know? 
This research is being funded by the National Institutes of Health, also called the sponsor. University of Southern California is being 
paid to conduct this study, but the study doctor and research staff have not received any direct income from the sponsor. There is no 
charge for you to participate in this study. You will be compensated with a $25 gift card for your participation. 
 
By beginning the survey, you acknowledge that you have read this information and agree to participate in this survey, with the 
knowledge that you are free to withdraw your participation at any time without penalty. 
 
 

56 
 
 
APPENDIX I. Variable List 
List of all elements to be collected: 
A.] EDW discrete variables in export 
• Patient level 
o Age 
o Race 
o Ethnicity 
o Sex 
o # of prior visits with clinician who orders opioid during study period 
o Medications ordered at encounter with opioid prescription 
o Other orders & referrals at encounter with opioid prescription 
o Diagnoses associated with opioid order 
o Insurance status 
o Yes/no to whether the following chronic conditions are on problem list at time of opioid order: 
Opioid use disorder 
Alzheimer’s Disease and related dementia 
Arthritis (Osteoarthritis and Rheumatoid) 
Asthma 

57 
 
 
Atrial Fibrillation 
Autism Spectrum Disorders 
Cancer (Breast, colorectal, lung and prostate) 
Chronic kidney disease 
Chronic Obstructive Pulmonary Disease 
Depression 
Diabetes 
Heart Failure 
Hepatitis (Chronic Viral B 7 C) 
Hyperlipidemia (High cholesterol) 
Hypertension (high blood pressure) 
Ischemic Heart Disease 
Osteoporosis 
Stroke 
• Clinician characteristics: 
o Specialty 
o Type: Attending/resident/NP/PA 
o FTE status 

58 
 
 
o PCP status (yes/not to patient’s PCP) 
o Site of care 
• Study related clinical decision support: 
o Clinician interactions with study BPAs and nudges 
• Opioid drug characteristics: 
o Order type (e.g., orders only, telephone encounter, in person encounter) 
o Drug name 
o Dose 
o Duration 
 
APPENDIX J. Coded Identifier List 
List of all identifiers to be collected or used in this study: 
• Medical record number 
• Date of birth 
• Dates of service: Date of opioid order; Date of encounters associated with opioid order; Date of Emergency Department encounter 
subsequent to opioid order 
 
 
 

59 
 
 
REFERENCES 
“Behavioral Insights to Encourage Judicious Prescribing of Opioids - Full Text View - ClinicalTrials.gov.” n.d. Accessed January 16, 2017. 
https://clinicaltrials.gov/ct2/show/NCT02790476. 
Bruine de Bruin, Wändi, Carmen E. Lefevre, Andrea L. Taylor, Suraje Dessai, Baruch Fischhoff, and Sari Kovats. 2016. “Promoting Protection 
against a Threat That Evokes Positive Affect: The Case of Heat Waves in the United Kingdom.” Journal of Experimental Psychology. 
Applied 22 (3): 261–71. 
Cantone, Jason A., and Richard L. Wiener. n.d. “Quicker and Sooner Counterfactuals: The Availability Heuristic and Negligence 
Decisionmaking.” PsycEXTRA Dataset. https://doi.org/10.1037/e506102012-061. 
Chang, Hsien-Yen, Matthew Daubresse, Stefan P. Kruszewski, and G. Caleb Alexander. 2014. “Prevalence and Treatment of Pain in EDs in the 
United States, 2000 to 2010.” The American Journal of Emergency Medicine 32 (5): 421–31. 
Cialdini, Robert B., John T. Cacioppo, Rodney Bassett, and John A. Miller. 1978. “Low-Ball Procedure for Producing Compliance: Commitment 
Then Cost.” Journal of Personality and Social Psychology 36 (5): 463–76. 
Daubresse, Matthew, Hsien-Yen Chang, Yuping Yu, Shilpa Viswanathan, Nilay D. Shah, Randall S. Stafford, Stefan P. Kruszewski, and G. Caleb 
Alexander. 2013. “Ambulatory Diagnosis and Treatment of Nonmalignant Pain in the United States, 2000–2010.” Medical Care 51 (10): 
870–78. 
Dowell, Deborah, Tamara Haegerich, and Roger Chou. 2019. “No Shortcuts to Safer Opioid Prescribing.” The New England Journal of Medicine 
380 (24): 2285–87. 
Dowell, Deborah, Tamara M. Haegerich, and Roger Chou. 2016. “CDC Guideline for Prescribing Opioids for Chronic Pain—United States, 
2016.” JAMA: The Journal of the American Medical Association 315 (15): 1624. 
Florence, Curtis S., Chao Zhou, Feijun Luo, and Likang Xu. 2016. “The Economic Burden of Prescription Opioid Overdose, Abuse, and 
Dependence in the United States, 2013.” Medical Care 54 (10): 901–6. 
Goldstein, Noah J., Robert B. Cialdini, and Vladas Griskevicius. n.d. “A Room with a Viewpoint: Using Norm-Based Appeals to Motivate 
Conservation Behaviors in a Hotel Setting.” PsycEXTRA Dataset. https://doi.org/10.1037/e633912013-996. 
Loewenstein, G., and D. Prelec. 1992. “Anomalies in Intertemporal Choice: Evidence and an Interpretation.” The Quarterly Journal of Economics 
107 (2): 573–97. 
Meeker, Daniella, Tara K. Knight, Mark W. Friedberg, Jeffrey A. Linder, Noah J. Goldstein, Craig R. Fox, Alan Rothfeld, Guillermo Diaz, and 
Jason N. Doctor. 2014. “Nudging Guideline-Concordant Antibiotic Prescribing: A Randomized Clinical Trial.” JAMA Internal Medicine 174 
(3): 425–31. 
“[No Title].” n.d. Accessed January 14, 2017. https://www.bostonfed.org/-/media/Documents/economic/conf/great-recovery-2016/Alan-B-
Krueger.pdf. 
Ochoa, Claudia, Kevin W. Jolly, and Harmon M. Hosch. n.d. “Facial Morphing and the Availability Heuristic in Alibi Evaluations.” PsycEXTRA 
Dataset. https://doi.org/10.1037/e509482008-001. 
Persell, Stephen D. 2006. “Assessing the Validity of National Quality Measures for Coronary Artery Disease Using an Electronic Health Record.” 
Archives of Internal Medicine 166 (20): 2272. 
Thaler, Richard H., and Shlomo Benartzi. 2004. “Save More TomorrowTM: Using Behavioral Economics to Increase Employee Saving.” The 
Journal of Political Economy 112 (S1): S164–87. 

60 
 
 
Tversky, Amos, and Daniel Kahneman. 1973. “Availability: A Heuristic for Judging Frequency and Probability.” Cognitive Psychology 5 (2): 
207–32. 
Vieweg, W. Victor R., William F. Carlyle Lipps, and Antony Fernandez. 2005. “Opioids and Methadone Equivalents for Clinicians.” Primary 
Care Companion to the Journal of Clinical Psychiatry 07 (03): 86–88. 
“Website.” n.d. Accessed January 7, 2017. http://wonder.cdc.gov/ucdicd10.html. 
Weimer, Melissa B., Daniel M. Hartung, Sharia Ahmed, and Christina Nicolaidis. 2016. “A Chronic Opioid Therapy Dose Reduction Policy in 
Primary Care.” Substance Abuse: Official Publication of the Association for Medical Education and Research in Substance Abuse 37 (1): 
141–47.