Nudging Provider Adoption of Clinical Decision Support
Phase: NA
Conditions: Pulmonary Embolism
Sponsor: NYU Langone Health

 
 
Official Title: 
Nudging Provider Adoption of Clinical Decision Support   
NCT Number: 
NCT05203185 
Study Number: 
21-01316 
Document Type:  
Study Protocol and Statistical Analysis Plan 
Date of the 
Document: 
• 
November 1, 2021 
 

Version Date: 08/18/2021 
 
Page 1 of 28 
 
 
 
 
RESEARCH PROTOCOL 
 
 
Protocol Title: 
Nudging Provider Adoption of Clinical Decision Support 
Principal Investigator: 
Michael Diefenbach, PhD 
Primary Contact Name: 
Sundas Khan 
Primary Contact Phone: 
516-600-1419 
Primary Contact E-mail: 
Skhan31@northwell.edu 
IRB Number: 
18-0714 
 
 
Guidelines for Preparing a Research Protocol 
 
Instructions: 
 
● You do not need to complete this document if you are submitting an Application for 
Exemption or Application for a Chart Review.   
● Do not use this template if: 
o  Your study involves an FDA regulated product.  In this case, use the Clinical 
Trial Protocol Template. 
o Your study has a protocol from a sponsor or cooperative group.  In this case, use 
the Protocol Plus.   
o Your study is a registry or repository for data and/or samples, In this case, use 
Protocol Template – Registry Studies.  .   
● If a section of this protocol is not applicable, please indicate such. 
● Do not delete any of the text contained within this document. 
● Please make sure to keep an electronic copy of this document.  You will need to use it, if 
you make modifications in the future.   
● Start by entering study information into the table above, according to these rules: 
o Protocol Title:  Include the full protocol title as listed on the application. 
o Investigator:  include the principal investigator’s name as listed on the application 
form 
o IRB Number:  Indicate the assigned IRB number, when known.  At initial 
submission, this row will be left blank.   
● Once the table information in entered, proceed to page 2 and complete the rest of the 
form. 
 
 
 
↓ Continue to next page to begin entering information about this study ↓ 
 
 

Version Date: 08/18/2021 
 
Page 2 of 28 
 
 
 
 
1.  PREVIOUS STUDY HISTORY 
 
Has this study ever been reviewed and rejected/disapproved by another IRB prior to 
submission to this IRB? 
 
 
 No  
 Yes −  if yes, please explain:  
 
 
2. BRIEF SUMMARY OF RESEARCH 
• The summary should be written in language intelligible to a moderately 
educated, non-scientific layperson.  
• It should contain a clear statement of the rationale and hypothesis of your study, 
a concise description of the methodology, with an emphasis on what will happen 
to the subjects, and a discussion of the results.  
• This section should be ½ page 
 
United States citizens receive only half of recommended medical care1 and a third of 
the care received is unnecessary2. The disparity between usual and evidence based 
clinical practice is responsible for a third of hospital deaths3 and waste estimated at 
380 billion dollars each year4. Computerized clinical decision support (CDS) attempts 
to close the gap by bringing meaningful and relevant evidence to health care 
providers at the point of decision making. Providers, policymakers, experts and 
consumers have identified these as key tools to revolutionize health care.5-7 However, 
the moderate improvements in care seen with CDS 8-14 have been significantly limited 
by consistently low provider adoption, estimated at 10% 15,16. Thus, there is a critical 
need to identify strategies to increase provider adoption of CDS. 
The central hypothesis of this proposal is that the addition of a theory-informed 
“nudge” to a CDS tool will address identified behavioral barriers to use and 
significantly improve adoption by providers. Nudges are applications of behavioral 
science, defined as positive reinforcement and indirect suggestions that have a non-
forced effect on decision making17. For example, default options for organ donation 
consent lead to striking differences in enrollment.18 Previous studies on the impact of 
nudges on provider behavior have demonstrated exceptional results (i.e. decreased 
inappropriate antibiotic prescribing from 20% to 4%19) but have not examined their 
impact on provider adoption of CDS. We propose to use a behavioral theory-
informed process to develop a new CDS tool that includes a nudge that addresses 
barriers to adoption. 
We designed a CDS tool (PE CALC) to reduce unnecessary computed tomography 
pulmonary angiography (CTPA) testing for pulmonary embolus (PE) that is 
efficacious (37% reduction among tool adopters) when used. However, its overall 
impact is limited by low provider adoption (15%). Users identified low social 
opportunity [“my colleagues don’t use the tool”], reflective motivation [“good 

Version Date: 08/18/2021 
 
Page 3 of 28 
 
 
doctors don’t need the tool”], automatic motivation [“the tool annoys me”] and 
psychological capability [“too many alerts to address”] as primary barriers to 
adoption on a survey based on the Capability Opportunity Motivation Behavior 
(COM-B) model [specifies that changing behavior requires changing capability, 
opportunity and/or motivation]20. We have identified nudges to address each of 
these barriers. The overall objective of this training application is to develop and 
evaluate the feasibility and preliminary efficacy on provider adoption of adding a 
behavioral theory-informed nudge to PE CALC. By conducting interviews with ED 
providers, we will develop a prototype of a new tool, “PERK”, which will include 
nudges to PE CALC. We will then perform “Think Aloud” and “Near Live” 
usability testing. Based on the results of these tests, we will develop PERK and test 
it in a six month pilot trial in the Emergency Departments (ED) at North Shore 
University Hospital (NSUH) and Long Island Jewish Medical Center (LIJMC). Our 
studies will provide us with information about the feasibility, acceptability, and 
preliminary efficacy of “PERK”, as well as broader knowledge of the impact of 
including nudges in CDS tools to increase provider adoption and decrease 
unnecessary patient testing.  
 
 
3. INTRODUCTION/BACKGROUND MATERIAL/PRELIMINARY STUDIES AND 
SIGNIFICANCE 
• Describe and provide the results of previous work by yourself or others, including 
animal studies, laboratory studies, pilot studies, pre-clinical and/or clinical 
studies involving the compound or device to be studied.  
• Include information as to why you are conducting the study and how the study 
differs from what has been previously researched, including what the knowledge 
gaps are. 
• Describe the importance of the knowledge expected to result 
 
Clinical decision support (CDS) has been shown to improve quality of care. Meta-
analyses of the effect of CDS have shown that providers who use decision support 
are more likely to order appropriate treatments and provide preventive care 
services.8,21-23 Extensive study of these tools has shown a morbidity decrease of 10 
to 18%, placing CDS at the top of the spectrum of quality improvement 
interventions.24 The expectation of CDS to improve the practice of medicine was 
communicated with the unprecedented federal resources ($30 billion) committed 
to supporting the use of EHRs through the HITECH Act of 2009, where 
“meaningful use” requirements demand the use of CDS.25 These tools have great 
potential to improve quality of care. 
 
In spite of the clear benefits of using CDS, meta-analyses have found provider 
adoption of these tools to be low26 with up to 96% of CDS being overridden27. 
There is clear evidence of providers not using well thought out CDS, undermining 
potential clinical impact.28 A systematic review of randomized controlled trials of 
CDS impact, found four system features to be independently predictive of 
successful clinical practice improvement.29 Each of these factors improve provider 

Version Date: 08/18/2021 
 
Page 4 of 28 
 
 
adoption (automatic triggering, including specific recommendations, workflow 
integration and computer based support). Improving provider adoption of CDS can 
help these tools reach their potential to improve quality of care. 
 
One potential method for increasing provider adoption is through the inclusion of 
nudges. Nudges are defined as positive reinforcement and indirect suggestions 
that have a non-forced effect on decision making17. Recent studies have shown 
that nudges can be effective in influencing provider decision making. For 
example, a recent large randomized controlled trial evaluating behavioral 
interventions found a social comparison “nudge” decreased provider antibiotic 
prescribing for upper respiratory infections from 19.9% to 3.7%.19 These results 
were durable at 3 months.31 Another study used a simple poster with the clinicians 
signature committing to antibiotic stewardship to decrease inappropriate antibiotic 
prescribing by 20% relative to control.30 Even simply changing the grouping of 
treatment options has demonstrated the ability to significantly impact provider 
behavior.32 
 
In our study, we will assess the impact of adding nudges to an existing CDS tool 
on efficacy provider adoption. We previously developed a tool, PE CALC, to 
reduce unnecessary imaging in the diagnosis of pulmonary embolism (PE) in the 
emergency department (ED). One common test for PE, computerized tomography 
pulmonary angiography (CTPA), is sensitive and specific for PE; however, it is 
associated with a median of 10 mSV of radiation, the equivalent of 137 chest x 
rays.33 A recent study including 2.5 million Medicare emergency visits found 
utilization to have increased fivefold between 2000 and 2009 with diagnostic 
yield declining.34 It is now thought that 1.5% to 2% of all future cancers in the 
United States may be the result of medical radiation.35 Surveyed emergency 
providers said that about a fifth of all imaging studies ordered are medically 
unnecessary,36 with factors including fear of missing a low-probability diagnosis 
and fear of litigation. Clinical prediction rules address these concerns. Clinical 
prediction rules are validated tools that quantify the contributions the history, 
physical and laboratory results make towards a diagnosis, prognosis or treatment 
response.37 They classify patients into low, intermediate and high probability of 
PE based on key elements of the history and physical exam, establishing pre-test 
probability before imaging. 
 
Each CTPA carries a 14% risk of contrast induced nephropathy38 and a lifetime 
malignancy risk that can be as high as 2.76%39. Incidental findings requiring 
diagnostic follow up are found in 24% of tests, increasing both costs and harms 
from repeat imaging.40 The use of clinical prediction rules to assess pre-test 
probability before CTPA reduces testing by 25% without any missed PEs.41,42 
Their routine use by providers would result in 600,000 fewer scans, 84,000 fewer 
cases of contrast induced nephropathy and prevent 3,000 malignancies as well as 
2,000 cancer deaths in the United States every year39.  

Version Date: 08/18/2021 
 
Page 5 of 28 
 
 
 
We based PE CALC on provider preferences, using a well-validated clinical 
prediction rule perceived as “most useful” by providers surveyed at Northwell 
Health (the Well’s Score for PE). We implemented this tool in the EDs of NSUH 
and LIJMC and found that the tool was efficacious when used, with an increase in 
CTPA yield from 8% to 12% (a 37% reduction in unnecessary CTPAs) among 
adopters. However, it did not increase CTPA yield for the ED as a whole, as it 
was only used by 15% of providers. The goal of this study is to add nudges to PE 
CALC to develop a new tool, which we will call Pulmonary Embolism Risk 
Kalculator, or PERK. To our knowledge, our study will be the first to use 
behavioral theory to illuminate the impact of nudges on provider adoption of 
CDS. We hope that the results of our study will include not only a new CDS tool 
with higher adoption rate and efficacy than our previous PE CALC, but also new 
insight into the effects of nudges on provider adoption of CDS tools.  
 
 
 
4. OBJECTIVE(S)/SPECIFIC AIMS AND HYPOTHESES 
• A concise statement of the goal(s) of the current study.  
• The rationale for and specific objectives of the study.  
• The goals and the hypothesis to be tested should be stated. 
 
The central hypothesis of this proposal is that the addition of a theory-informed 
“nudge” to a CDS tool will address identified behavioral barriers to use and 
significantly improve adoption by providers. Nudges are applications of behavioral 
science, defined as positive reinforcement and indirect suggestions that have a non-
forced effect on decision making. For example, default options for organ donation 
consent lead to striking differences in enrollment.18 Previous studies on the impact of 
nudges on provider behavior have demonstrated exceptional results (i.e. decreased 
inappropriate antibiotic prescribing from 20% to 4%19) but have not examined their 
impact on provider adoption of CDS. We propose to use a behavioral theory-
informed process to develop a new CDS tool that includes a nudge that addresses 
barriers to adoption. 
We designed a CDS tool (PE CALC) to reduce unnecessary computed tomography 
pulmonary angiography (CTPA) testing for pulmonary embolus (PE) that is 
efficacious (37% reduction among tool adopters) when used. However, its overall 
impact is limited by low provider adoption (15%). Users identified low social 
opportunity [“my colleagues don’t use the tool”], reflective motivation [“good 
doctors don’t need the tool”], automatic motivation [“the tool annoys me”] and 
psychological capability [“too many alerts to address”] as primary barriers to 
adoption on a survey based on the Capability Opportunity Motivation Behavior 
(COM-B) model [specifies that changing behavior requires changing capability, 
opportunity and/or motivation]20. We have identified nudges to address each of 
these barriers. The overall objective of this training application is to develop and 

Version Date: 08/18/2021 
 
Page 6 of 28 
 
 
evaluate the feasibility and preliminary efficacy on provider adoption of adding a 
behavioral theory-informed nudge to PE CALC. We will address this with the 
following Specific Aims:  
AIM 1: To develop nudges designed to address identified behavioral barriers 
to adoption. We will create four nudges designed to address four identified 
barriers of tool use, using on our survey results and COM-B as a framework.  
AIM 2: To conduct iterative usability testing on an evolving prototype of a 
new tool, “PERK”. This will involve both qualitative and quantitative assessments 
of ease of use of “PERK”. We will perform “think aloud” and “near live” testing to 
develop a prototype that will be revised and evaluated.   
AIM 3: To evaluate the feasibility, acceptability and preliminary efficacy on 
provider adoption of a new tool compared to the current tool, in a pilot trial. We 
will conduct a 6 month pre-post analysis of “PERK” by comparing it to retrospective 
data from the first 6 months of PE CALC. Our primary feasibility outcome is post-
pilot provider satisfaction. Secondary outcomes include stakeholder acceptability. 
H1: PERK will demonstrate higher rates of provider satisfaction as compared to PE 
CALC. Our primary efficacy outcome is increased provider adoption. Secondary 
outcomes include increased CTPA yield [% of positive tests], and decreased CTPA 
ordering. H2: “PERK” will demonstrate higher provider adoption as compared to 
PE CALC. 
 
 
 
5.  RESOURCES AVAILABLE TO CONDUCT THE HUMAN RESEARCH 
● Explain the feasibility of meeting recruitment goals of this project and demonstrate a 
potential for recruiting the required number of suitable subjects within the agreed 
recruitment period 
o How many potential subjects do you have access to? 
● Describe your process to ensure that all persons assisting with the trial are 
adequately informed about the protocol and their trial related duties and functions 
 
The study will collect data from the Emergency Departments (ED) of both North 
Shore University Hospital (NSUH) and Long Island Jewish Medical Center (LIJMC). 
Eligible participants for all three aims will be medical doctors and physician 
assistants working full time in either of these Emergency Departments (86 eligible at 
NSUH; 63 eligible at LIJMC). We will recruit from within this pool for AIM 1 and 
AIM 2. For AIM 3, all eligible providers will be automatically enrolled, as we will be 
collecting data from the EHR regarding the use of PERK.  
 
All persons assisting with the trial will review the research protocol and be informed 
of their specific duties prior to the commencement of any research activities. We will 
also hold a study launch meeting for all involved in the research.  
 
 
6.  RECRUITMENT METHODS 

Version Date: 08/18/2021 
 
Page 7 of 28 
 
 
• Describe the source of potential subjects 
• Describe the methods that will be used to identify potential subjects 
• Describe any materials that will be used to recruit subjects.  A copy of any 
advertisements (flyers, radio scripts, etc.) should be submitted along with the 
protocol.  
• If monetary compensation is to be offered, this should be indicated in the 
protocol 
 
Eligible subjects for all three aims of this study will be medical doctors and physician 
assistants working full time at NSUH or LIJMC. This includes residents in the 
Emergency Medicine/Internal Medicine program.  
 
For AIM 1, potential subjects will be identified using the Electronic Medical Records 
(EMR) based on PE CALC use. We will contact participants via our secure email 
system. We will inform participants about the study, offering a $50 reimbursement 
for their time.  
 
In addition, we will contact department heads at NSUH and LIJMC, as well as 
administrators, attendings, chief residents, administrative residents, and head PAs, 
and request names of potential participants for Aims 1 and 2. We will email the 
identified providers, mentioning the person who recommended that they might be 
interested in participating, and asking if they would like to participate in our study. 
Alternatively, we will ask department heads, administrators, or other faculty or staff 
to email potential participants and inform them of our study. We may also present our 
study at ED faculty meetings and/or resident meetings to raise awareness of our 
study. Alternatively, we will ask department heads, administrators, or other faculty or 
staff to present our study in information sessions to promote participation.  
 
For AIM 2, eligibility and recruitment methods will be the same as for AIM 1, as 
indicated in the previous paragraph. We will contact participants using the health 
system email. We may also ask department heads, administrators, or other faculty or 
staff to distribute and/or display flyers to promote our study. The flyer may also be 
distributed through the physician Teams chat. We will inform potential participants 
and solicit interest, offering a $50 reimbursement for their time. There may be one 
follow-up phone call.  
 
For AIM 3, full time ED providers will be eligible. We will conduct a pilot trial of 
PERK among 150 Emergency Medicine providers at NSUH and LIJMC. We expect 
150 providers who meet the inclusion criteria to be in each of the pre- and post- 
groups.  
 
 
7.  ELIGIBILITY CRITERIA 
• Describe the characteristics of the subject population, including their anticipated 
number, age, ranges, sex, ethnic background, and health status. Identify the 
criteria for inclusion or exclusion of any subpopulation.  

Version Date: 08/18/2021 
 
Page 8 of 28 
 
 
• Explain the rationale for the involvement of special classes of subjects, such as 
fetuses, pregnant women, children, prisoners or other institutionalized 
individuals, or others who are likely to be vulnerable.   You cannot include these 
populations in your research, unless you indicate such in the protocol 
• Similarly, detail exclusionary criteria: age limits, special populations (minors, 
pregnant women, decisionally impaired), use of concomitant medications, 
subjects with other diseases, severity of illness, etc. 
 
 
Eligible subjects for all three aims of this study will be medical doctors and physician 
assistants working full time at NSUH or LIJMC. Participants will not be excluded by 
gender, race, or ethnicity. The providers in this study are expected to reflect the 
gender and racial/ethnic composition of the Emergency Department providers at 
North Shore University Hospital (NSUH) and Long Island Jewish Medical Center 
(LIJMC), who are approximately 35% women, 55% non-Hispanic White, 32% Asian, 
6% Hispanic/Latino (of any race), 3% Black, and 4% multiracial/other. As this study 
focuses on adult medical providers, children will not be included. We anticipate adult 
participants age 25-70.  
 
For AIM 1, we expect to recruit 12 participants to complete semi-structured 
interviews; however, we will recruit until thematic saturation is achieved.  
 
For AIM 2, we expect to recruit 10 participants (5 for “Think Aloud” testing; 5 for 
“Near Live” testing); however, we will recruit until thematic saturation is achieved.  
 
For AIM 3, we expect to include 150 providers who meet the inclusion criteria in the 
prospective cohort pilot study, and 150 in the retrospective cohort control group.   
 
 
 
8.  NUMBER OF SUBJECTS 
• Indicate the total number of subjects to be accrued locally.  If applicable, 
distinguish between the number of subjects who are expected to be pre-screened, 
enrolled (consent obtained), randomized and complete the research procedures. 
• If your study includes different cohorts, include the total number of subjects in 
each cohort. 
• If this is multisite study, include total number of subjects across all sites 
 
For AIM 1, we expect to recruit 12 participants to complete semi-structured 
interviews; however, we will recruit until thematic saturation is achieved.  
 
For AIM 2, we expect to recruit 10 participants (5 for “think aloud” testing; 5 for 
“near live” testing); however, we will recruit until thematic saturation is achieved.  
 
For AIM 3, we expect to include 150 providers who meet the inclusion criteria in the 
prospective cohort pilot study, and 150 in the retrospective cohort control group.  

Version Date: 08/18/2021 
 
Page 9 of 28 
 
 
 
 
 
9. STUDY TIMELINES 
● Describe the duration of an individuals participation in the study 
● Describe the duration anticipated to enroll all study subjects 
● The estimated date of study completion 
 
Enrollment will occur on a rolling basis throughout the testing period for AIM 1. 
Interviews for AIM 1 will take place during the first six months of Year 1, beginning 
in January 2019. Each participant will be interviewed for a 20-30 minute period, 
during which they will also be consented. It is possible that participants will be 
contacted again during the six month interview period for clarification; however, we 
do not anticipate contacting most providers outside of their interview sessions. 
Following the interview period, we will develop a “PERK” prototype in the second 
half of Year 1.  
 
Enrollment will occur on a rolling basis throughout the testing period for AIM 2. For 
AIM 2, usability testing and iterative revision of the prototype will take place during 
Year 2 of the grant. Each participant will take part in a usability testing session 
approximately 20-30 minutes in duration, during which they will also be consented.  
Participants will also be given the option to fill out the pre-session questionnaire, 
consent form, and Audio Visual Authorization form on REDCap prior to their 
session. It is possible that participants will be contacted again during the one year 
period of usability testing sessions for clarification; however, we do not anticipate 
contacting most providers outside of their usability sessions.    
 
During the first half of Year 3, we will develop the new PERK tool. In the second half 
of Year 3, we will recruit for the pilot clinical trial in AIM 3. The new PERK tool will 
be active for 6 months, and all full-time employed providers working in the 
emergency department at NSUH and LIJMC will be eligible for participation. We will 
perform the pilot trial in Year 4. The clinical decision support tool will be integrated 
into the electronic medical record and will be accessible for all full-time employed 
emergency department providers (n=150). Providers will be trained on using the tool 
through 30 minute sessions delivered during emergency department grand rounds. 
Additional education sessions will be delivered to continue participant engagement 
over time. In the second half of Year 4, to assess feasibility, providers will be 
contacted by email to participate in a post-pilot satisfaction survey. Reminder emails 
to complete the survey will be sent out up to five times. To assess acceptability, 
providers will be contacted by email to participate in one-time stakeholder interviews. 
A follow up phone call will be conducted with providers to provide participant 
information and solicit interest. The pilot trial will end in June 2022. We will 
complete our analyses in the first quarter of Year 5. The anticipated end date for this 
grant is August 31st, 2023.  
 
 

Version Date: 08/18/2021 
 
Page 10 of 28 
 
 
10.  ENDPOINTS 
● Describe the primary and secondary study endpoints 
● Describe any primary or secondary safety endpoints  
 
The endpoint of this study is provider adoption of the PERK tool. Secondary 
outcomes include increased CTPA yield [% of positive tests], and decreased CTPA 
ordering. 
 
Our primary feasibility outcome is post-pilot provider satisfaction. 
 
 
11.  RESEARCH PROCEDURES 
● Include a detailed description of all procedures to be performed on the research 
subject and the schedule for each procedure. 
● Include any screening procedures for eligibility and/or baseline diagnostic tests 
● Include procedures being performed to monitor subjects for safety or minimize risks 
● Include information about drug washout periods 
● If drugs or biologics are being administered provide information on dosing and route 
of administration 
● Clearly indicate which procedures are only being conducted for research purposes. 
● If any specimens will be used for this research, explain whether they are being 
collected specifically for research purposes.   
● Describe any source records that will be used to collect data about subjects 
● Indicate the data to be collected, including long term follow-up 
 
In AIM1, we will conduct semi-structured interviews with low, intermediate, 
and high users of the current tool, PE CALC (n=12 total, 4 each, continuing 
until data saturation) to refine our understanding of barriers to tool use and 
receive feedback on nudges’ acceptability and ability to address identified 
barriers. Guided by our preliminary survey data and the COM-B framework, we 
will develop an interview guide. The interview will have three goals: 1) refinement 
of the four most prevalent barriers, 2) assessment of how well each nudge addresses 
the behavioral barrier, and 3) acceptability of the nudge. Dr. Safiya Richardson will 
conduct structured interviews at NSUH, LIJMC, 600 Community Drive, Manhasset, 
and other clinical sites including 257-07 Union Tpke, Glen Oaks with Dr. Katherine 
Dauber, PhD, an experienced researcher. The interviews will be audio recorded 
(using two digital voice recorders; one for backup in case of technical failure) and 
professionally transcribed. The guide will be revised based on the ongoing analysis 
of interview data. 
 
In AIM 2, we will perform iterative usability testing of an evolving prototype of 
“PERK”. We will build the prototype into the wireframing tool, Axure. The 
PERK prototype will add 4 small nudges to PE CALC. Usability of the 
prototype will be tested in two steps, “Think Aloud” and “Near Live” (n=5 
participants each). During “Think Aloud” testing, participants verbalize their 
thought processes while interacting with the prototype. Participants will be 
given written clinical scenarios with low, intermediate, or high probability PE 
cases and asked to complete the specific steps necessary to use the tool. During 

Version Date: 08/18/2021 
 
Page 11 of 28 
 
 
“Near Live” testing, participants will interact with a standardized patient 
pretending to have a low, intermediate, or high probability of PE. Participants 
will use the tool as they would during a real patient encounter to highlight higher 
level workflow usability issues.  
All five participants for each round of testing will interact with the prototype at 
each step. The tool will be modified after “Think Aloud” and again after “Near 
Live” based on testing results. Before every session participants will be asked to 
complete a questionnaire to gather demographic information. They will be given 
a laptop with the PERK prototype and screen capture software. After every 
session they will complete a questionnaire to collect retrospective impressions of 
the tool, and a quantitative survey, the System Usability Scale. Each session will 
be audio-taped and Morae usability testing software will be used to screen-
capture record all human-EHR interactions.  
 
AIM 2 participants will be tested either in person or remotely via 
videoconferencing (such as with Microsoft Teams). Participants being tested 
remotely will fill out consent forms, the Audio Visual Authorization Form, and the 
surveys on REDCap. Participants being tested in person will be given the option to 
fill out these forms on REDCap or on paper. Audio and screen capture recordings 
(using Hypercam, Teams, or Morae) will be taken for each session. Prior to or at the 
beginning of each session, participants will fill out the Usability Pre-session 
Questionnaire. At the end of each session, participants will verbally answer 
questions from the Post-session Interview and PERK Tool (Feedback Survey Aim 2) 
documents; they will also fill out the System Usability Scale (on REDCap for 
remote testing, on paper or REDCap for in-person testing).   
 
In AIM 3, we will conduct a pilot trial to evaluate the feasibility, acceptability, 
and preliminary efficacy on provider adoption of the new tool, “PERK”, 
compared to the current PE CALC tool. This will consist of a six month 
prospective cohort pilot study with a retrospective cohort control group. All full time 
ED providers at NSUH and LIJMC will be eligible for participation. Training on 
PERK use will be conducted as a presentation during ED Grand Rounds. This 
will be led by Dr. Safiya Richardson and a research coordinator. It will last 
about 30 minutes with three basic components: 1) overview of the evidence 
supporting tool use, 2) hands-on session to practice using the tool, and 3) 
question and answer session. Similar educational training was conducted before 
launching the current tool, PE CALC. 
For the pilot study, we will use EHR reporting data to generate a PERK 
monitoring dashboard. The dashboard will include trigger rates (percentage of 
ED patients for which the tool triggers), adoption rates (the percentage of times 
the provider completes the calculator once the tool is triggered), CTPA order 
rates (percentage of total ED patients for which a CTPA is ordered for the 
evaluation of PE), and CTPA yield (the percentage of tests ordered to evaluate 
for PE that are positive for PE). 
 
 
 
12.  STATISTICAL ANALYSIS 
● Describe how your data will be used to test the hypotheses.  

Version Date: 08/18/2021 
 
Page 12 of 28 
 
 
● State clearly what variables will be tested and what statistical tests will be uForsed. 
● Include sample size calculations.  
● If this is a pilot study, state which variables will be examined for hypothesis 
generation in later studies.  
 
For AIM 1, we will perform qualitative analyses of our semi-structured interviews.  
We will use NVivo software (QSR International, Inc.) for qualitative data storage and 
organization. Use of computer-assisted techniques improves procedure 
standardization, enhances completeness, and permits greater flexibility in revising the 
analysis process. We will code and analyze the transcripts using a modified grounded 
theory informed by COM-B. In grounded theory, themes are not specified prior to 
analysis, but our modified approach will use the framework of COM-B to organize 
emergent themes (e.g., psychological capability, social opportunity). Dr. Dauber and 
Dr. Richardson will develop a codebook reflecting emerging themes after iteratively 
reading the transcriptions so that triangulation can be assessed and a broader and more 
complex understanding of the data attained. Using the constant comparative method, 
additional readings of the transcriptions will lead to the consolidation of these coding 
schemes until no further refinement is required. Coding discrepancies will be resolved 
by consensus. Coder agreement will be calculated with Cohen’s kappa. In order to 
improve accuracy, validity, and transferability of the results, respondent validation 
will be implemented. Participants will be shown an analysis summary and asked to 
provide feedback if it conforms to their responses during interviews ensuring that the 
participant meanings and perspectives are accurately represented.  
 
For AIM 2, we will perform qualitative analyses of our “Think Aloud” and “Near 
Live” usability testing. After all usability testing, the audio recordings of subjects 
will be transcribed verbatim and linked to screen capture videos. Two independent 
coders will read the annotated transcripts and watch the corresponding video for 
each case. All discrepancies in the coding will be resolved by discussion to achieve 
a consensus. Each coded segment of the transcribed text will be categorized for: (a) 
the CDS components involved, (b) usability/workflow issues as conveyed through 
providers’ verbalizations and actions, (c) type of commentary provided by the 
subject (i.e. “positive”, “neutral” or “negative” commentary). We will perform a 
thematic analysis of the transcribed audio recordings as recommended by 
Boyatzis.69 Frequencies of codes will be tabulated for each subject. Morae software 
allows annotation of each of the components being used, time of use and usability 
theme. During “Near Live” usability testing the onscreen video recordings will be 
reviewed for time to CDS completion and time spent in workflow components. We 
will use NVivo software (QSR International, Inc.) for qualitative data storage and 
organization. We will code and analyze the transcripts using a modified grounded 
theory informed by COM-B. In grounded theory, themes are not specified prior to 
analysis, but our modified approach will use the framework of COM-B to organize 
emergent themes (e.g., psychological capability, social opportunity). Dr. 
Richardson and a trained research assistant develop a codebook reflecting emerging 
themes after iteratively reading the transcriptions. Using the constant comparative 
method, additional readings of the transcriptions will lead to consolidation of these 

Version Date: 08/18/2021 
 
Page 13 of 28 
 
 
coding schemes until no further refinement is required. Coding discrepancies will 
be resolved by consensus.  
 
For AIM 3, we will determine the feasibility of PERK using post-pilot provider 
satisfaction surveys, where mean scores for each response on a Likert scale will be 
reported with standard deviations and 95% confidence intervals. Acceptability will be 
determined from post-pilot stakeholder interviews with providers, nurses and the ED 
chairs at both institutions. Preliminary efficacy will be assessed by comparing 
provider adoption of PERK to provider adoption for the first six months of PE CALC. 
Providers in the pre- and post- groups will be summarized using descriptive statistics. 
Provider age, gender, race and type (attending vs. resident vs. physician assistant) will 
be compared between the two groups using two-sample t-test or Wilcoxon Rank Sum 
test for continuous variables and Chi-squared or Fisher’s exact test for categorical 
variables. Since the variability of the adoption rate of PERK for each provider 
depends, in part, on the number of encounters, providers with larger numbers of 
encounters should carry more weight. Therefore, a weighted two-sample t-test will be 
used to test the difference in the weighted means of adoption rates between the 
groups. Weighted least squares linear regression with number of encounters as weight 
for each observation (provider) will be performed, to see whether there is difference 
in the adoption rate between the two groups after controlling for provider age, gender, 
race, type and other confounding variables. 
 
Power Calculations: We expect 150 providers who meet the inclusion criteria to be in 
each of pre- and post- groups. We also assume the mean adoption rate of the pre 
group to be 15% with the standard deviation of 10% based on the historical data, and 
the mean adoption rate of the post- group to be 20%, 25% and 30% with equal 
standard deviation. With the sample size of 150 in each group, there will be at least 
99% power to detect the difference of 5%, 10% and 15% in the mean adoption rates 
using a two-sided two-sample t-test at a significance level of 0.05. We are aware that 
the weighted two-sample t-test should be used to calculate the power. However, due 
to the lack of information about weights, the unweighted two-sample t-test was used 
instead. 
 
 
 
 
13. SPECIMEN BANKING 
● If specimens will be banked for future research, describe where the specimens will be 
stored, how long they will be stored, how they will be accessed and who will have 
access to the specimens 
● List the information that will be stored with each specimen, including how specimens 
are labeled/coded 
● Describe the procedures to release the specimens, including:  the process to request 
release, approvals required for release, who can obtain the specimens, and the 
information to be provided with the specimens. 
 

Version Date: 08/18/2021 
 
Page 14 of 28 
 
 
n/a 
 
 
14.  DATA MANAGEMENT AND CONFIDENTIALITY 
● Describe the data and specimens to be sent out or received.  As applicable, describe: 
o What information will be included in that data or associated with the 
specimens? 
o Where and how data and specimens will be stored? 
o How long the data will be stored? 
o Who will have access to the data? 
o Who is responsible for receipt or transmission of data and specimens? 
● Describe the steps that will be taken to secure the data during storage, use and 
transmission. 
 
Data obtained for this study include the audio/video recordings of interviews, as well 
as “Think Aloud” and “Near Live” usability testing. We will also be extracting EHR 
data. We plan on storing study data on the health system’s encrypted shared drive. 
Data will be stored for 7 years following the completion of the study, which is the 
minimum allowed by our institution’s the IRB. Only study team members that are on 
the Institutional Review Board (IRB) approved protocol will have access to the data 
that will be saved on a secured Northwell Health drive, Northwell REDCap, and/or 
Northwell OneDrive. Research team members will also be responsible for the 
transmission of data. They will meet the minimum and mandatory requirement to 
access the identifiable information as per the standards set by Northwell Health. 
Access and use of the data will be strictly monitored by the Principal Investigator (PI) 
and the research team. Northwell Health’s Administrative Policy and Procedure 
Manual policy number 800.42 titled Confidentiality of Protected Health Information 
particularly addresses the access and use of the ePHI. 
  
 
 
15.  DATA AND SAFETY MONITORING PLAN 
 
A specific data and safety monitoring plan is only required for greater than minimal risk 
research.  For guidance on creating this plan, please see the Guidance Document on the 
HRPP website. 
 
Part I – this part should be completed for all studies that require a DSMP.   
Part II – This part should be completed when your study needs a Data and 
Safety Monitoring Board or Committee (DSMB/C) as part of your Data and 
Safety Monitoring Plan.   
 
Part I:  Elements of the Data and Safety Monitoring Plan 

Version Date: 08/18/2021 
 
Page 15 of 28 
 
 
●  Indicate who will perform the data and safety monitoring for this study.   
● Justify your choice of monitor, in terms of assessed risk to the research subject’s 
health and well being.  In studies where the monitor is independent of the study staff, 
indicate the individual’s credentials, relationship to the PI, and rationale for 
selection 
● List the specific items that will be monitored for safety (e.g. adverse events, protocol 
compliance, etc) 
● Indicate the frequency at which accumulated safety and data information (items listed 
in # above) will be reviewed by the monitor (s) or the DSMB/C.   
● Where applicable, describe rules which will guide interruption or alteration of the 
study design.   
● Where applicable, indicate dose selection procedures that will be used to minimize 
toxicity. 
● Should a temporary or permanent suspension of your study occur, in addition to the 
IRB, indicate to whom will you report the occurrence.   
 
Please see APPENDIX A: Data Safety Monitoring Plan (DSMP).   
 
 
Part II:  Data and Safety Monitoring Board or Committee 
 
 
●  When appropriate, attach a description of the DSMB.   
● Provide the number of members and area of professional expertise.   
● Provide confirmation that the members of the board are all independent of the study. 
 
n/a 
 
 
16. WITHDRAWAL OF SUBJECTS 
● Describe anticipated circumstances under which subjects will be withdrawn from the 
research without their consent 
● Describe procedures for orderly termination 
● Describe procedures that will be followed when subjects withdraw from the research, 
including partial withdrawal from procedures with continued data collection.   
 
For AIM 1 and AIM 2, we will explain to providers at the time of consent that their 
participation is voluntary and they are able to withdraw at any time. To withdraw, 
participants will need to notify the PI in writing of their withdrawal. There is no 
consenting, and therefore no withdrawal, for AIM 3.  
 
 
17. RISKS TO SUBJECTS 
● Describe any potential risks and discomforts to the subject (physical, psychological, 
social, legal, or other) and assess their likelihood and seriousness and whether side 

Version Date: 08/18/2021 
 
Page 16 of 28 
 
 
effects are reversible. Where appropriate, describe alternative treatments and 
procedures that might be advantageous to the subjects.  
● Include risks to others , like sexual partners (if appropriate) 
● Discuss why the risks to subjects are reasonable in relation to the anticipated benefits 
and in relation to the importance of the knowledge that may reasonably be expected 
to result.  
● Describe the procedures for protecting against or minimizing any potential risks, 
including risks to confidentiality, and assess their likely effectiveness.   
 
Providers participating in this study will be at minimal risk. We do not expect any 
adverse events directly attributable to the interviews, usability testing, or pilot study. 
Although adverse events associated with behavioral protocols such as ours are highly 
unlikely, in order to protect the safety of participants, all are provided with numbers 
for NWH’s Emergency Rooms and referrals to the Department of Social Work. 
 
A risk of potential breach of confidentiality of study data exists but the study team 
has put measures in place to prevent a breach from occurring; we plan on storing 
study data on the health system’s encrypted shared drive. Data will also be stored on 
Northwell OneDrive and in Northwell REDCap, Only study team members that are 
on the Institutional Review Board (IRB) approved protocol will have access to the 
data that will be saved on a secured Northwell Health drive. They will meet the 
minimum and mandatory requirement to access the identifiable information as per the 
standards set by Northwell Health. Access and use of the data will be strictly 
monitored by the Principal Investigator (PI) and the research team. Northwell 
Health’s Administrative Policy and Procedure Manual policy number 800.42 titled 
Confidentiality of Protected Health Information particularly addresses the access and 
use of the ePHI. 
 
Throughout the development of the CDS, we will ensure the highest levels of security 
and HIPAA compliance. All communications will be encrypted during transit and all 
data will be encrypted at the system level.  
 
We do not anticipate any significant physical, psychological, or social risk to the 
study providers. All risks and benefits of participation will be explained to providers 
and included in the written informed consent forms. It will also be explained to 
providers that their participation is voluntary and they are able to withdraw at any 
time. 
 
 
 
 
18. RESEARCH RELATED HARM/INJURY 
● Describe the availability of medical or psychological resources that subjects might 
need as a result of anticipated problems that may be known to be associated with the 
research. 

Version Date: 08/18/2021 
 
Page 17 of 28 
 
 
● If the research is greater than minimal risk, explain any medical treatments that are 
available if research-related injury occurs, who will provide it, what will be provided, 
and who will pay for it.   
 
We do not expect any adverse events directly attributable to the interviews, usability 
testing, or pilot study. Although adverse events associated with behavioral protocols 
such as ours are highly unlikely, in order to protect the safety of participants, all are 
provided with numbers for NWH’s Emergency Rooms and referrals to the 
Department of Social Work. 
 
 
19. POTENTIAL BENEFIT TO SUBJECTS 
 
● Explain what benefits might be derived from participation in the study, noting in 
particular the benefit over standard treatment (e.g. a once-a-day administration 
instead of four times a day, an oral formulation over an IV administration).  
● Also state if there are no known benefits to subjects, but detail the value of knowledge 
to be gained 
 
Participants will not directly benefit from taking part in this study. This study will 
improve our understanding of how to develop clinical decision support that is adopted 
by providers. Improved provider adoption of clinical decision support helps to close 
the gap between usual and evidenced based clinical care, reducing morbidity, 
mortality and health care waste. Adoption of this particular clinical decision support 
tool will decrease unnecessary testing for pulmonary embolism with computed 
tomography with contrast, reducing contrast induced nephropathy and malignancies 
from radiation exposure.     
 
 
20. PROVISIONS TO PROTECT PRIVACY INTERESTS OF SUBJECTS 
 
● Describe the methods used to identify potential research subjects, obtain consent and 
gather information about subjects to ensure that their privacy is not invaded.  
● In addition consider privacy protections that may be needed due to communications 
with subjects (such as phone messages or mail).   
 
All participants will be given Subject Numbers by which they will be identified. The 
names of participants will not be associated with the participants' responses..  
 
For AIM 1 and AIM 2, a member of the research team will obtain informed consent 
from providers at the time of enrollment. Providers will be informed of the study’s 
risk and benefits. Researchers will explain to each participant that they have the 
ability to opt out of the study at any time. Whether or not they choose to participate in 
the study will have no effect on their employment or compensation. For HIPPA 
privacy purposes, providers will be informed that their names are not attached to 
study documents. Before being consented, providers will have the opportunity to ask 

Version Date: 08/18/2021 
 
Page 18 of 28 
 
 
the researcher any questions to clarify inquiries or concerns. Documents given to 
providers will contain telephone numbers of appropriate personnel, if further 
questions about the study arise. Providers will also be asked separately to give 
consent to be audio taped. As per Northwell Health policy 100.47, clinical providers 
will also be asked to sign an Audio/Video recording authorization form. Audio 
recordings and other PHI will be stored on the health system’s encrypted shared drive 
and/or Northwell OneDrive and will only be accessible by the study team.  
 
 
 
 
 
21.  COSTS TO SUBJECTS 
 
● Describe any foreseeable costs that subjects may incur through participation in the 
research 
● Indicate whether research procedures will be billed to insurance or paid for by the 
research study.   
 
There is no cost subjects. 
 
 
 
22. PAYMENT TO SUBJECTS  
 
● Describe the amount of payment to subjects, in what form payment will be received 
and the timing of the payments.   
 
Participants who complete the semi-structured interview in AIM 1 or the usability 
testing in AIM 2 will be reimbursed $50. Payment will be made directly to 
participants as an addition to their regular paychecks. There will be no payment to 
subjects in AIM 3.  
 
 
23.  CONSENT PROCESS 
 
If obtaining consent for this study, describe: 
● Who will be obtaining consent 
● Where consent will be obtained 
● Any waiting period available between informing the prospective participant and 
obtaining consent 
● Steps that will be taken to assure the participants’ understanding 
● Any tools that will be utilized during the consent process 
●   Information about how the consent will be documented in writing.  If using a  
standard consent form, indicate such.   
● Procedures for maintaining informed consent.   
 

Version Date: 08/18/2021 
 
Page 19 of 28 
 
 
For AIM 1 and AIM 2, informed consent will be obtained for all testing at the 
Emergency Departments at NSUH and LIJMC. A member of the research team 
will obtain informed consent from patients and surrogates at the time of 
enrollment. Providers will be informed of the study’s risk and benefits. 
Researchers will explain to each participant that they have the ability to opt out of 
the study at any time. Whether or not they choose to participate in the study will 
have no effect on their employment or compensation. For HIPPA privacy 
purposes, providers will be informed that their names are not attached to study 
documents. Before being consented, providers will have the opportunity to ask the 
researcher any questions to clarify inquiries or concerns. Documents given to 
providers will contain telephone numbers of appropriate personnel, if further 
questions about the study arise. Providers will also be asked separately to give 
consent to be audio taped. As per Northwell Health policy 100.47, clinical 
providers will also be asked to sign an Audio/Video recording authorization form. 
 
AIM 2 participants being tested remotely will fill out consent forms, the Audio 
Visual Authorization Form, and the surveys on REDCap. AIM 2 participants 
being tested in person will be given the option to do this online or on paper.  
 
For AIM 3, we will collect pilot data from the EHR on the use of PERK. All 
providers at NSUH and LIJMC will be automatically enrolled. There will be no 
consenting for AIM 3; however, we hold an information session for providers on 
how to use PERK prior to launching the tool.  
 
 
 
In the state of NY, any participants under the age of 18 are considered children.  If your 
study involves children, additional information should be provided to describe: 
● How parental permission will be obtained 
● From how many parents will parental permission be obtained 
● Whether permission will be obtained from individuals other than parents, and if 
so, who will be allowed to provide permission.  The process used to determine 
these individual’s authority to consent for the child should be provided 
● Whether or not assent will be obtained from the child 
● How will assent be documented 
● Whether child subjects may be expected to attain legal age to consent to the 
procedures for research prior to the completion of their participation in the 
research.  If so, describe the process that will be used to obtain their legal 
consent to continue participation in the study.  Indicate what will occur if consent 
is not obtained from the now-adult subjects.   
 
n/a 
 
 
If the study involves cognitively impaired adults, additional information should be 
provided to describe: 

Version Date: 08/18/2021 
 
Page 20 of 28 
 
 
● The process to determine whether an individual is capable of consent 
● Indicate who will make this assessment 
● The plan should indicate that documentation of the determination  and 
assessment will be placed in the medical record, when applicable, in addition to 
the research record. 
● If permission of a legally authorized representative will be obtained, 
o  list the individuals from who permission will be obtained in order of priority 
o Describe the process for assent of subjects; indicate whether assent will be 
required of all, some or none of the subjects.  If some, which subjects will be 
required to assent and which will not. 
o If assent will not be obtained from some or all subjects, provide an 
explanation as to why not 
o Describe whether assent will be documented and the process to document 
assent 
o Indicate if the subject could regain capacity and at what point you would 
obtain their consent for continued participation in the study 
 
n/a 
 
 
If the study will enroll non-English speaking subjects: 
● Indicate what language(s) other than English are understood by prospective 
subjects or representatives 
● Indicate whether or not consent forms will be translated into a language other 
than English 
● Describe the process to ensure that the oral and written information provided 
to those subjects will be in that language 
● If non-English speaking subjects will be excluded, provide a justification for 
doing so 
 
n/a 
 
 
24.  WAIVER OR ALTERATION OF THE CONSENT PROCESS         
 
Complete this section if you are seeking an alteration or complete waiver of the consent   
process. 
● Describe the possible risks of harm to the subjects involved in this study and 
explain why the study involves no more than minimal risk to the subject:   
● Explain why the waiver/ alteration will not adversely affect the rights and welfare 
of subjects 
● Explain why it is impracticable to conduct this research if informed consent is 
required   
● If appropriate, explain how the subjects will be provided with additional pertinent 
information after participation.   If not appropriate to do so, explain why.  
 

Version Date: 08/18/2021 
 
Page 21 of 28 
 
 
For AIM 3, in which we will collect pilot data on the feasibility, acceptability, 
and preliminary efficacy on provider adoption of the PERK tool, we are 
requesting a waiver of informed consent. A waiver of informed consent will be 
necessary in order to have the PERK tool activated in the EMR profiles of the 
providers.  
 
The research involves no more than minimal risks to the subjects: The study as 
described in the proposal would be considered a minimal risk study because it 
involves collection of de-identified aggregate data from the EHR at the NSUH 
and LIJMC Emergency Departments.  
 
The waiver or alteration will not adversely affect the rights and welfare of the 
subjects: Due to the functionality of the tool, providers will be offered the tool 
and have the ability to dismiss the tool. Use of the tool will not affect 
employment status of the providers at the clinical site/hospital/health system. 
We will hold an information session for providers regarding the research study 
and their participation. 
 
The research could not practicably be carried out without the waiver: The 
primary objective of AIM 3 is to assess the feasibility, acceptability, and 
preliminary efficacy on provider adoption of PERK compared to the current 
PE CALC tool in a pilot trial. In order to study this, all patients assessed for PE 
must be included. The decision support is completely computerized and will be 
done in real time. It would not be possible or practical to obtain consent from 
every provider in the Emergency Departments of NSUH and LIJMC prior to 
their use of the PERK.  
 
Whenever appropriate, the subjects will be provided with additional pertinent 
information after participation: While providers will not be consented, we will 
hold an information session explaining how to use PERK prior to its 
incorporation into the EHR. Providers will be trained on using the tool through 
sessions delivered during emergency department grand rounds. Recruitment 
and training will be led by Dr. Richardson and a research coordinator. Training 
will last about 30 minutes with three basic components: 1) overview of the 
evidence supporting tool use, 2) hands-on session to practice using the tool and 
3) question and answer session. Additional education sessions will be 
delivered to continue participant engagement over time. 
 
Complete this section if you are obtaining informed consent but you are requesting a 
waiver of the documentation of consent (i.e., verbal consent will be obtained). To 
proceed with a waiver based on these criteria, each subject must be asked whether they 
wish to have documentation linking them to this study.  Only complete subsection 1 OR 
subsection 2.  
 
  SUBSECTION 1 
 
● Explain how the only record linking the subject to the research would be the 
consent document. 

Version Date: 08/18/2021 
 
Page 22 of 28 
 
 
● Explain how the principal risk of this study would be the potential harm resulting 
from a breach in the confidentiality 
● Indicate whether or not subjects will be provided with a written statement 
regarding the research. 
 
 
 
SUBSECTION 2 
●  Describe the possible risks of harm to the subjects involved in this study and 
explain why the study involves no more than minimal risk.   
● Confirm that the research only involves procedure for which consent is not 
normally required outside the research context.  
● Indicate whether or not subjects will be provided with a written statement 
regarding the research. 
 
 
 
. 
 
25. WAIVER OF HIPAA AUTHORIZATION  
 
 
 x N/A       
 
Complete this section if you seek to obtain a full waiver of HIPAA authorization to use 
and/or disclose protected health information.  
● Describe the risks to privacy involved in this study and explain why the study 
involves no more than minimal risk to privacy:   
● Describe your plan to protect identifiers from improper use or disclosure and to 
destroy them at the earliest time. 
●  Indicate why it is not possible to seek subjects’ authorization for use or 
disclosure of PHI. 
● Indicate why it is not possible to conduct this research without use or disclosure 
of the PHI.   
● Indicate if PHI will be disclosed outside NSLIJ Health System, and if so, to whom.   
Note:  PHI disclosed outside NSLIJ Health System, without HIPAA authorization needs 
to be tracked. Please see guidance at www.nslij.com/irb for information about 
tracking disclosures. 
 
 
Complete this section if you seek to obtain a partial waiver of the patient’s 
authorization for screening/recruitment purposes (i.e., the researcher does not have 
access to patient records as s/he is not part of the covered entity) 
Note: Information collected through a partial waiver for recruitment cannot be shared 
or disclosed to any other person or entity. 
● Describe how data will be collected and used:  
● Indicate why you need the PHI (e.g.PHI is required to determine eligibility, 
identifiers are necessary to contact the individual to discuss participation, other) 

Version Date: 08/18/2021 
 
Page 23 of 28 
 
 
● Indicate why  the research cannot practicably be conducted without the partial 
waiver (e.g. no access to medical records or contact information of the targeted 
population, no treating clinician to assist in recruitment of the study population, 
other) 
 
 
 
26.  VULNERABLE POPULATIONS: 
 
Indicate whether you will include any of these vulnerable populations. If indicated, 
submit the appropriate appendix to the IRB for review: 
 
☐  C hildren or viable neonate 
☐  C ognitively impaired 
☐  Pregnant W om en, Fetuses or neonates of uncertain viability or nonviable  
☐  Prisoners 
X☐  N SLIJ Em ployees, residents, fellow s, etc 
☐  poor/uninsured 
☐  Students 
☐  M inorities 
☐  Elderly 
☐  H ealthy C ontrols 
 
If any of these populations are included in the study, describe additional safeguards that 
will be used to protect their rights and welfare. 
 
Employees, residents, fellows, etc.: In order to protect subjects from undue influence in 
joining this study, employees/residents/fellows will not be consented to the study by any 
personnel who hold a supervisory position over them.  Potential participants will be 
informed that their participation (or non-participation) will have no effect on their 
employment at the clinical site/hospital/health system, or future 
employability/opportunities at that site. All data will be kept confidential.  
 
 
27.  MULTI-SITE HUMAN RESEARCH (COORDINATING CENTER) 
 
If this is a multi-site study where you are the lead investigator, describe the management 
of information (e.g. results, new information, unanticipated problems involving risks to 
subjects or others, or protocol modifications) among sites to protect subjects. 
n/a 
 
 
 
28. REFERENCES/BIBIOGRAPHY 
 

Version Date: 08/18/2021 
 
Page 24 of 28 
 
 
Provide a reasonable list of references directly related to the study. Any diagrams for 
new medical devices or brief reprints from journals might also prove useful. 
 
1.  
McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in 
the United States. New England journal of medicine. 2003;348(26):2635-2645. 
2. 
Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ: 
British Medical Journal (Online). 2016;353. 
3. 
Pronovost PJ. Enhancing physicians’ use of clinical guidelines. Jama. 
2013;310(23):2501-2502. 
4. 
Berwick DM, Hackbarth AD. Eliminating waste in US health care. Jama. 
2012;307(14):1513-1516. 
5. 
Donaldson MS, Corrigan JM, Kohn LT. To err is human: building a safer health system. 
Vol 6: National Academies Press; 2000. 
6. 
Baker A. Crossing the quality chasm: a new health system for the 21st century. BMJ: 
British Medical Journal. 2001;323(7322):1192. 
7. 
Asch SM, McGlynn EA, Hogan MM, et al. Comparison of quality of care for patients in 
the Veterans Health Administration and patients in a national sample. Annals of internal 
medicine. 2004;141(12):938-945. 
8. 
Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a 
systematic review. Annals of internal medicine. 2012;157(1):29-43. 
9. 
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and 
clinical decision support systems on medication safety: a systematic review. Archives of 
internal medicine. 2003;163(12):1409-1416. 
10. 
Bonnabry P, Despont-Gros C, Grauser D, et al. A risk analysis method to evaluate the 
impact of a computerized provider order entry system on patient safety. Journal of the 
American Medical Informatics Association. 2008;15(4):453-460. PMCID: PMC2442261 
11. 
Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised 
clinical decision support systems: meta-regression of 162 randomised trials. Bmj. 
2013;346:f657. PMCID: PMC4232126 
12. 
Souza NM, Sebaldt RJ, Mackay JA, et al. Computerized clinical decision support systems 
for primary preventive care: a decision-maker-researcher partnership systematic review 
of effects on process of care and patient outcomes. Implement Sci. 2011;6(1):87-99. 
13. 
McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical 
decision support in primary care practices: a randomized clinical trial. JAMA internal 
medicine. 2013;173(17):1584-1591. 
14. 
Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information 
technology on quality, efficiency, and costs of medical care. Annals of internal medicine. 
2006;144(10):742-752. 
15. 
Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through 
clinical decision support and knowledge management. Evidence report/technology 
assessment. 2012(203):1-784. 
16. 
van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in 
computerized physician order entry. Journal of the American Medical Informatics 
Association : JAMIA. 2006;13(2):138-147. PMCID: PMC1447540 
17.  
Kosters M, Van der Heijden J. From mechanism to virtue: Evaluating Nudge theory. 
Evaluation 2015; 21.3: 276-291. 

Version Date: 08/18/2021 
 
Page 25 of 28 
 
 
18. 
Johnson EJ, Goldstein D. Do defaults save lives? : American Association for the 
Advancement of Science; 2003. 
19. 
Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate 
antibiotic prescribing among primary care practices: a randomized clinical trial. Jama. 
2016;315(6):562-570. 
20.  
Michie S, van Stralen MM, West R. The behaviour change wheel: A new method for 
characterising and designing behaviour change interventions. Implementation Science. 
2011; 6(1):42. PMCID: PMC3096582 
21. 
Richardson S, McCullagh L, Khan S, et al. Clinical Decision Support Decreases 
Variability in CTPA Yield. Paper presented at: SGIM Annual Meeting2017; Washington, 
DC. 
22. 
Durieux P, Trinquart L, Colombet I, et al. Computerized advice on drug dosage to 
improve prescribing practice. Cochrane Database Syst Rev. 2008;3(3). 
23. 
Eslami S, Abu-Hanna A, De Keizer NF. Evaluation of outpatient computerized physician 
medication order entry systems: a systematic review. Journal of the American Medical 
Informatics Association. 2007;14(4):400-406. PMCID: PMC2244893 
24. 
Bennett JW, Glasziou PP. Computerised reminders and feedback in medication 
management: a systematic review of randomised controlled trials. Medical Journal of 
Australia. 2003;178(5):217-222. 
25. 
Moja L, Kwag KH, Lytras T, et al. Effectiveness of computerized decision support 
systems linked to electronic health records: a systematic review and meta-analysis. 
American journal of public health. 2014;104(12):e12-e22. 
26. 
Adler-Milstein J, Jha AK. HITECH Act Drove Large Gains In Hospital Electronic Health 
Record Adoption. Health Affairs. 2017;36(8):1416-1422. 
27. 
Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systemsa 
systematic review. Annals of internal medicine. 2012;157(1):29-43. 
28. 
Van Der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in 
computerized physician order entry. Journal of the American Medical Informatics 
Association. 2006;13(2):138-147. PMCID: PMC1447540 
29. 
Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on 
management of asthma and angina in adults in primary care: cluster randomised 
controlled trial. Bmj. 2002;325(7370):941. 
30.  
Linder JA, Doctor JN, Friedberg MW, et al. Time of day and the decision to prescribe 
antibiotics. JAMA internal medicine. 2014;174(12):2029-2031. PMID: 25286067 
31.  
Linder JA. Antibiotic prescribing for acute respiratory infections—success that's way off 
the mark: comment on “A cluster randomized trial of decision support strategies for 
reducing antibiotic use in acute bronchitis”. JAMA internal medicine. 2013;173(4):273-
275. 
32. 
Linder J, Meeker D, Fox C, et al. Durability of Benefits of Behavioral Interventions on 
Inappropriate Antibiotic Prescribing in Primary Care: Follow-up From a Cluster 
Randomized Clinical Trial. Paper presented at: Open Forum Infectious Diseases2016. 
33.  
Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for 
characterising and designing behaviour change interventions. Implementation science. 
2011;6(1):42. PMID: 21513547 

Version Date: 08/18/2021 
 
Page 26 of 28 
 
 
34. 
Smith-Bindman R, Lipson J, Marcus R, et al. Radiation dose associated with common 
computed tomography examinations and the associated lifetime attributable risk of 
cancer. Archives of internal medicine. 2009;169(22):2078-2086. PMID: 20008690 
35. 
Venkatesh AK, Agha L, Abaluck J, Rothenberg C, Kabrhel C, Raja AS. Trends and 
Variation in the Utilization and Diagnostic Yield of Chest Imaging for Medicare Patients 
With Suspected Pulmonary Embolism in the Emergency Department. AJR. American 
journal of roentgenology. 2018:1-6. PMID: 29364724 
36. 
Brenner DJ, Hall EJ. Computed tomography—an increasing source of radiation exposure. 
New England Journal of Medicine. 2007;357(22):2277-2284. PMID: 18046031 
37. 
Kanzaria HK, Hoffman JR, Probst MA, Caloyeras JP, Berry SH, Brook RH. Emergency 
physician perceptions of medically unnecessary advanced diagnostic imaging. Academic 
Emergency Medicine. 2015;22(4):390-398. PMID: 25807995 
38.  
Feng LB, Pines JM, Yusuf HR, Grosse SD. US trends in computed tomography use and 
diagnoses in emergency department visits by patients with symptoms suggestive of 
pulmonary embolism, 2001–2009. Academic Emergency Medicine. 2013;20(10):1033-
1040. 
39. 
Mitchell AM, Jones AE, Tumlin JA, Kline JA. Prospective Study of the Incidence of 
Contrast‐induced Nephropathy Among Patients Evaluated for Pulmonary Embolism by 
Contrast‐enhanced Computed Tomography. Academic Emergency Medicine. 
2012;19(6):618-625. PMID: 22687176 
40. 
Niemann T, Zbinden I, Roser H, Bremerich J, Remy-Jardin M, Bongartz G. Computed 
tomography for pulmonary embolism: assessment of a 1-year cohort and estimated 
cancer risk associated with diagnostic irradiation. Acta Radiologica. 2013;54(7):778-784. 
PMID: 2376154412 
41. 
Hall WB, Truitt SG, Scheunemann LP, et al. The prevalence of clinically relevant 
incidental findings on chest computed tomographic angiograms ordered to diagnose 
pulmonary embolism. Archives of internal medicine. 2009;169(21):1961-1965. PMID: 
19933956 
42. 
Wang RC, Bent S, Weber E, Neilson J, Smith-Bindman R, Fahimi J. The impact of 
clinical decision rules on computed tomography use and yield for pulmonary embolism: a 
systematic review and meta-analysis. Annals of emergency medicine. 2016;67(6):693-
701. e693. PMID: 26747217

Version Date: 4/2/2020 
 
Page 26 of 28 
 
 
APPENDIX A: Data Safety Monitoring Plan (DSMP) 
 
The principal investigator (PI), mentors, collaborators and research coordinator will be 
responsible for monitoring the scientific integrity and participant safety for the full duration of 
the study. They will meet weekly to discuss the progress of the study. 
 
We will be responsible for maintaining our data and maintaining security. We will use standard 
IRB-approved and HIPAA-compliant measures to maintain confidentiality, privacy and data 
security. Data privacy and security procedures will include: (1) training staff on data sensitivity 
and protocols for safeguarding confidentiality, (2) storing and processing sensitive hardcopy in a 
secured, centralized location, (3) securing sensitive hardcopies in locked files when not in use, 
(4) removing names, addresses, and other direct identifiers from hardcopy and computer-readable 
data when they are no longer necessary for patient tracking and then using encrypted codes for 
subsequent identification of participants, (5) destroying all identifiable linkages to data after data 
accuracy has been verified and final analyses have been completed, (6) capturing and storing 
follow-up assessments in REDCap, a secure web-based HIPAA compliant application designed 
to support data capture for research studies, and (7) using restricted logon identification and 
password protection computer protocols for all computerized entry, retrieval, and analysis. Since 
protected health information (PHI) data will be collected and transmitted to the HIE hosted by 
Northwell Health, we have developed a clear and concise privacy and security plan. Northwell is 
well suited to develop such a plan as we are currently engaging in similar health technology 
projects that secure the electronic submittal of health information from patient to health system. 
The Office of the Chief Informatics Officer (OCIO) and HIE have set in place policies and 
procedures to ensure the security and privacy of the participants’ PHI and will be advisors on our 
grant to ensure the clinical decision support system (CDSS) solution and underlying platform 
also meets Northwell security requirements. 
 
Monitoring the progress of trials and the safety of participants. 
The proposed project is a low-risk study. The risk of minimal fatigue using the tool will be 
monitored and participants may dismiss the tool at any time. The study team will review the 
data on a weekly basis. The work will be conducted with approval from the Program for the 
Protection of Human Subjects (PPHS). 
 
As per Northwell Health (NWH) PPHS policies, the PI is required to notify the PPHS promptly 
of any unanticipated problems involving risks to subjects or others that might occur. The PI 
will monitor the progress of the study and safety of participants on an ongoing basis. The 
procedures of this study, such as regular meetings with research staff, will ensure discussion 
and reporting of all possible outcomes including any, though unlikely, adverse events. If the 
adverse event is due to the study and is unexpected, the PI will draft a safety report and send a 
copy to the PPHS. The PPHS committee will serve as an objective review mechanism. This 
policy/procedure means that any potential conflict of interest inherent in the PI being the sole 
reviewers of serious adverse advents is avoided.   

Version Date: 4/2/2020 
 
Page 27 of 28 
 
 
 
Plans for assuring adherence with requirements regarding the reporting of adverse events (AEs). 
All serious AEs (e.g., medical occurrences resulting in death) that occur during the study 
defined by the given protocol, regardless of the relation to the research, must be reported to the 
PPHS by telephone, e-mail, or fax within 24 hours of the investigator’s awareness of the 
occurrence of the event. The PI will report SAEs to the PPHS and will disseminate information 
to other agencies as necessary. These initial reports are followed by a safety report which is a 
written account of the serious AE determined by a sponsor/investigator to be both related to 
the treatment under investigation and to be unexpected in nature. Serious AEs will be 
summarized annually in the PPHS application for continuation or termination of research. 
 
All expected non-serious AEs that occur at a greater frequency or severity than anticipated and 
all unexpected non-serious AEs will be reported to the PPHS within 15 working days of the 
investigators becoming aware of the event. These AEs are also summarized annually in the 
PPHS application for continuation or termination of the research. 
 
Plans for assuring that any action resulting in a temporary or permanent suspension of 
funded study is reported to the grant program director responsible.  
Tiffany Chapman, Assistant Director, Grants Management Office at The Feinstein Institute at 
NWH, will provide prompt written notification of any action resulting in a temporary or 
permanent suspension of this protocol to the NIH grant program director responsible for the 
grant. 
 
Plans for assuring data accuracy and protocol adherence. 
The PI will follow procedures outlined under Elements 1 and 2 above which serve as part of the 
quality control procedure. The PI will meet with the mentors and research coordinator on a 
regular basis to ensure data accuracy and protocol adherence. Quarterly meetings (either in 
person or via phone) with the entire study team will also ensure protocol adherence. To ensure 
the validity and integrity of study data, the PI will also oversee all data management 
responsibilities. The PI will discuss all data management issues with the study team at NWH.