Electronic Health Record-based 

Interventions for Reducing 

Inappropriate Imaging in the

Clinical Setting: A Systematic 

Review of the Evidence

January 2015

Prepared for:

Department of Veterans Affairs

Veterans Health Administration

Quality Enhancement Research Initiative

Health Services Research & Development Service

Washington, DC 20420

Prepared by:

Evidence-based Synthesis Program (ESP) Center

West Los Angeles VA Medical Center

Los Angeles, CA

Paul G. Shekelle, MD, PhD, Director

Department of Veterans Affairs

Health Services Research & Development Service

Investigators:

Principal Investigator: 

Paul G. Shekelle, MD, PhD

Contributing Investigators: 

Caroline Lubick Goldzweig, MD, MS

Greg Orshansky, MD

Neil M. Paige, MD, MSHS

Brett A. Ewing, MS

Research Associates: 

Isomi M. Miake-Lye, BA

Jessica M. Beroes, BS

Evidence-based Synthesis Program


EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

PREFACE 

Quality Enhancement Research Initiative’s (QUERI) Evidence-based Synthesis Program (ESP) 

was established to provide timely and accurate syntheses of targeted healthcare topics of 

particular importance to Veterans Affairs (VA) clinicians, managers and policymakers as they 

work to improve the health and healthcare of Veterans. The ESP disseminates these reports 

throughout the VA, and some evidence syntheses inform the clinical guidelines of large 

professional organizations. 

QUERI provides funding for four ESP Centers and each Center has an active university 

affiliation. The ESP Centers generate evidence syntheses on important clinical practice topics, 

and these reports help:

• develop clinical policies informed by evidence;

• guide the implementation of effective services to improve patient 

outcomes and to support VA clinical practice guidelines and performance 

measures; and

• set the direction for future research to address gaps in clinical knowledge.

In 2009, the ESP Coordinating Center was created to expand the capacity of HSR&D Central 

Office and the four ESP sites by developing and maintaining program processes. In addition, the 

Center established a Steering Committee comprised of QUERI field-based investigators, VA 

Patient Care Services, Office of Quality and Performance, and Veterans Integrated Service 

Networks (VISN) Clinical Management Officers. The Steering Committee provides program 

oversight, guides strategic planning, coordinates dissemination activities, and develops 

collaborations with VA leadership to identify new ESP topics of importance to Veterans and the 

VA healthcare system.

Comments on this evidence report are welcome and can be sent to Nicole Floyd, ESP 

Coordinating Center Program Manager, at Nicole.Floyd@va.gov. 

Recommended citation: Goldzweig C L, Orshansky G, Paige N M, Ewing B A, Miake-Lye I M, 

Beroes J M, Shekelle P G. Electronic Health Record-Based Interventions for Reducing 

Inappropriate Imaging in the Clinical Setting: A Systematic Review of the Evidence. VA ESP 

Project #05-226; 2014. 

This report is based on research conducted by the Evidence-based Synthesis Program (ESP) Center 

located at the West Los Angeles VA Medical Center, Los Angeles, CA, funded by the Department of 

Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality 

Enhancement Research Initiative. The findings and conclusions in this document are those of the 

author(s) who are responsible for its contents; the findings and conclusions do not necessarily represent 

the views of the Department of Veterans Affairs or the United States government. Therefore, no 

statement in this article should be construed as an official position of the Department of Veterans Affairs. 

No investigators have any affiliations or financial involvement (eg, employment, consultancies, honoraria, 

stock ownership or options, expert testimony, grants or patents received or pending, or royalties) that 

conflict with material presented in the report.

i
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

TABLE OF CONTENTS 

Preface ........................................................................................................................................ i

EXECUTIVE SUMMARY

Introduction .............................................................................................................................1

Methods..................................................................................................................................1

Data Sources and Searches ..................................................................................................1

Study Selection....................................................................................................................1

Data Abstraction and Quality Assessment............................................................................2

Data Synthesis and Analysis................................................................................................2

Results ....................................................................................................................................2

Results of Literature Search.................................................................................................2

Summary of Results for Key Question 1 and 2 ....................................................................3

Summary of Results for Key Question 3..............................................................................4

Discussion...............................................................................................................................4

Key Findings and Quality of Evidence.................................................................................4

Applicability........................................................................................................................5

Research Gaps/Future Research...........................................................................................5

Conclusions.........................................................................................................................5

EVIDENCE REPORT

Introduction ................................................................................................................................7

Methods......................................................................................................................................8

Topic Development.................................................................................................................8

Search Strategy .......................................................................................................................8

Study Selection .......................................................................................................................9

Data Abstraction ...................................................................................................................10

Quality Assessment...............................................................................................................10

Data Synthesis.......................................................................................................................11

Rating the Body of Evidence ................................................................................................. 11

Peer Review ..........................................................................................................................12

Results......................................................................................................................................13

Literature Flow......................................................................................................................13

ii
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Key Question 1: What is the effectiveness of EHR-based interventions in reducing 

unnecessary or inappropriate imaging?..................................................................................18

Key Question 2: Do EHR-based interventions vary in results by system? ..............................18

Effect of Intervention Characteristics, Setting, Implementation, and Target 

on Effectiveness ................................................................................................................21

VA Studies........................................................................................................................26

Summary of Findings and Quality of Evidence for Key Question 1 and 2..........................26

Key Question 3: What are the harms or potential harms associated with EHR-based 

interventions used to reduce inappropriate imaging?..............................................................27

Summary of Findings ........................................................................................................27

Quality of Evidence for Key Question 3 ............................................................................27

Summary and Discussion ..........................................................................................................28

Summary of Evidence by Key Question ................................................................................28

Key Question 1: What is the effectiveness of EHR-based interventions in reducing 

unnecessary or inappropriate imaging? ..............................................................................28

Key Question 2: Do EHR-based interventions vary in results by system? ..........................28

Key Question 3..................................................................................................................28

Limitations............................................................................................................................28

Publication Bias.................................................................................................................28

Study Quality ....................................................................................................................28

Heterogeneity....................................................................................................................29

Applicability of Findings to the VA Population ................................................................. 29

Research Gaps/Future Research ............................................................................................29

Conclusions...........................................................................................................................29

References ................................................................................................................................30

APPENDICES

Appendix A. Evidence Tables...................................................................................................34

Appendix B. Search Strategies..................................................................................................46

Appendix C. Description of Outcomes Used as Measures of Appropriate or Inappropriate Use . 49

Appendix D. Data Abstraction ..................................................................................................50

Appendix E. Peer Review Comments/Author Responses...........................................................51

iii
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

TABLES

Abbreviations Table ....................................................................................................................6 

Table 1. Criteria Used to Assess the Quality of Included Studies ............................................... 10 

Table 2. Articles with Type of Outcome Presented.................................................................... 15 

FIGURES

Figure 1. Literature Flow Chart ................................................................................................. 13 

Figure 2. Studies Reporting Both Appropriateness and Utilization Outcomes............................ 17 

Figure 3. Results for Appropriateness from Individual Studies .................................................. 19 

Figure 4. Results for Utilization from Individual Studies........................................................... 21 

Figure 5. Appropriateness Results Stratified by Intervention Characteristics and by Setting ...... 23 

Figure 6. Appropriateness Results Stratified by Implementation Characteristics and by Target.. 25 

iv
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

EXECUTIVE SUMMARY

INTRODUCTION

There are widespread concerns within Veterans Affairs (VA) healthcare and in non-VA US 

healthcare that the costs of healthcare are rising at unsustainable rates. One driver of cost is the 

increasing use of radiology imaging procedures, particularly advanced imaging techniques such 

as computed tomography (CT) scanning, magnetic resonance imaging (MRI), and CT 

angiography. Most authorities agree that more appropriate use of certain imaging tests could both 

improve quality and save costs. 

The recognition that more appropriate use of imaging could improve quality and reduce costs has 

led to the development of interventions to encourage more appropriate radiology utilization. 

Some of these interventions have made use of the clinical decision support capabilities of 

electronic health records (EHR). VA has been a leader in the use of electronic health records and 

clinical decision support. 

In recognition of the risks and costs associated with inappropriate imaging, VA leadership has 

requested an evidence synthesis which evaluates studied methods for reducing inappropriate 

imaging that center around the electronic health record (EHR). The final key questions are:

Key Question 1: What is the effectiveness of EHR-based interventions in reducing unnecessary 

or inappropriate imaging?

Key Question 2: Do EHR-based interventions vary in results by system (type of EHR 

intervention)? 

Key Question 3: What are the harms or potential harms associated with EHR-based interventions 

used to reduce inappropriate imaging?

METHODS

Data Sources and Searches

We searched the references of existing broad based health information technology (IT) 

systematic reviews covering the period 1995-2011, and also performed a search of PubMed and 

Web of Science from 2011 to 9/10/2014 using terms including “Medical Informatics 

Applications[Mesh],” “Decision Support Systems, Clinical[Mesh],” “medical records systems, 

computerized,” “health information technolog*,” “electronic medical record*,” and “radiology 

department.”

Study Selection

Participants: Adult population. Studies aimed only at children were excluded. Studies with mixed 

populations were included.

Intervention: EHR-based interventions for reducing imaging for diagnostic purposes (as opposed 

to screening) considered inappropriate or unnecessary based on clinical guidelines. This meant 

that studies seeking to increase the use of radiographic imaging like mammography for breast 

1
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

cancer screening were excluded. Studies of systems running on personal digital assistants were 

excluded. Studies of web-based interventions or computerized, stand-alone systems that we 

judged could be easily incorporated into the EHR were included.

Comparator (study design): Usual care.

Outcome: Rates of imaging procedures judged as unnecessary based on existing clinical 

guidelines. Studies that reported on changes in appropriateness (as opposed to decrease in 

appropriateness) were also included. Studies that targeted the use of imaging procedures stated as 

being overused and then reporting only utilization data were included. Utilization outcomes were 

considered separate from appropriateness outcomes. A table of all outcomes included as 

“appropriateness” is in Appendix C. 

Timing: All times

Setting: Ambulatory, hospital, and emergency department settings.

Data Abstraction and Quality Assessment

Data were extracted by 2 reviewers, and discrepancies were reconciled with the group. Articles 

had data abstracted on study design, time period, setting, imaging modality, intervention, 

comparison, sample size, target of intervention, findings, IT design, data entry for intervention, 

and implementation characteristics. We assessed the quality of studies by their design and the

degree to which they reported information about intervention and implementation characteristics. 

Data Synthesis and Analysis

We used as the primary outcome the effect of the intervention on the appropriateness outcome. 

This could have been the increase in appropriate use or the decrease in inappropriate use; studies 

rarely reported both. As a secondary outcome, we used the effect on utilization. Random effects 

meta-analyses were conducted using the H-K variance estimator. After collecting data on the 

interventions, implementations, and settings, but prior to extraction of outcomes data, we 

developed 4 hypotheses regarding effectiveness of the intervention, one in each category of 

intervention characteristics, settings, implementation, and target. 

RESULTS

Results of Literature Search

From all sources, we retrieved 1,195 titles. From these, we identified 172 titles as being 

potentially relevant. After reviewing these 172 abstracts, we identified 105 titles for full text 

review. Of these, we rejected a total of 82 articles, with 10 rejected for their study design (eg, a 

commentary, editorial, review, etc), 27 rejected as being about radiology imaging for screening, 

one rejected for being in a child-aged population, and 39 rejected as not being EHR-based or not 

about clinical decision support (CDS) or not about imaging.

Of the 23 articles included, 3 were randomized trials, 7 were time series studies, and 13 were 

pre-post studies. Seven studies collected data prior to the year 2000, 7 studies included data 

collection within the past 5 years (2009 or later). Ten interventions targeted what was sometimes 

called “high cost imaging,” which usually included CT and MRI and occasionally nuclear 

2
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

medicine tests as well. Four interventions targeted pulmonary CT angiography, 2 studies targeted 

chest x-ray, 4 interventions targeted multiple radiologic investigations, and 3 studies had other 

radiologic targets.

In 5 studies, the intervention consisted of simply the display of information, such as the cost of 

tests, relevant guidelines, or an appropriateness rating for the requested radiology examination 

for that indication. Nine studies displayed patient-specific information about whether or not the 

requested study was consistent with existing guidelines (or something similar). Four studies 

included what we characterized as a “soft stop,” meaning for radiology orders that the CDS rated 

as inconsistent with guidelines or inappropriate, the provider needed to enter a reason why the 

CDS advice was being over-ridden. Five studies included a “hard stop,” meaning providers were 

prevented from ordering radiology examinations the CDS classified as inappropriate without 

getting approval from some external person, usually a radiologist or senior clinician. 

Two studies did not present data sufficient to include in our quantitative analysis, one because it 

did not present comparative data without the intervention and one because the outcome was an 

aggregate measure of many tests, and data specific to the radiology targets were not presented. 

Of the remaining 21 studies, 13 reported an appropriateness outcome and 13 reported a 

utilization outcome; 5 studies reported both. 

Summary of Results for Key Question 1 and 2

Thirteen studies contributed to each pooled analysis, one pooled analysis for appropriateness and 

one pooled analysis for utilization. Four studies contributed data to both. Our primary outcome 

was the effect on appropriateness. Nine of the 13 studies reported statistically significant benefits 

of the intervention, 2 reported a benefit that was not statistically significant, and 2 studies 

reported no effect. The random effects pooled estimate from all 13 studies was an effect size of 

0.48 (95% CI: -0.71, -0.25). This equates to a “moderate” sized effect, according to a 

conventional classification.

Thirteen studies reported utilization outcomes. Six studies reported statistically significant 

benefits of the intervention, and 7 studies reported essentially no effect. The random effects 

pooled estimate from all 13 studies was an effect size of 0.13 (95% CI: -0.23, -0.04). This 

equates to a “small” sized effect, according to a conventional classification.

We explored 4 hypotheses regarding effectiveness, one each for characteristics of the 

intervention, the setting (integrated care delivery versus other settings), the implementation 

process (the use of audit and feedback was the only implementation characteristic with sufficient 

data to support a stratified analysis), and the radiologic target of the intervention. We had 

insufficient studies to support robust pooled estimates of individual strata or to support 

multivariable analyses. Nevertheless, some patterns are apparent.

All of the interventions with a “hard stop” reported moderate-to-large effects on appropriateness. 

Studies using other interventions reported more variable effects or had insufficient numbers to 

draw conclusions. 

The 3 studies conducted in integrated care settings all reported large effects on appropriateness 

and small-to-moderate effects on utilization. Studies conducted at the US institutions that are 

3
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

leaders in health IT, and other settings, produced more mixed results, although all 5 of the 6 

studies of appropriateness at the health IT leaders reported statistically significant benefits of the 

intervention.

There were too few studies using audit-and-feedback to draw conclusions and no apparent 

pattern in studies of interventions at different radiology targets. 

Summary of Results for Key Question 3 

Four studies reported on harms associated with their interventions. One study, evaluating a 

decision support tool to reduce unnecessary pre-operative testing, found that with the 

intervention there was an increase in the percent of pre-operative chest x-rays inappropriately not 

ordered. Prior to the intervention 1.9% of patients did not get a chest x-ray when indicated, 

compared to 9.3% after the intervention. The clinical impact of this is not known. Another study 

of a decision support tool to reduce abdominal kidney, ureter, bladder (KUB) x-rays identified 12 

KUB studies out of a total of 255 performed against the advice of the tool where there were 

positive findings. Of these 12, six KUB studies were felt to have significantly influenced patient 

outcomes, making it unclear whether following the locally developed guidance could have 

endangered the patient. The 2 other studies reported on qualitative information from physician 

surveys which primarily identified lack of interest in using the decision support tools because of 

time constraints and perceived inefficiencies. 

DISCUSSION

Key Findings and Quality of Evidence

Key Question 1 and 2

Summary of Findings and Quality of Evidence

Twenty-one studies provide moderate-quality evidence that EHR-based interventions can reduce 

inappropriate test ordering by a moderate amount, and reduce overall utilization by a small 

amount. Low-quality evidence supports that interventions that include a “hard stop,” preventing 

ordering clinicians from overriding a decision support determination that a test is inappropriate, 

and implementation in an integrated care delivery setting, are associated with greater 

effectiveness. Audit-and-feedback may be a useful implementation tool, but data are too sparse 

to draw conclusions. We judged the quality of evidence regarding appropriateness and utilization 

as moderate, due to heterogeneity in the results. We judged the quality of evidence regarding the 

characteristics as low, due to the sparseness of the data and indirect nature of the comparisons 

That is, these characteristics have not been tested as a priori hypotheses for differential 

effectiveness within the same study. 

Key Question 3

Summary of Findings

There are few data on the potential harms of decision support tools to reduce inappropriate 

radiology test ordering. Future studies should evaluate for harms – particularly investigating 

whether guidelines when applied in practice provide unanticipated results, or when there are 

issues related to workflow, efficiency, or provider dissatisfaction that could impact a decision 

4
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

support tool’s effectiveness. For example, in a study of CDS to prevent drug-drug interactions, 

the use of a “hard stop” intervention – while effective in changing prescribing – resulted in 

delays in treatment for 4 patients, resulting in preventative stopping of the study by the 

Institutional Review Board.1 Another study, excluded from our review because it assessed a 

pediatric population, surveyed physicians and found that most felt the CDS was “a nuisance” and 

“not relevant to the complex or high risk patients they had to treat.”2 This highlights the need for 

assessment of harms and unintended effects in every evaluation. 

Quality of Evidence

We judged the quality of evidence for harms as very low, meaning any estimate is uncertain.

Applicability

Only 3 studies were performed in integrated care delivery settings, and only one study was 

performed in the VA. However, there is evidence suggesting that interventions implemented in 

integrated care delivery settings may be more effective than in other settings, indicating VA may 

realize benefits equal to or greater than the average benefit reported here.

Research Gaps/Future Research

We identified the following research gaps:

Direct comparisons are needed of different intervention characteristics. We found suggestive 

evidence that interventions with a “hard stop” are more effective than other interventions, but to 

prove this hypothesis requires testing the 2 methods head-to-head in the same study. This should 

be easy to do, since randomization can occur at the provider level, and consist of the CDS with 

and without the hard stop.

More research is needed on possible harms. Harms of a CDS system intervention with a “hard 

stop” have been reported in other clinical situations. An explicit assessment of harms should be 

incorporated into every study of interventions.

About half of our included studies collected data more than 7 years ago, and information 

technology and attitudes about the use of information technology change over time, so data from 

more recent time periods would be helpful.

One study reported differential effectiveness by target, and this should be assessed in future 

studies.

Like all health IT evaluations, more information about context and implementation is needed.

Conclusions

Computerized decision support integrated with the electronic health record can reduce 

inappropriate use of diagnostic radiology testing by a moderate amount. The use of a “hard stop” 

as part of the intervention, the use of audit-and-feedback as part of the implementation, and use 

in an integrated care delivery setting may all increase effectiveness. There are few data on the 

potential harms of decision support tools to reduce inappropriate radiology test ordering. Future 

studies should evaluate for harms.

5
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Abbreviations Table

CDS clinical decision support

CT computed tomography

MRI magnetic resonance imaging

EHR electronic health record

IT information technology

KUB kidney, ureter, bladder

ED emergency department

RCT randomized controlled trial

CPOE computerized physician order entry

GRADE Grading of Recommendations Assessment, Development and Evaluation

UGI upper gastrointestinal

VQ ventilation/perfusion lung scan

CXR chest x-ray

GI gastrointestinal

POE physician order entry

PE pulmonary embolism

AXR abdominal x-ray

AP anteroposterior

PA posterioranterior

US ultrasound (only when used in evidence tables)

EKG electrocardiogram

6
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

EVIDENCE REPORT

INTRODUCTION

There are widespread concerns within Veterans Affairs (VA) healthcare and in non-VA US 

healthcare that the costs of healthcare are rising at unsustainable rates. One driver of cost is the 

increasing use of radiology imaging procedures, particularly advanced imaging techniques such 

as computed tomography (CT) scanning, magnetic resonance imaging (MRI), and CT 

angiography. Advances in imaging capabilities allow physicians to image ever-finer areas within 

the body, and the ease with which many of these tests are ordered has led to dramatic increases 

in the rates of use of many tests. For example, the use of CT scans in the Emergency Department 

(ED) grew by 330% in the 12 years from 1996 through 2007, at a time when the rate of ED visits 

grew by only 11%.3 Similarly, other investigators reported a 3-fold increase in the likelihood of 

getting a CT scan or MRI during an ED visit between 1998 and 2007.4

These dramatic increases in utilization have led to increased scrutiny regarding the clinical value 

of these imaging studies. In some cases, strong evidence exists that the imaging studies provide 

no value, or even harm patients. For example, a meta-analysis of early lumbar imaging for 

patients with acute low back pain included 5 randomized controlled trials (RCTs) where patients 

were randomized to receive or not receive early imaging in the form of a plain film, a CT, or an 

MRI. At 3 months, there was no improvement in pain or function among patients who had 

received imaging.

5 In other cases, strong professional opinion considers certain tests to be of 

little value, mainly because alternate tests are preferred or the probability of an abnormal image 

is exceedingly remote. 

There is widespread agreement that more appropriate use of certain imaging tests could both 

improve quality and save costs. When the American Board of Internal Medicine Foundation 

asked physician specialty groups to identify procedures or tests that they judged were of little 

value, imaging tests were frequently identified, such as the use of CT scans for minor head injury 

in the ED (American College of Emergency Physicians), imaging studies in patients with 

nonspecific low back pain (American College of Physicians), imaging for uncomplicated 

headache (American College of Radiology), CT angiography for patients with low clinical

probability of pulmonary embolus and a negative D-dimer assay (American College of Chest 

Physicians), and stress cardiac imaging in patients without high-risk markers for coronary artery 

disease (American College of Cardiology).

The recognition that more appropriate use of imaging could improve quality and reduce costs has 

led to the development of interventions to encourage more appropriate radiology utilization.

Some of these interventions have made use of the clinical decision support capabilities of 

electronic health records. VA has been a leader in the use of electronic health records and 

clinical decision support, and VA leadership therefore requested a review of published studies 

assessing the effect of electronic health record (EHR)-based interventions to improve the 

appropriateness of imaging.

7
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

METHODS

TOPIC DEVELOPMENT

This report was developed based on a nomination from the VHA Choosing Wisely Workgroup, 

including Dr. David Atkins, Director of Health Services Research and Development, Dr. Charles 

Anderson, Chief Consultant for Diagnostic Services, and Sherrill Snuggs, Utilization Officer in 

the Office of Patient Care Services, in an effort to implement some of the segments of the 

“Choosing Wisely” campaign. In 2011, the American Board of Internal Medicine, working with 

Consumer Reports, organized the “Choosing Wisely“ campaign in conjunction with 9 other 

leading medical professional groups, including the American College of Physicians, the leading 

medical organization dealing with adult medical care and highly relevant to Veterans’ care.

6

These organizations were asked to identify 5 tests or treatments that in their professional 

opinions, based on published guidelines, were often used inappropriately. Leading this list were 

several imaging studies including studies such as stress cardiac imaging in asymptomatic 

patients, routine preoperative chest x-rays in asymptomatic patients, and routine imaging in the 

absence of “red flag symptoms” for low back pain.

In recognition of the risks and costs associated with inappropriate imaging, VA leadership has 

requested an evidence synthesis which evaluates studied methods for reducing inappropriate 

imaging that center around the EHR. 

The final key questions are:

Key Question 1: What is the effectiveness of EHR-based interventions in reducing unnecessary 

or inappropriate imaging?

Key Question 2: Do EHR-based interventions vary in results by system?

Key Question 3: What are the harms or potential harms associated with EHR-based interventions 

used to reduce inappropriate imaging?

The PROSPERO registration number is CRD42014009557. 

SEARCH STRATEGY 

Our search strategy had 4 components: 

The primary component was a search of an existing database developed from 4 prior, broadbased reviews of health information technology (IT).7-10 These 4 reviews were done using similar 

search strategies (covering the period 1995-2013) and inclusion/exclusion criteria, and were 

designed to identify all published hypothesis-testing studies of clinical health IT. Hypothesistesting studies included randomized trials, controlled before-and-after studies, time series 

studies, and pre-post studies. These studies were further classified according to the health IT 

functionality, including clinical decision support (CDS), computerized provider order entry, 

patient care reminders, e-prescribing, health information exchange, etcetera. In the most recent 

review, summary data were presented which showed that of 1057 health IT studies in the 

8
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

database, 417 were classified as CDS. These 417 titles and abstracts were searched for studies 

eligible for this review (eg, CDS aimed at reducing inappropriate radiology use). 

The second component of our search was reference mining of 3 potentially relevant systematic 

reviews: 

1. The impact of computerized provider order entry systems on medical imaging services: a 

systematic review.

11

This review included 8 studies classified as ordering of medical imaging examinations by 

the provider.

2. Computerized clinical decision support systems for chronic disease management: a 

decision-maker-researcher partnership systematic review.

12

This review included 36 studies. It also contained a list of criteria for characterizing CDS 

systems that included whether it was integrated with the electronic health record (EHR) 

or computerized physician order entry (CPOE), whether the CDS was automated through 

the EHR, whether it was pilot tested, whether it gave feedback at the time of care, 

whether the CDS suggested procedures, and whether the authors of the article were also 

the developers of the CDS. We adapted this list for our own use.

3. Systematic review of clinical decision support interventions with potential for inpatient 

cost reduction.13

This review identified 3 studies of radiology interventions.

Our third component was a targeted search of PubMed from 2011 to 9/10/2014, looking 

specifically at decision support for imaging, along with 2 searches constructed to identify articles 

related to 2 key references in Web of Science and PubMed (see Appendix B for this search 

strategy). 

Our fourth component was to reference mine all included articles.

STUDY SELECTION

All reference titles and abstracts were screened in duplicate. If either reviewer selected a title or 

abstract, it was included for further review. Full text articles were then reviewed in duplicate, 

with all discrepancies discussed with the group. References were selected based on the following 

inclusion criteria:

Participants: Adult population. Studies aimed only at children were excluded. Studies with mixed 

populations were included.

Intervention: EHR-based interventions for reducing imaging for diagnostic purposes (as opposed 

to screening) considered inappropriate or unnecessary based on clinical guidelines. This meant 

that studies seeking to increase the use of radiographic imaging like mammography for breast 

cancer screening were excluded. Studies of systems running on personal digital assistants were 

9
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

excluded. Studies of web-based interventions or computerized, stand-alone systems that we 

judged could be easily incorporated into the EHR were included.

Comparator (study design): Usual care. 

Outcome: Rates of imaging procedures judged as unnecessary based on existing clinical 

guidelines. Studies that reported on changes in appropriateness (as opposed to decrease in 

appropriateness) were also included. Studies that targeted the use of imaging procedures stated as 

being overused and then reporting only utilization data were included. Utilization outcomes were 

considered separate from appropriateness outcomes. A table of all outcomes included as 

“appropriateness” is in Appendix C. 

Timing: All times. 

Setting: Ambulatory, hospital, and emergency department settings.

DATA ABSTRACTION

Data were extracted by 2 reviewers, and discrepancies were reconciled with the group. Articles 

had data abstracted on study design, time period, setting, imaging modality, intervention, 

comparison, sample size, target of intervention, findings, IT design, data entry for intervention, 

and implementation characteristics. See Appendix D for the full list of data abstracted from each 

article.

QUALITY ASSESSMENT

We assessed the quality of studies by their design and the degree to which they reported 

information about intervention and implementation characteristics (see Table 1).

Table 1. Criteria Used to Assess the Quality of Included Studies

IT Design

Is it integrated with CPOE?

Does it give real time feedback at point of care?

Does the CDS suggest a recommended course of action?

Intervention Classification

Data Entry Source 

Is it automated through EHR (eg, only uses data already being entered for clinical care)?

Does clinical staff enter data specifically for intervention?

Implementation Characteristics

Was it pilot tested or used an iterative process of development/ implementation?

Was there any user training/ clinician education?

Are the authors also the developers and part of the user group for the CDS?

Was there use of audit-and-feedback (or other internal incentive)?

Are there any other implementation components not already discussed?

10
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

DATA SYNTHESIS

We constructed evidence tables showing the study characteristics and results for all included 

studies. We used as the primary outcome the effect of the intervention on the appropriateness 

outcome. As a secondary outcome, we used the effect on the utilization. For studies presenting 

count data or for which a count could be calculated (from a percentage), an odds ratio and 

associated standard error were calculated. For comparability, the log odds ratios and their 

standard errors were converted into Cohen’s d effect sizes.14 For studies presenting means and 

measures of variation, Cohen’s d effect sizes were calculated directly. For each study, we used 

the difference between the pre-intervention period and the post-intervention period, or the 

difference between the time series projection of performance in the absence of the intervention 

compared to actual performance during the intervention, or the difference between providers 

randomized to the intervention or control, as appropriate to the study design and the available 

data. Results were converted to effect sizes for the analysis. Random effects meta-analyses were 

conducted using the H-K variance estimator. 

After collecting data on the interventions, implementations, and settings but prior to extraction 

of outcomes data, we developed 4 hypotheses regarding effectiveness of the intervention, one in 

each category of intervention characteristics, settings, implementation, and target: 

1. Interventions will vary in their effectiveness according to the following rank order: (a) 

interventions that present only information, (b) interventions that include a pop-up or 

reminder that the selected radiographic examination does not meet current guidelines, (c) 

interventions that require an active override for providers to continue to order a 

radiographic examination not supported by guidelines (ie, "soft stop"), or (d) 

interventions that forbid providers from ordering a radiographic examination not 

supported by guidelines unless/until consultation with a peer or expert (ie, "hard stop"). 

2. Interventions will be more effective in settings that are integrated networks of care (eg, 

VA, Kaiser) than in other settings. 

3. Interventions will be more effective if they include other implementation components, 

such as audit and feedback, academic detailing, etcetera.

4. Interventions may vary by the radiographic modality they target.

RATING THE BODY OF EVIDENCE

The evidence was assessed using the Grading of Recommendations Assessment, Development 

and Evaluation (GRADE) criteria, which uses the domains of study design limitations, 

inconsistency, indirectness, and imprecision in results.15 The GRADE Working Group classified 

the quality of evidence across outcomes according to the following criteria:

• High = Further research is very unlikely to change our confidence on the estimate of 

effect.

• Moderate = Further research is likely to have an important impact on our confidence in 

the estimate of effect and may change the estimate.

11
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

• Low = Further research is very likely to have an important impact on our confidence in 

the estimate of effect and is likely to change the estimate.

• Very Low = Any estimate of effect is very uncertain.

PEER REVIEW

A draft version of this report was reviewed by 2 technical experts and 2 members of VA clinical 

leadership. Reviewer comments were addressed and our responses were incorporated into the 

final report. The complete set of comments and responses can be found in Appendix E. 

12
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

RESULTS

LITERATURE FLOW

From all sources, we retrieved 1,195 titles. From these, we identified 172 titles as being 

potentially relevant. After reviewing these 172 abstracts, we identified 105 titles for full text 

review. Of these, we rejected a total of 82 articles, with 10 rejected for their study design (eg, a 

commentary, editorial, review, etc), 27 rejected as being about radiology imaging for screening, 

one rejected for being in a child-aged population, and 39 rejected as not being EHR-based or not 

about CDS or not about imaging (see Figure 1 for literature flow details). 

Figure 1. Literature Flow Chart

* Results from prior systematic reviews (N=152), the existing database (N=226), the update searches 

(N=793), and articles identified during reference mining (N=24) were deduplicated to reach this number.

** Manuscript reference list includes additional references cited for background and methods plus websites 

relevant to key questions. 

Of the 23 articles included, 3 were randomized trials,16-18 7 were time series studies,19-25 and 13

were pre-post studies.26-38 Seven studies collected data prior to the year 2000,16-18,20,21,27,36 and 7 

studies included data collection within the past 5 years (2009 or later).19,22,29-31,33,38 Ten 

interventions targeted what was sometimes called “high cost imaging,” which usually included 

CT and MRI and occasionally nuclear medicine tests as well.16,19,22,24,25,31,33-35,37,38 Four

interventions targeted pulmonary CT angiography,23,28,32,38 Two studies targeted chest x-ray,

17,30

4 interventions targeted multiple radiologic investigations,20,26,29,36 and 3 studies had other 

radiologic targets.18,21,27

Search results: 

1,195 references*

Pulled for full text review: 

105 references

Included studies:

23**

Excluded = 1,023 references

• Not relevant to topic: 1,014

• Duplicates: 9

Included for abstract 

screening: 172 references

Excluded = 67 references

• Not relevant to topic: 67

Excluded = 82 references

• Not research: news brief, editorial, 

commentaries, etc: 10

• Population: 1

• Screening imaging only: 27

• Not CDS, imaging, or EHR-based: 39

• Background: 5

13
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Studies were mostly single institution implementations, and at US academic medical centers. 

Thirteen studies, or just over half of all studies, were from the “HIT leaders,” meaning US 

academic centers who have a long history of development, implementation, evaluation, and 

publication of health IT.16-18,22-25,31,33,36-38 Ten of these studies (35%) came from institutions 

affiliated with Harvard.16,18,22-25,31,33,37,38 Three studies came from integrated healthcare delivery 

organizations, in particular Kaiser,21 VA,32 and Virginia Mason.19 

In 5 studies, the intervention consisted of simply the display of information, such as the cost of 

tests, relevant guidelines, or an appropriateness rating for the requested radiology examination 

for that indication.16,17,21,29,36 Nine studies displayed patient-specific information about whether 

or not the requested study was consistent with existing guidelines (or something 

similar).18,23,25,27,28,30,31,35,38 Four studies included what we characterized as a “soft stop,” 

meaning for radiology orders that the CDS rated as inconsistent with guidelines or inappropriate, 

the provider needed to enter a reason why the CDS advice was being over-ridden.20,24,26,34 Five 

studies included a “hard stop,” meaning providers were prevented from ordering radiology 

examinations the CDS classified as inappropriate without getting approval from some external 

person, usually a radiologist or senior clinician.19,22,32,33,37

Almost all interventions were integrated with CPOE, gave real-time feedback, and a 

recommended course of action. Only one study specifically indicated that the intervention was 

developed iteratively or pilot tested,18 about one-third of studies reported clinician training was 

part of the implementation process,19,23,26-28 5 studies reported including audit-and-feedback as 

part of the implementation,19,22,24,26,33 and 7 studies reported on other implementation 

characteristics, barriers, and facilitators (including having a culture of evidence-based medicine, 

a phased implementation, salaried physicians, and a risk contract with the payor of 

care).19,22,25,26,28,33,35 The Evidence Tables present details of all 23 included studies (see Appendix 

A).

Two studies did not present data sufficient to include in our quantitative analysis, one because it 

did not present comparative data without the intervention26 and one because the outcome was an 

aggregate measure of many tests, and data specific to the radiology targets were not presented.36

Of the remaining 21 studies, 13 reported an appropriateness outcome and 13 reported a 

utilization outcome, while 5 studies reported both. See Table 2 for which outcomes were 

reported in which studies. 

14
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Table 2. Articles with Type of Outcome Presented

Author, Year Appropriateness Utilization

Bates, 199716 X

Blackmore, 201119 X

Carton, 200220 X

Chin, 199921 X X

Day, 199527 X X

Dresher, 201128 X

Durand, 201329 X

Flamm, 201330 X

Gupta, 201431 X

Harpole, 199718 X

Ip, 201433 X X

Ip, 201322 X

Raja, 201223 X X

Raja, 201438 X X

Rosenthal, 200624 X

Sanders, 200134 X

Sistrom, 200925 X

Solberg, 201035 X

Soo Hoo, 201132 X

Tierney, 198817 X

Vartanians, 201037 X

15
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 2 presents the results from the 5 studies reporting both appropriateness and utilization 

outcomes. While the 2 outcomes are directionally consistent within study, in 3 studies the effect 

size for appropriateness was much greater than for utilization. In general, this is to be expected. 

Most radiology examinations are not ordered for inappropriate reasons, and thus an intervention 

targeted at reducing inappropriate test ordering will have a larger effect on appropriateness than 

it will on utilization. For example, if a particular radiological procedure is ordered appropriately 

70% of the time and inappropriately 30% of the time among 100 consecutive orders, then an 

intervention that successfully reduces inappropriate use by 50% will result in only 15 

inappropriate test orders, while utilization will decrease only 15% (from 100 test orders to 85 test 

orders, the latter consisting of 70 appropriate orders and 15 inappropriate orders). Thus, we 

expect that utilization will be less affected than appropriateness for these interventions. The 

study by Raja and colleagues published in 2014 reported both at an appropriateness outcome, as 

in the proportion of radiologic tests ordered that met a national standard and the patient positive 

yield of radiologic examinations. We included both here for comparison. 

16
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 2. Studies Reporting Both Appropriateness and Utilization Outcomes

ES=effect size, CI=confidence interval

17
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

KEY QUESTION 1: What is the effectiveness of EHR-based 

interventions in reducing unnecessary or inappropriate imaging?

KEY QUESTION 2: Do EHR-based interventions vary in results by 

system?

We combine our presentation of results for these 2 key questions, since they are interrelated.

Thirteen studies contributed to each pooled analysis, one pooled analysis for appropriateness and

one pooled analysis for utilization. Five studies contributed data to both. Our primary outcome 

was the effect on appropriateness. 

Figure 3 displays the results for individual studies reporting appropriateness. Nine of the 13

studies reported statistically significant benefits of the intervention, 2 reported a benefit that was 

not statistically significant, and 2 studies reported no effect. The random effects pooled estimate 

from all 13 studies was an effect size of 0.48 (95% CI: -0.71, -0.25). This equates to a 

“moderate” sized effect, according to a conventional classification.39 Substantial heterogeneity is 

present, as indicated by the I2 statistic of 99.5%, and the visual inspection of the plot. Neither 

Begg’s nor Egger’s test indicated the presence of publication bias (p=0.951, p=0.339, 

respectively). A sensitivity analysis that included the appropriateness outcome instead of the 

yield outcome for Raja (2014) yielded little difference in the pooled estimate (0.52, 95% CI -

0.73, -0.31).38

As a clinical example of what constitutes a moderate-sized effect, consider the results of the 

study by Gupta and colleagues, which reported an effect size of 0.67 associated with 

implementation of the CDS intervention. In clinical terms, this meant that before CDS

implementation the percentage of head CT examinations ordered for appropriate reasons in the 

ER for patients with mild traumatic brain injury was 49%. After implementation of the CDS, this 

rate increased to 76.5%.

31 As a second example, the study by Rosenthal and colleagues reported 

an effect size of 0.63 associated with implementation of a CDS, giving appropriateness scores for 

a number of radiologic procedures. Before the intervention, 6% of procedures were judged as 

being of low utility; after the intervention this value dropped to 2%.24

18
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 3. Results for Appropriateness from Individual Studies

19
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 4 displays the results for the 13 studies reporting utilization outcomes. Six studies 

reported statistically significant benefits of the intervention, and 7 studies reported essentially no 

effect. The random effects pooled estimate from all 13 studies was an effect size of 0.13 (95% 

CI: -0.23, -0.04). This equates to a “small” sized effect, according to a conventional 

classification. Despite a narrower range of effect sizes in these 13 studies (10 of the 13 studies 

cluster between zero and an effect size of 0.21) compared to the range of effect sizes for the 

appropriateness outcomes, like the appropriateness studies substantial heterogeneity is present, as 

indicated by the I2 statistic of 100%. Neither Begg’s nor Egger’s test indicated the presence of 

publication bias (p=0.392, p=0.259, respectively).

20
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 4. Results for Utilization from Individual Studies

Effect of Intervention Characteristics, Setting, Implementation, and Target on 

Effectiveness

We explored 4 hypotheses regarding effectiveness, one each for characteristics of the 

intervention, the setting (integrated care delivery versus other settings), the implementation 

process (the use of audit and feedback was the only implementation characteristic with sufficient 

data to support a stratified analysis), and the radiologic target of the intervention. We had 

insufficient studies to support robust pooled estimates of individual strata or to support 

multivariable analyses. Furthermore, since appropriateness was our primary outcome we present 

stratified results only for this outcome. Nevertheless, some patterns are apparent.

Figure 5 presents the results of the appropriateness outcomes, stratified by intervention 

characteristics and settings. “A” interventions provided information only; in the included study it 

presented internally developed guidelines any time an upper GI series order was placed.21 “B” 

21
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

interventions presented information on appropriateness or guidelines specifically tailored to the 

individual patient, often as a pop-up or alert. Some of these interventions also recommended 

alternative interventions, but did not include any barrier for the clinician to order the test. “C” 

interventions in general were similar to “B” interventions, but required the ordering clinician to 

justify with free text why they were overriding the decision support recommendation that a study 

was inappropriate. We called this a “soft stop.” “D” interventions included what we called a 

“hard stop,” meaning the intervention prevented the clinician from ordering a test contrary to the 

CDS determination of inappropriateness, until additional discussion with or permission obtained 

from another clinician or radiologist. 

All of the “D” studies reported moderate-to-large effects on appropriateness, while the “B” and 

the “C” studies reported more variable and generally lesser effects. The one “A” study, by Chin 

and colleagues,21 reported a large effect. This study displayed locally developed guidelines for 

appropriate upper gastro intestinal (UGI) ordering at the time an order was placed. This study 

was conducted in an integrated care delivery setting with a high baseline rate of inappropriate 

use (45%), which may partly explain these strikingly successful findings. 

Regarding setting, the 2 studies conducted in integrated care settings all reported large effects on 

appropriateness. Studies conducted in other settings found much smaller effects, while studies at

10 US institutions that are leaders in health IT reported heterogeneous results, although 4 of the 6 

studies of appropriateness at the health IT leaders reported statistically significant benefits of the 

intervention.

22
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 5. Appropriateness Results Stratified by Intervention Characteristics and by Setting

23
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 6 presents the results of the appropriateness, respectively, stratified by whether or not the 

publication reported that audit-and-feedback was part of the implementation process and by the 

target of the intervention. There did not seem to be clear patterns, although both studies using 

audit-and-feedback reported moderate-to-large effects. In the study by Rosenthal and colleagues, 

“physicians, who [ordered] more than a few examinations with low utility scores [were] 

contacted by one of their senior clinical leaders and counseled about appropriate ordering.”24 In 

the study by Ip and colleagues, audit-and-feedback was accomplished by sending to primary care 

providers quarterly practice pattern variation reports depicting the number of back MRIs ordered 

per the number of low back pain related visits.33 

No pattern of differential effectiveness is as clearly apparent as in the preceding figures, although 

in a study of one intervention a statistically significant effect was seen for CT scans and nuclear 

radiology tests but not MRI,22 suggesting the possibility of a differential effect of interventions 

based on their target.

24
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Figure 6. Appropriateness Results Stratified by Implementation Characteristics and by 

Target

25
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

Two studies which otherwise met our inclusion criteria could not be included in the pooled 

analysis because they did not present sufficient data. In one study a rural family medicine clinic 

in Canada made computerized decision support available to any interested physician. The 

majority of physicians were infrequent users of the system, which was voluntary. Among 904 

diagnostic imaging studies ordered using the system, clinical guidelines applied to 58%. Of 

these, 29% were identified as inappropriate and an alternative diagnostic strategy suggested. Of 

these, physicians followed the suggestion in 25% of cases. This study could not be included in 

our pooled analysis because it did not present data from the pre-intervention period.26 The 

second study was a time series study of displaying the costs for a large number of outpatient 

diagnostic tests (urinalysis, complete blood count, serum electrolytes, etc).36 Three radiologic 

tests relevant to this review were included: chest x-ray, head CT scan, and head MRI. This study 

could not be included in our pooled analysis because it did not report results separately for the 

individual tests. The effect of the intervention on the aggregate of all these diagnostic tests was a 

statistically significant 14% reduction in utilization. This significant finding is in contrast to 3 

other studies of an intervention that presented cost data, all of which reported no effect on 

utilization.

VA Studies

We identified one study conducted in a VA setting.

32 This was a single site, pre-post study at an 

urban, academically affiliated VA. Locally developed guidelines were developed for the 

appropriate use of CT angiography for suspected pulonary embolus. These were then 

incorporated in the CPOE fuction of the VA electronic health record (CPRS version 1.0.27). The 

order entry menu screen included the Wells criteria point score. The ordering physician was 

required to complete the checklist to generate a Wells score. If the score was 4 or less, the 

physician was required to get a D-dimer test. If the D-dimer test was above a certain threshold, 

the order was allowed to proceed without any additional barriers. If the D-dimer was below that 

threshold, the ordering physician was required to consult with the on-call chest radiology 

attending for approval (a “hard stop”). In the 2 years prior to the intevention, the yield of positive 

CT angiography studies was 3.1%. This increased to 16.5% after inplementation of the 

intervention. There was no assessment of possible patient harms or physician reactions to the use 

of the intervention. 

Summary of Findings and Quality of Evidence for Key Question 1 and 2 

Twenty-one studies provide moderate quality evidence that EHR-based interventions can reduce 

inappropriate test ordering by a moderate amount, and reduce overall utilization by a small 

amount. Low quality evidence supports that interventions that include a “hard stop,” preventing 

ordering clinicians from overriding a decision support determination that a test is inappropriate, 

and implementation in an integrated care delivery setting, are associated with greater 

effectiveness. Audit-and-feedback may be a useful implementation tool, but data are too sparse 

to draw conclusions. We judged the quality of evidence regarding appropriateness and utilization

as moderate, due to heterogeneity in the results. We judged the quality of evidence regarding the 

characteristics as low, due to the sparseness of the data and indirect nature of the comparisons. 

That is, these characteristics have not been tested as a priori hypotheses for differential 

effectiveness within the same study. 

26
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

KEY QUESTION 3: What are the harms or potential harms associated 

with EHR-based interventions used to reduce inappropriate imaging?

Four studies reported on harms associated with their interventions.18,26,28,30 One study evaluating 

a decision support tool to reduce unnecessary pre-operative testing found that with the 

intervention there was an increase in the percent of pre-operative chest x-rays inappropriately not 

ordered.30 Prior to the intervention 1.9% of patients did not get a chest x-ray when indicated 

compared to 9.3% after the intervention. The clinical impact of this is not known. Another study 

of a decision support tool to reduce abdominal kidney, ureter, bladder (KUB) x-rays identified 12 

KUB studies out of a total of 255 performed against the advice of the tool where there were 

positive findings.18 Of these 12, six KUB studies were felt to have significantly influenced 

patient outcomes, making it unclear whether following the locally developed guidance could 

have endangered the patient. The 2 other studies reported on qualitative information from 

physician surveys which primarily identified lack of interest in using the decision support tools 

because of time constraints and perceived inefficiencies.26,28

Summary of Findings

There are few data on the potential harms of decision support tools to reduce inappropriate 

radiology test ordering. Future studies should evaluate for harms – particularly investigating 

whether guidelines when applied in practice provide unanticipated results, or when there are 

issues related to workflow, efficiency, or provider dissatisfaction that could impact a decision 

support tool’s effectiveness. For example in a study of CDS to prevent drug-drug interactions,

the use of a “hard stop” intervention – while effective in changing prescribing – resulted in 

delays in treatment for 4 patients, resulting in preventative stopping of the study by the 

Institutional Review Board.1 Another study, excluded from our review because it assessed a 

pediatric population, surveyed physicians and found that most felt the CDS was “a nuisance” and 

“not relevant to the complex or high risk patients they had to treat.”2 This highlights the need for 

assessment of harms and unintended effects in every evaluation.

Quality of Evidence for Key Question 3 

We judged the quality of evidence for harms as very low, meaning any estimate is uncertain.

27
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

SUMMARY AND DISCUSSION

SUMMARY OF EVIDENCE BY KEY QUESTION

Key Question 1: What is the effectiveness of EHR-based interventions in reducing 

unnecessary or inappropriate imaging?

Key Question 2: Do EHR-based interventions vary in results by system?

Twenty-one studies provide moderate quality evidence that EHR-based interventions can reduce 

inappropriate test ordering by a moderate amount, and reduce overall utilization by a small 

amount. Low-quality evidence supports that interventions that include a “hard stop,” preventing 

ordering clinicians from overriding a decision support determination that a test is inappropriate, 

the use of audit-and-feedback as part of the implementation, and that are conducted in an 

integrated care delivery setting, are associated with greater effectiveness. 

Key Question 3

There are few data on the potential harms of decision support tools to reduce inappropriate 

radiology test ordering. Future studies should evaluate for harms – particularly investigating 

whether guidelines when applied in practice provide unanticipated results, or when there are 

issues related to workflow, efficiency, or provider dissatisfaction that could impact a decision 

support tool’s effectiveness. For example, in a study of CDS to prevent drug-drug interactions, 

the use of a “hard stop” intervention – while effective in changing prescribing – resulted in 

delays in treatment for 4 patients, resulting in preventative stopping of the study by the 

Institutional Review Board.1 Another study, excluded from our review because it assessed a 

pediatric population, surveyed physicians and found that most felt the CDS was “a nuisance” and 

“not relevant to the complex or high risk patients they had to treat.”2 This highlights the need for 

assessment of harms and unintended effects in every evaluation.

LIMITATIONS

Publication Bias

The most important limitation to this review is the likely existence of publication bias. Although 

we did not detect any statistical evidence of publication bias, there surely must be more 

implementations of EHR-based interventions to improve appropriateness of radiology test 

ordering then the 22 published studies we found. Our expectation is that many such interventions 

are done and never formally evaluated or published. How the results of these implementations 

may differ from the published studies is unknown, but we expect both effective and ineffective 

implementations have likely occurred and not been published. This lack of publication is a major 

impediment toward more rapid learning of how health IT can best be implemented. 

Study Quality

In common with many other areas of health IT, we found key information on context and 

implementation to be lacking in published studies. For example, only one study reported on pilot

testing of the intervention, and only about one-third of studies reported on clinician training as 

part of the implementation. This may perpetuate the belief that these kinds of health IT 

interventions can be developed separate from the workflow of practicing clinicians, and then

28
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

simply “turned on” with the expectation that clinicians will know how to use the intervention and 

use it correctly. The dearth of reporting of possible harms is another key limitation. Every 

intervention has to be expected to potentially cause harms and these need to be explicitly 

measured. 

Heterogeneity

Heterogeneity in effectiveness is a prominent finding of our review. Our stratified analyses 

explained some of this heterogeneity, but the greater portion remains unexplained. It has been 

postulated that the majority of heterogeneity in health IT evaluations is due to details of the 

context and implementation that go unreported in published study. We expect the same to be true 

here. Heterogeneity is a primary contributor to low quality of evidence assessments.

Applicability of Findings to the VA Population

Only 3 studies were performed in integrated care delivery settings, and only one study was 

performed in VA. However, there is evidence suggesting that interventions implemented in 

integrated care delivery settings may be more effective than in other settings, indicating VA may 

realize benefits equal to or greater than the average benefit reported here. 

RESEARCH GAPS/FUTURE RESEARCH

We identified the following research gaps:

Direct comparisons are needed of different intervention characteristics. We found suggestive 

evidence that interventions with a “hard stop” are more effective than other interventions, but to 

prove this hypothesis requires testing the 2 methods head-to-head in the same study. This should 

be easy to do, since randomization can occur at the provider level, and consist of the CDS with

and without the hard stop.

More research is needed on possible harms. Harms of a CDS intervention with a “hard stop” 

have been reported in other clinical situations. An explicit assessment of harms should be 

incorporated into every study of interventions.

About half of our included studies collected data more than 7 years ago, and information 

technology and attitudes about the use of information technology change over time, so data from 

more recent time periods would be helpful.

One study reported differential effectiveness by target, and this should be assessed in future 

studies.

Like all health IT evaluations, more information about context and implementation is needed.

CONCLUSIONS

Computerized decision support integrated with the electronic health record can reduce 

inappropriate use of diagnostic radiology testing by a moderate amount. The use of a “hard stop” 

as part of the intervention and use in an integrated care delivery setting may all increase 

effectiveness. There are few data on the potential harms of decision support tools to reduce 

inappropriate radiology test ordering. Future studies should evaluate for harms.

29
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

REFERENCES

1. Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician 

order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled 

trial. Archives of Internal Medicine. 2010;170(17):1578-1583. 

2. Bowen S, Johnson K, Reed MH, Zhang LP, Curry L. The Effect Of Incorporating 

Guidelines Into a Computerized Order Entry System for Diagnostic Imaging. Journal of 

the American College of Radiology. Apr 2011;8(4):251-258. 

3. Kocher KE, Meurer WJ, Fazel R, Scott PA, Krumholz HM, Nallamothu BK. National 

trends in use of computed tomography in the emergency department. Ann. Emerg. Med. 

2011;58(5):452-462. e453.

4. Korley FK, Pham JC, Kirsch TD. Use of advanced radiology during visits to US 

emergency departments for injury-related conditions, 1998-2007. JAMA. 

2010;304(13):1465-1471. 

5. Chou R, Fu R, Carrino JA, Deyo RA. Imaging strategies for low-back pain: systematic 

review and meta-analysis. The Lancet. 2009;373(9662):463-472. 

6. Cassel CK, Guest JA. Choosing wisely: helping physicians and patients make smart 

decisions about their care. JAMA. 2012;307(17):1801-1802. 

7. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information 

technology on quality, efficiency, and costs of medical care. Ann Intern Med. May 16 

2006;144(10):742-752. 

8. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information 

technology: a review of the recent literature shows predominantly positive results. Health 

Affairs. 2011;30(3):464-471. 

9. Goldzweig C, Orshansky G, Paige N, et al. Electronic patient portals: evidence on health 

outcomes, satisfaction, efficiency, and attitudes: a systematic review. Annals of Internal 

Medicine. 2013;159(10):677. 

10. Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated 

systematic review with a focus on meaningful use. Annals of Internal Medicine. 

2014;160(1):48-54-54. 

11. Georgiou A, Prgomet M, Markewycz A, Adams E, Westbrook JI. The impact of 

computerized provider order entry systems on medical-imaging services: a systematic 

review. Journal of the American Medical Informatics Association. 2011;18(3):335-340. 

12. Roshanov PS, Misra S, Gerstein HC, et al. Computerized clinical decision support 

systems for chronic disease management: a decision-maker-researcher partnership 

systematic review. Implement Sci. 2011;6(1):92. 

30
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

13. Fillmore CL, Bray BE, Kawamoto K. Systematic review of clinical decision support 

interventions with potential for inpatient cost reduction. BMC Med. Inform. Decis. Mak. 

Dec 2013;13.

14. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Chapter 7. Introduction to metaanalysis: John Wiley & Sons; 2011.

15. Grading of Recommendations Assessment DaEGWG. GRADE working group. 2013; 

http://www.gradeworkinggroup.org/index.htm. Accessed 9/5/2014.

16. Bates DW, Kuperman GJ, Jha A, et al. Does the computerized display of charges affect 

inpatient ancillary test utilization? Arch Intern Med. Nov 24 1997;157(21):2501-2508. 

17. Tierney WM, McDonald CJ, Hui SL, Martin DK. Computer predictions of abnormal test 

results: effects on outpatient testing. JAMA. 1988;259(8):1194-1198. 

18. Harpole LH, Khorasani R, Fiskio J, Kuperman GJ, Bates DW. Automated evidencebased critiquing of orders for abdominal radiographs: impact on utilization and 

appropriateness. J Am Med Inform Assoc. Nov-Dec 1997;4(6):511-521. 

19. Blackmore CC, Mecklenburg RS, Kaplan GS. Effectiveness of clinical decision support 

in controlling inappropriate imaging. J Am Coll Radiol. Jan 2011;8(1):19-25. 

20. Carton M, Auvert B, Guerini H, et al. Assessment of radiological referral practice and 

effect of computer-based guidelines on radiological requests in two emergency 

departments. Clinical radiology. 2002;57(2):123-128. 

21. Chin HL, Wallace P. Embedding guidelines into direct physician order entry: simple 

methods, powerful results. Paper presented at: Proceedings of the AMIA 

Symposium1999. 

22. Ip IK, Schneider L, Seltzer S, et al. Impact of provider-led, technology-enabled radiology 

management program on imaging. Am J Med. Aug 2013;126(8):687-692. 

23. Raja AS, Ip IK, Prevedello LM, et al. Effect of computerized clinical decision support on 

the use and yield of CT pulmonary angiography in the emergency department. Radiology. 

Feb 2012;262(2):468-474. 

24. Rosenthal DI, Weilburg JB, Schultz T, et al. Radiology order entry with decision support: 

initial clinical experience. Journal of the American College of Radiology. 

2006;3(10):799-806. 

25. Sistrom CL, Dang PA, Weilburg JB, Dreyer KJ, Rosenthal DI, Thrall JH. Effect of 

Computerized Order Entry with Integrated Decision Support on the Growth of Outpatient 

Procedure Volumes: Seven-year Time Series Analysis 1. Radiology. 2009;251(1):147-

155. 

31
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

26. Curry L, Reed MH. Electronic decision support for diagnostic imaging in a primary care 

setting. Journal of the American Medical Informatics Association. May 2011;18(3):267-

270. 

27. Day F, Hoang LP, Ouk S, Nagda S, Schriger DL. The impact of a guideline-driven 

computer charting system on the emergency care of patients with acute low back pain. 

Proc Annu Symp Comput Appl Med Care. 1995:576-580. 

28. Drescher FS, Chandrika S, Weir ID, et al. Effectiveness and acceptability of a 

computerized decision support system using modified Wells criteria for evaluation of 

suspected pulmonary embolism. Ann. Emerg. Med. 2011;57(6):613-621. 

29. Durand DJ, Feldman LS, Lewin JS, Brotman DJ. Provider Cost Transparency Alone Has 

No Impact on Inpatient Imaging Utilization. Journal of the American College of 

Radiology. Feb 2013;10(2):108-113. 

30. Flamm M, Fritsch G, Hysek M, et al. Quality improvement in preoperative assessment by 

implementation of an electronic decision support tool. J Am Med Inform Assoc. Jun 

2013;20(e1):e91-96. 

31. Gupta A, Ip IK, Raja AS, Andruchow JE, Sodickson A, Khorasani R. Effect of clinical 

decision support on documented guideline adherence for head CT in emergency 

department patients with mild traumatic brain injury. J Am Med Inform Assoc. Feb 17 

2014. 

32. Hoo GW, Wu CC, Vazirani S, Li Z, Barack BM. Does a clinical decision rule using Ddimer level improve the yield of pulmonary CT angiography? AJR Am J Roentgenol. 

May 2011;196(5):1059-1064. 

33. Ip IK, Gershanik EF, Schneider LI, et al. Impact of IT-Enabled Intervention on MRI Use 

for Back Pain. Am J Med. Feb 7 2014.

34. Sanders DL, Miller RA. The effects on clinician ordering patterns of a computerized 

decision support system for neuroradiology imaging studies. Proc AMIA Symp. 

2001:583-587. 

35. Solberg LI, Wei F, Butler JC, Palattao KJ, Vinz CA, Marshall MA. Effects of electronic 

decision support on high-tech diagnostic imaging orders and patients. Am J Manag Care. 

Feb 2010;16(2):102-106. 

36. Tierney WM, Miller ME, McDonald CJ. The effect on test ordering of informing 

physicians of the charges for outpatient diagnostic tests. N Engl J Med. May 24 

1990;322(21):1499-1504. 

37. Vartanians VM, Sistrom CL, Weilburg JB, Rosenthal DI, Thrall JH. Increasing the 

Appropriateness of Outpatient Imaging: Effects of a Barrier to Ordering Low-Yield 

Examinations 1. Radiology. 2010;255(3):842-849. 

32
EHR Interventions for Reducing Inappropriate Imaging Evidence-based Synthesis Program

 

38. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to 

measure documented adherence to a national imaging quality measure. Acad Radiol. Mar 

2014;21(3):378-383. 

39. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: 

Lawrence Erlbaum Assoc, Inc; 1988.

33
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

APPENDIX A. Evidence Tables

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Bates, 

199716

RCT; AprilOct 1994; also 

evaluated 

historical data

35 most 

commonly 

ordered studies 

including x-rays, 

US, CTs, VQ 

scan and MRI 

brain/lumbar 

spine

Charges for the 35 most 

common radiological 

tests were displayed at 

the time of electronic 

provider order entry 

(POE)

POE without 

display of 

charge data

8728 

Intervention 

patients; 

8653 Control 

patients 

Assess whether 

knowledge of 

cost (charges) 

can affect 

physician 

ordering 

behavior

Computerized display of cost 

information did not impact the 

number of radiological tests 

ordered and performed 

(mean=$276 vs $275, p=0.88)

Blackmore, 

201119

Time Series; 

Jan 2003- Dec 

2009 

Lumbar MRI, 

head MRI, head 

CT, sinus CT

Clinical decision 

support at the time of 

POE for 3 radiology 

procedures

Historical 

control, 

utilization rates 

from before and 

during the 

intervention

49,967 Decision support 

to lower 

utilization of 

lumbar MRI, 

head MRI, and 

sinus CT

Targeted use of decision 

support can decrease 

inappropriate utilization.

Carton, 

200220

Time Series; 

June-Nov 

1998 

Radiology 

studies ordered

in 2 EDs

Providers ordering 

radiology tests selected 

from a list of clinical 

contexts related to the 

exam and were alerted 

if it did not conform to 

guidelines. 

Notification 

that request did 

not meet 

guidelines 

6869 

radiology 

exam 

requests

Reduce 

unnecessary 

medical imaging

The display of 

recommendations reduced 

non-guideline adherent 

requests by 20% relative at 

hospital A (p=0.02) and 23% 

at hospital B (p=0.0001). 

[19.4% to 15.6% & 39.9% to 

(-90% 30.0%, respectively)] 

The 3 most commonly 

requested tests not 

conforming to guidelines were 

abdominal plain radiographs

(76%), CXR (25%), and head 

CT (16%).

Chin, 199921 Time series, 

descriptive 

quantitative; 

1995 

Upper GI studies 

chest x-rays (but 

no data given to 

CXR)

Guidelines are 

displayed within 

electronic order at time 

of order entry.

Web-based 

guideline 

publication

Not provided Increase 

guidelineadherent 

ordering 

UGI ordering which 

conformed to guidelines 

improved from 55% to 88% 

(once guidelines introduced at 

POE). Also decrease in orders 

from 10.6 per 1000 members 

to 5.6/1000. CXRs decreased 

by 20%.

34
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Curry, 

201126

Pre-post; dates 

not stated

radiology studies Decision support 

during POE

None 904 orders clinical guideline 

acceptance 

through decision 

support during 

POE

Physicians supported the 

concept of decision support 

but were reluctant to change

Day, 199527 Pre-post; Pre: 

May-Nov 

1992, Post: 

May-Dec 1993

Lumbosacral xrays for back 

pain

Emergency department 

charting system with 

guideline-based care 

recommendation

Usual care 

without 

computerized 

order entry 

Stand alone

103 patients 

in pre-period, 

259 patients 

in 

intervention 

period (79%

were treated 

using the 

CPOE/CDS

intervention)

Improve 

appropriateness 

of care for low 

back pain and 

reduce costs.

There was no difference in the 

appropriateness of testing or 

cost-effectiveness of care.

Drescher, 

201128

Pre-post; dates 

not stated

CT angiography Decision support 

calculating a Wells 

score for each order of 

CT angiography during 

POE

historical preinterventionpre: 205; 

post: 229

Increase the 

positive rate of 

CT angiogram 

results

Decision support during POE 

lead to higher positive CT 

angiogram results for PE. CT 

angiogram positive rate 

increased from 8.3% pre to 

12.7% post (difference -4.4%, 

95% CI: -1.4, 10.1)

Durand, 

201329

Pre-post; 2008 

- May 2010

10 most 

frequently 

ordered imaging 

tests - AP CXR, 

AXR, CT head, 

Renal US, 

Vascular US 

(intervention); 

Extremity US, 

PA/Lat CXR, 

CT abdomen 

with contrast, 

Abdominal US, 

CT chest with 

contrast (control)

Present providers with 

cost information for 5 

imaging studies

No cost 

information for 

5 different 

imaging studies

#tests ordered 

baseline: 

34,776 

intervention 

studies and 

4914 control 

studies; #tests 

ordered postintervention: 

34,776 

intervention 

studies and 

4846 control 

studies 

Reduce ordering 

of imaging 

studies 

There was no significant 

difference in numbers of 

imaging studies ordered 

between the baseline and 

intervention periods.

35
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Flamm, 

201330

Pre-post with 

historical 

controls; 

Intervention 

2009, control 

2007 

Chest x-ray Web-based tool 

(PROP) that provides 

pre-operative test 

recommendations based 

on inputted patient and 

procedure data

Patients 

referred for preoperative 

evaluation prior 

to PROP 

implementation

Intervention 

1148; 

Controls 

1363 

Improve 

preoperative 

guideline 

adherence and 

reduce 

unnecessary 

testing

For chest x-ray, there were 

significantly fewer 

unnecessary pre-op x-rays

performed in the intervention 

group (1.9% vs 25.2%, 

p<0.001). However, 

intervention patients were also 

more likely to NOT receive xrays when clinically indicated 

(9.3% vs 1.9%, P<0.001). 

Gupta, 

201431

Pre-post; 

August 2007-

October 2009,

Dec 2009-

February 2012

Head CT A CDS for orders for 

mild head CT in 

traumatic brain injury 

that required clinicians 

to answer additional 

clinical questions

Web-based 

CPOE without 

CDS

Random 

sample of 

200 head CT 

examinations 

for mild 

traumatic 

brain injury 

in pre and 

post period

Adherence to 

evidence, head 

guidelines for 

use of head CT 

Adherence to guidelines was 

49% pre intervention and 

76.5% post intervention

Harpole,199

718

Phase 1: Case 

series; 

8/1/1995-

9/30/1995 

KUB CPOE with pop-up 

message indicating 

KUB for a specific 

indication was low 

yield, or another view 

or modality (eg, 

ultrasound) was more 

worthwhile

N. A. 190 patients, 

380 KUB 

orders

Reducing low 

yield KUB

Low yield KUBs were 

canceled in 3% of 258 orders. 

KUB order was changed to 

other view or modality in 38% 

of 109 orders.

Phase 2: RCT; 

11/10/1995-

3/21/1996 

CPOE with amended 

pop-up message further

emphasizing KUB for a 

specific indication was 

low yield, or another 

view or modality (eg, 

ultrasound) was more 

worthwhile

Original pop-up 

message from 

phase 1

491 patients; 

864 KUB 

orders

Low yield KUBs were 

canceled in 4% of 283 orders. 

KUB order was changed to 

other modality in 55% of 176 

orders.

36
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Ip, 201433 Pre-Post 2007-

2010 

Lumbosacral 

MRI

Presentation of 

ACP/APS guidelines 

for MRI imaging in 

LBP and mandatory 

"near real time" peerto-peer telephonic 

consultation with a 

radiologist or internist 

when attempting to 

override the guidelines; 

audit-and-feedback of 

performance to 

individual provider. 

Existing EHR 

with CPOE but 

without 

guidelines or 

other 

interventions. 

21,445 

primary care 

visits. 930 

visits for 

LBP had 

MRI ordered.

Reduce 

inappropriate use 

of MRI for back 

pain

Reduce use of LS MRI 

decreased from 5.3% of LBPrelated primary care visits to 

3.7% after implementation of 

the intervention. Guideline

adherence rate increased from 

78% to 98% with the 

intervention. 

Ip, 201322 Time Series; 

2004-2009 

CT, MRI, and 

nuclear 

cardiology 

procedures

CDS embedded in EHR 

that gives real-time 

feedback about 

appropriateness and 

regular peer-to-peer 

consultation to 

complete orders 

deemed uncertain or 

inappropriate

EHR before 

CDS 

implementation

50,336 

procedures

reduce imaging 

utilization

After implementation, use of 

procedures decreased from 

17.5 to 14.4 CTs per 1,000 

patient months, from 10.7 to 

11.1 MRIs per 1,000 patient 

months, and from 2.4 to 1.4 

cardiac procedures per 1,000 

patient months.

Raja, 201223 Time series; 

Oct 2003-Sept 

2007 

CT pulmonary 

angiography

CDS integrated into 

hospital CPOE system, 

that required physicians 

to order a D-dimer and 

give a clinical suspicion 

of high, medium, or 

low

CPOE before 

the intervention

6,838 

patients 

Reducing 

inappropriate CT 

pulmonary 

angiography in 

the ED

After CDS implementation, 

the rate of CT pulmonary 

angiography decreased by 

20%, from 26.4 to 21.1 

examinations per 1,000 

patients. The proportion of 

positive CT angiograms 

increased after the 

intervention from 5.9 to 9.8%.

Raja, 201438 Pre-Post

2009-2011 

CT 

pulmonary 

angiography

CDS integrated into 

CPOE, that required 

mandatory data input 

for each unique clinical 

attribute of the wells 

criteria and the D-dimer 

level, it required 9 

mouse clicks

1st generation 

CDS as 

described in 

Raja, 2012

2,423 

patients 

Appropriate use 

of CT pulmonary 

angiography ED

After the advanced CDS 

implementation, 

appropriateness increased 

from 56.9% to 75.6%, 

however use was constant and 

yield was relatively 

unchanged (10.4% pre, 10.4% 

post)

37
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Rosenthal, 

200624

Time series; 

Jan 2002-Dec 

2005 

CT/MRI, nuclear 

cardiology 

exams 

Computerized 

radiology order entry 

system that assigned a 

utility score to each 

ordered examination

Radiology 

order entry 

system without 

decision 

support 

71,966 post 

decision 

support tests

Reducing low 

utility tests

The rate of low utility 

examination declined from

6% to 2% across all 

examinations and all 

physician specialties. 19.4% 

of low utility scores resulted 

in immediate cancelation.

Sanders, 

200134

Pre-Post; Pre:

9/30/2000 – 

12/4/2000, 

Post: 

12/5/2000 – 

1/3/2001 

Brain MRI, head 

CT 

Implementation of 

WizOrder’s DSS 

(decision support 

system) which provides 

a recommended test 

(CT or MRI, contrast or 

non-contrast) based on 

ICD-9 codes and free 

text indications

Pre-intervention 

paper-based 

guidelines 

742 tests in 

pre-period, 

704 tests in 

post-period

Appropriateness 

of 

neuroradiology 

imaging requests

Significant difference in the 

distribution of orders for each 

study type with a trend 

towards ordering the 

recommended tests in the 

post-intervention period, with 

an increase in non-contract 

MRI being most prominent.

Sistrom, 

200924,25

Time-series; 

2000-2007 

CT, MRI, US Implementation of a 

Web-based radiology 

order entry system with 

decision support 

providing feedback on 

appropriateness based 

on provider-entered 

clinical information

Paper, facsimile 

and telephone 

methods

100% sample 

of radiology 

tests by 

quarter. 

Which were 

approximatel

y 13,000 CT, 

9,000 MR, 

and 11,000 

US each 

quarter.

Growth rates of 

outpatient CT, 

MRI and US 

volumes 

CT and US volumes growth 

and growth rates decreased 

significantly after 

implementation of 

computerized order entry with 

decision support; MRI growth 

rate also decreased 

significantly 3.0 to 0.25%, 2.2 

to 0.9%, 2.9 to 1.7%, 

respectively.

Solberg, 

201035

Pre-Post; 

2006-2007 

CT, MRI of the 

head, MRI of the 

lumbar spine

Implementation of 

decision support that 

identified 

appropriateness criteria 

(in 3 categories: A,B, 

or C)for the imaging 

studies 

HER with no 

decision 

support 

151 cases in 

pre-period, 

148 cases in 

post-period; 

all randomly 

chosen

Reduce 

inappropriate 

imaging studies. 

Volume of completed test 

orders decreased by 36.5% for 

head CT, and 20% for spine 

MRI, but increased by 3.3% 

for head MRI. Only MRI of 

the head and spine showed an 

increase in meeting 

appropriateness criteria postimplementation.

38
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Soo Hoo, 

201132

Pre-post; Dec

2006-Nov 

2008 

CT pulmonary 

angiography

CDS requiring 

physician-entered data 

to calculate a Wells 

score

CPOE without 

CDS

196 

examinations 

in the preintervention 

period, 261 

CT 

examinations 

on 252 

patients in 

the postintervention 

period

increase the 

yield of positive 

examinations

After implementation of the 

intervention, the proportion of 

positive examinations 

increased from 3.1 to 16.5%.

Tierney, 

199036

Pre-Post; Jan 

1988; 14 week 

preintervention, 

26 week 

intervention 

period, and 19 

week postintervention 

period

CXR; 

Abdominal 

sonography; CT 

Scan (head); 

Magnetic 

Resonance 

Imaging (head)

Charge for test 

displayed at the time of 

test ordering at 

academic general 

internal medicine 

practice.

Usual ordering 

without charge 

display 

Preintervention 

period: 3362 

patients in 

the control 

group and 

3511 in the 

intervention 

group; 

Intervention 

period: 4138 

control and 

4254 

intervention; 

Postintervention 

2806 control 

and 4461 

intervention. 

Assess whether 

knowledge of 

cost (charges) 

can affect 

physician 

ordering 

behavior

No specific data on imaging 

modalities. Data pooled and 

displayed as cost for all test 

including blood, urine, EKG 

and imaging studies. For both 

attendings and residents there 

was a statistically significant 

reduction in test ordering and 

charges.

39
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Author, 

Year

Study Design Modality 

(Radiology)

Intervention Comparison Sample size Target Finding

Tierney, 

198817

RCT; March 

24-Sept 30, 

1986 

CXR, as one of 8 

intervention tests

Display of predicted 

probabilities of positive 

test abnormalities 

provided at the time of 

test ordering. academic 

general internal 

medicine practice for 

scheduled patients.

Usual ordering 

without 

predicted 

probability 

display 

7658 

Scheduled 

visits in 

intervention 

group; 7590 

scheduled 

visits in 

control 

group; 487 

Chest X-rays

Reduction in 

ordering low 

probability tests 

with resultant 

decrease in 

charges per visit

There was non-significant

13.8% decrease in charges 

related to CXR ordering.

Vartanians, 

201037

Pre-post; Pre: 

April – Dec 

2006; Post: 

April – Dec 

2007 

Outpatient 

CT/MRI, and 

nuclear medicine 

examinations

Computerized 

Radiology order entry 

system rule change that 

prevented non-clinician

support staff from 

completing orders that 

received initial lowyield decision support 

score (required order to 

be entered directly by 

clinician)

CPOE before 

the intervention

42, 737 

orders in 

control group 

and 76, 238 

orders in 

study group

Reduce number 

of low-yield 

imaging studies

Reduction in number of lowyield tests decreased: 2106 of 

38,801 (5.43%) to 1261 of 

65,765 (1.92%),(P<0.001) .

40
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Study 

Author, 

Year Setting

IT Design

Data Entry 

Source Implementation Characteristics

Is it integrated with CPOE?Does it give real time feedback at point of care?Does the CDS suggest a recommended course of action?Intervention Classification*Is it automated through EHR?** Does clinical staff enter data specifically for intervention?Was it pilot tested or used an iterative process of development/ implementation?Was there any user training/ clinician education?Are the authors also the developers and part of the user group for the CDS?Was there use of auditand-feedback (or other internal incentive)?Are there any other implementation components not already discussed?

Bates, 199716

Harvardaffiliated 

academic 

medical center Yes Yes No A1 Yes No

Not 

stated

Not 

stated Yes No Not stated

Blackmore, 

201119

Integrated 

multidisciplinary 

healthcare 

network, 

Virginia Mason

Yes Yes Yes D No Yes

Not 

stated Yes

Not 

stated Yes

Authors consider 

context of facility 

important for success 

- salaried physicians 

in an integrated 

health network 

reducing utilization 

was an important 

institutional goal. 

There is a culture of 

evidence-based 

medicine and local 

development of 

protocols

Carton, 

200220

Two French 

teaching 

hospitals No Yes Yes C No Yes

Not 

stated

Not 

stated

Not 

stated

Not 

stated Not stated

Chin, 199921 Kaiser Yes Yes Yes A2 Yes No Not 

stated

Not 

stated

Not 

stated

Not 

stated No

41
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Study 

Author, 

Year Setting

IT Design

Data Entry 

Source Implementation Characteristics

Is it integrated with CPOE?Does it give real time feedback at point of care?Does the CDS suggest a recommended course of action?Intervention Classification*Is it automated through EHR?** Does clinical staff enter data specifically for intervention?Was it pilot tested or used an iterative process of development/ implementation?Was there any user training/ clinician education?Are the authors also the developers and part of the user group for the CDS?Was there use of auditand-feedback (or other internal incentive)?Are there any other implementation components not already discussed?

Curry, 201126

Rural Manitoba 

family medicine 

clinic

Yes Yes Yes

C 

Yes Yes Not 

stated Yes No Yes

Implemented at site 

indicating leadership 

interest in adoption

Day, 199527

UCLA-affiliated 

academic 

medical center No Yes Yes B No Yes No Yes Yes No No

Drescher, 

201128 VA

Yes Yes Yes B No Yes

Not 

stated Yes

Not 

stated

Not 

stated

Adherence by 

physicians in use of 

the CDS was 

documented and 

varied widely

Durand, 

201329

Johns Hopkinsaffiliated 

academic 

medical center Yes No No A1 Yes No No No Yes No No

Flamm, 

201330

Salzburg, 

Austria hospital No Yes Yes B No Yes No No No No No

Gupta, 201431

Harvardaffiliated 

academic 

medical center

Yes Yes Yes

B 

No Yes No No Unclear No No

42
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Study 

Author, 

Year Setting

IT Design

Data Entry 

Source Implementation Characteristics

Is it integrated with CPOE?Does it give real time feedback at point of care?Does the CDS suggest a recommended course of action?Intervention Classification*Is it automated through EHR?** Does clinical staff enter data specifically for intervention?Was it pilot tested or used an iterative process of development/ implementation?Was there any user training/ clinician education?Are the authors also the developers and part of the user group for the CDS?Was there use of auditand-feedback (or other internal incentive)?Are there any other implementation components not already discussed?

Harpole,1997

18

Harvardaffiliated 

academic 

medical center

Yes Yes Yes

B 

No Yes

Yes 

Phase 1 

led to 

Phase 2

No Probabl

y 

No No

Ip, 201433

Harvardaffiliated 

academic 

medical center Yes Yes Yes D Yes No

Not 

stated

Not 

stated

Not 

stated Yes

Mandatory peer-topeer telephone 

consult needed to 

override an alert

Ip, 201322

Harvardaffiliated 

academic 

medical center

Yes Yes Yes D Yes No No No Yes Yes

Academic detailing 

for high utilization 

outlier physicians. A 

risk contract with the 

payor was an external 

stimulus.

Raja, 201223

Harvardaffiliated 

academic 

medical center Yes Yes Yes B No Yes No Yes Yes No No

Raja, 201438

Harvardaffiliated 

academic 

medical center

Yes Yes Yes B No Yes No No Yes No

Authors assessed 

fidelity of the entered 

information to the 

medical record, 83% 

concordance was 

found. 

43
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Study 

Author, 

Year Setting

IT Design

Data Entry 

Source Implementation Characteristics

Is it integrated with CPOE?Does it give real time feedback at point of care?Does the CDS suggest a recommended course of action?Intervention Classification*Is it automated through EHR?** Does clinical staff enter data specifically for intervention?Was it pilot tested or used an iterative process of development/ implementation?Was there any user training/ clinician education?Are the authors also the developers and part of the user group for the CDS?Was there use of auditand-feedback (or other internal incentive)?Are there any other implementation components not already discussed?

Rosenthal, 

200624

Harvardaffiliated 

academic 

medical center Yes Yes Yes C No No No No Yes Yes No

Sanders, 

200134

Vanderbiltaffiliated 

academic 

medical center Yes Yes Yes C No Yes No No Yes No No

Sistrom, 

200925 24

Harvardaffiliated 

academic 

medical center Yes Yes Yes B No Yes No No Yes No

Phased 

implementation of 

web-based systems, 

but big-bang for 

decision support.

Solberg, 

201035

large 

multispecialty 

group in 

Minneapolis St. 

Paul Yes Yes No A2 Yes No No No

Not 

clear No

No financial 

incentives

Soo Hoo, 

201132 VA Yes Yes Yes D No Yes No No Yes No No

Tierney, 

199036

Regenstrief 

Health Center Yes Yes Yes A1

Yes*

** No No No Yes No No

44
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Study 

Author, 

Year Setting

IT Design

Data Entry 

Source Implementation Characteristics

Is it integrated with CPOE?Does it give real time feedback at point of care?Does the CDS suggest a recommended course of action?Intervention Classification*Is it automated through EHR?** Does clinical staff enter data specifically for intervention?Was it pilot tested or used an iterative process of development/ implementation?Was there any user training/ clinician education?Are the authors also the developers and part of the user group for the CDS?Was there use of auditand-feedback (or other internal incentive)?Are there any other implementation components not already discussed?

Tierney, 

198817

Regenstrief 

Health Center Yes Yes No A3

Yes*

** No No No Yes No No

Vartanians, 

201037

Harvardaffiliated 

academic 

medical center Yes Yes Yes D No No No No Yes No No

*Intervention Classification: “A” interventions provided information only; “B” interventions presented information on appropriateness or guidelines specifically tailored 

to the individual patient, often as a pop-up or alert. Some of these interventions also recommended alternative interventions, but did not include any barrier for the 

clinician to order the test; “C” interventions in general were similar to “B” interventions, but required the ordering clinician to justify with free text why they were 

overriding the decision support recommendation that a study was inappropriate (ie, a “soft stop”). “D” interventions included a “hard stop,” meaning the intervention 

prevented the clinician from ordering a test contrary to the CDS determination of inappropriateness, until additional discussion with or permission obtained from another 

clinician or radiologist.

** Eg, only uses data already being entered for clinical care

*** Integrated into an EHR precursor

45
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

APPENDIX B. Search Strategies

Search #1

DATABASE SEARCHED & TIME PERIOD COVERED: PubMed – 1/1/2011-9/10/2014

LANGUAGE: English

SEARCH STRATEGY:

Medical Informatics Applications[MESH:NoExp] OR Decision Making, ComputerAssisted[Mesh:NoExp] OR Decision Support Techniques[Mesh:NoExp] OR Information 

Systems[Mesh:NoExp] OR Decision Support Systems, Clinical[Mesh] OR hospital Information 

Systems[Mesh:NoExp] OR Management Information Systems[Mesh:NoExp] OR Medical Order 

Entry Systems OR automatic data processing[majr] OR medical informatics[majr] OR public 

health informatics[majr] OR electronics, medical[majr] OR (computers[mh] OR computers, 

handheld OR microcomputers OR medical records systems, computerized OR computer systems 

OR software[mh] OR computer-based[tiab] OR computerize*[tiab] OR cpoe OR cdss OR paper 

chart* OR electronic chart* OR health information technolog* OR electronic medical record* 

OR emr OR computerized physician order entry OR computerized order entry OR computerize 

order entry OR electronic health record* OR ehr OR information technology OR e-health OR 

health information OR hospital information OR health informatic* OR medical informatic* OR 

Medical Order Entry System* OR information infrastructure* OR ehealth

AND

radiology department OR magnetic resonance imaging OR tomography, x-ray computed OR 

imaging[tiab] OR radiolog*[tiab] OR neuroradiolog*[tiab] OR tomograph*[tiab] OR x-ray[tiab]

AND

appropriat* OR inappropriat* OR unnecessary OR behavior

AND

test OR tests OR testing

AND

utilization OR utilize OR utilizing OR order* OR request*

NUMBER OF RESULTS: 630

------

46
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

Search #2

DATABASE SEARCHED & TIME PERIOD COVERED: Web of Science – 1/1/2011-

9/10/2014

LANGUAGE: English

SEARCH STRATEGY:

FORWARD SEARCHES ON THE FOLLOWING ARTICLES:

1997. D. W. Bates, G. J. Kuperman, A. Jha, J. M. Teich, E. J. Orav, N. Ma'luf, A. Onderdonk, R. 

Pugatch, D. Wybenga, J. Winkelman, T. A. Brennan, A. L. Komaroff and M. J. Tanasijevic. 

"Does the computerized display of charges affect inpatient ancillary test utilization?" Arch Intern 

Med 157(21): 2501-8. 

2001. D. L. Sanders and R. A. Miller. "The effects on clinician ordering patterns of a 

computerized decision support system for neuroradiology imaging studies." Proc AMIA Symp: 

583-7. 

2002. M. Carton, B. Auvert, H. Guerini, J.-C. Boulard, J.-F. Heautot, M.-F. Landre, A. Beauchet, 

M. Sznajderi, D. Brun-Ney and S. Chagnon. "Assessment of radiological referral practice and 

effect of computer-based guidelines on radiological requests in two emergency departments." 

Clinical radiology 57(2): 123-128. 

2010. V. M. Vartanians, C. L. Sistrom, J. B. Weilburg, D. I. Rosenthal and J. H. Thrall. 

"Increasing the Appropriateness of Outpatient Imaging: Effects of a Barrier to Ordering LowYield Examinations 1." Radiology 255(3): 842-849. 

NUMBER OF RESULTS AFTER REMOVING DUPLICATES: 114

-----

Search #3

DATABASE SEARCHED & TIME PERIOD COVERED: PubMed – 1/1/2011 - 9/10/2014

LANGUAGE: English

SEARCH STRATEGY:

RELATED ARTICLE SEARCHES ON THE FOLLOWING ARTICLES: 

1997. D. W. Bates, G. J. Kuperman, A. Jha, J. M. Teich, E. J. Orav, N. Ma'luf, A. Onderdonk, R. 

Pugatch, D. Wybenga, J. Winkelman, T. A. Brennan, A. L. Komaroff and M. J. Tanasijevic. 

"Does the computerized display of charges affect inpatient ancillary test utilization?" Arch Intern 

Med 157(21): 2501-8. 

47
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

2001. D. L. Sanders and R. A. Miller. "The effects on clinician ordering patterns of a 

computerized decision support system for neuroradiology imaging studies." Proc AMIA Symp: 

583-7. 

2002. M. Carton, B. Auvert, H. Guerini, J.-C. Boulard, J.-F. Heautot, M.-F. Landre, A. Beauchet, 

M. Sznajderi, D. Brun-Ney and S. Chagnon. "Assessment of radiological referral practice and 

effect of computer-based guidelines on radiological requests in two emergency departments." 

Clinical radiology 57(2): 123-128. 

2010. V. M. Vartanians, C. L. Sistrom, J. B. Weilburg, D. I. Rosenthal and J. H. Thrall. 

"Increasing the Appropriateness of Outpatient Imaging: Effects of a Barrier to Ordering LowYield Examinations 1." Radiology 255(3): 842-849. 

RESULTS AFTER REMOVING DUPLICATES: 49

48
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

APPENDIX C. Description of Outcomes Used as Measures of 

Appropriate or Inappropriate Use

Study Method of Reporting Outcome

Day Was decision appropriate (according to guidelines)?

Chin % of UGI orders that conformed to a guideline

Carton % of radiologic examinations not in agreement with guidelines

Rosenthal Change in rate of radiology orders judged as “low utility” 

according to appropriateness criteria

Vartanians % of examinations ordered judged to be of low-yield

Presauer % of radiology ordering decisions not adherent with the CPSS

Hoo Yield of positive CT angiography examinations 

Raja, 2012 Yield of positive CT angiography examination 

Raja, 2014 Appropriateness of CT angiography ordering

Yield of positive CT angiography examination 

Flamm Numbers of radiology tests indicated or not indicated

Solberg Proportion of radiology studies meeting “high utility” 

according to appropriateness criteria 

Ip Guideline adherence rate 

49
EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program

 

APPENDIX D. Data Abstraction 

Study Design

Dates of Study

Modality (Radiology)

Intervention Description

Comparison

Sample Size

Target

Findings 

Setting 

IT Design

Is it integrated with CPOE? 

Does it give real time feedback at point of care? 

Does the CDS suggest a recommended course of action? 

Intervention Classification:

“A” interventions provided information only; 

“B” interventions presented information on appropriateness or guidelines 

specifically tailored to the individual patient, often as a pop-up or alert. Some of 

these interventions also recommended alternative interventions, but did not 

include any barrier for the clinician to order the test; 

“C” interventions in general were similar to “B” interventions, but required the 

ordering clinician to justify with free text why they were overriding the decision 

support recommendation that a study was inappropriate (ie, a “soft stop”);

“D” interventions included a “hard stop,” meaning the intervention prevented the 

clinician from ordering a test contrary to the CDS determination of 

inappropriateness, until additional discussion with or permission obtained from 

another clinician or radiologist.

Data Entry Source

Is it automated through EHR (eg, only uses data already being entered for clinical care)?

Does clinical staff enter data specifically for intervention?

Implementation Characteristics

Was it pilot tested or used an iterative process of development/ implementation?

Was there any user training/ clinician education?

Are the authors also the developers and part of the user group for the CDS?

Was there use of audit-and-feedback (or other internal incentive)?

Are there any other implementation components not already discussed?

50
 

 

 

 

 

 

 

 

 

EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program 

APPENDIX E. Peer Review Comments/Author Responses

Comment Author Responses 

This is a generally clear and complete report. The authors do a good job of distilling the We have added a new appendix table that indicates exactly 

key findings in a clear way. I have one methods question and a few comments that the appropriateness outcome used. In all the 

might make the report more useful for decision makers: 1) on page 9, under outcomes, appropriateness cases, the denominator was the number of 

please describe more completely the different ways that appropriateness and utilization radiologic examinations for which the guidelines (or 

were measured. It appears the most common way was to report appropriateness as a appropriateness criteria, etc) were judged to apply. So we 

ratio of appropriate/total tests before and after the intervention (as opposed to a rate). judge it as the same or at least sufficiently similar, constrict in 

However, it isn't clear that the denominator for those ratios was always the same -- only each article to currant pooling (Note: we are not assuming 

those tests that might have been impacted by the intervention or some that might not homogenous effect sizes. We are using a random effects 

have been. I assume utilization was usually reported as a rate of test ordering per analysis, we identified heterogeneity, and attempted to 

patient population, but the ability to detect a change obviously depends a lot on how explain some of it. 

specifically the population is defined -- all patients in a specific clinic (orthopedic clinic) 

or all patients with a specific indication for testing (patients with back pain). Given these 

differences is the assumption of homogenous effect size valid 

It would be helpful to translate the effect sizes observed into some more clinically We have added clinical examples to help readers understand 

meaningful differences, such as in relative reduction in inappropriate testing, or even # the effect size. 

tests averted, based on some reasonable assumptions about the baseline utilization 

and baseline proportion of inappropriate testing. the terms small or moderate effect 

otherwise have little meaning to clinicians. Given the larger effects of "hard stop" 

interventions it would be good to contrast the effects of best interventions (hard stop, 

audit and feedback, in integrated systems) vs. average, translated into real world 

estimates. 

In harms, it would be useful to report whether any studies examined perceived harms of 

"hard stop" interventions, including burden on clinicians, delays in appropriate testing, 

inconvenience for patients. 

Reporting of hazards was minimal. This is an area needed 

for future research. 

Given report is for VA, it might be good in text and summary included more specific 

mention of the one VA study. 

We added a paragraph about one VA study. 

The titles for the figures are potentially confusing -- rather than saying "Implementation 

Appropriateness" and "Implementation Utilization" I think you mean to say "Effects of 

Implementation Strategy on Appropriateness..." etc 

We have revised the titles for these figures. 

This review does a good job of summarizing evidence for EHR-based interventions for We have clarified that “system” in KQ 2 meant the EHR 

reducing unnecessary imaging. I have only minor comments for improvement : 1) Clarify intervention, not the setting. 

confusion between system (used in key question 2) and settings (used in the rest of the 

review) 

2) Explore context issues a bit more in discussion. The findings related to integrated 

systems perhaps gets to this issue? This is a topic area that this group is very well 

versed with and it would be good to expand discussion even if it’s a limitation that 

context was not well addressed in detail. 

Unfortunately, we’ve reported as much contextual 

information as is contained in these articles. 

 51 
 

 

 

 

 

EHR Interventions for Inappropriate Imaging Evidence-based Synthesis Program 

3) Can the review elaborate what was the precise type of advice that the intervention 

offered; for eg. Do this alternative versus do not do this. I like the 4 types mentioned but 

some granularity would be good. 

We have added more text on this. 

4) It would be good to clarify what they mean as audit and feedback upfront. This would 

be read by a broad audience and it wasn’t always clear what was being audited or wo 

was getting feedback in the authors use of the term 

We added specificity about the audit-and-feedback studies. 

5) Some discussion of unintended consequences of hard stops is warranted to balance 

the findings. For eg. Strom et al published about potential harm in Archives. 

(Unintended Effects of a Computerized Physician Order Entry Nearly Hard-Stop Alert to 

Prevent a Drug Interaction) 

We added the reference to the study by Strom. 

6) Can the authors expand on some more specific takeaways for VA operations who 

want to explore what exactly they should do to respond to the current pressures of 

reducing costs related to excessive imaging and using technology to do so. For 

example, could there be other options or other types of CDS that was not found in 

previous literature? 

We judge it beyond our scope for the Evidence Review to 

make these kinds of recommendations. It would be a great 

topic for the VA partner to produce. 

7) It appears that only the existing database was searched but was anything crosschecked to see if key articles were not left out? For eg. The editorials could have 

referenced some articles but I am not sure if these were looked at. 

We reference-mined all included studies and didn’t find any 

additional eligible studies. We don’t think going back and redoing the decreases in Chaudhry, Goldzweig, Buntin and 

Jones is likely to be worth the effort. 

8) For residents ordering MRI in at least some of the VAs, the template generally 

displays “Discussed with Attending X”. Has this been studied and what type of 

intervention will this be? 

We did not find such a study in our search. If the ePOE 

would prohibit an order without this consultation, then we 

would classify it as a hard stop. 

It appears that only the existing database was searched but was anything crosschecked to see if key articles were not left out? For eg. The editorials could have 

referenced some articles but I am not sure if these were looked at. 

We also reference-mind included studies. We don’t believe 

going back and re-doing the searches of Chaudhry, 

Goldzweig, Buntin and Jones is likely to be worth the effort. 

 52 