The Use of Audit-and-Feedback to Improve Antimicrobial-Prescribing in Emergency Departments: a Quasi-experimental Study
Phase: NA
Conditions: Anti-Bacterial Agents, Respiratory Tract Infections
Sponsor: Iowa City Veterans Affairs Medical Center

1 
 
Using audit-and-feedback to improve antimicrobial-prescribing in Emergency 
Departments: a multicenter quasi-experimental study in the Veterans Health 
Administration 
 
The clinical trial was registered on ClinicalTrias.gov, NCT03349567. 
This document was last revised December 24, 2020 to modify statistical methods for the 
interrupted time-series analysis of the primary outcome.  
All outcomes were defined prior to study initiation. 
 
 

2 
 
INTRODUCTION 
Antimicrobial use is a major contributor to the spread and emergence of antimicrobial 
resistance, a persistent public health problem. Antimicrobials are frequently prescribed in both 
hospitals and ambulatory care, but the majority of antimicrobial-prescribing is in outpatient 
settings.1 An estimated 30% of all outpatient antimicrobial use is unnecessary.2, 3  
Audit-and-feedback is an effective strategy for improving antimicrobial-prescribing.4 
Audit-and-feedback involves reviewing a clinician’s antimicrobial-prescribing behavior and 
providing feedback to help the clinician adjust his/her performance. The use of audit-and-
feedback for outpatient antimicrobial stewardship has been largely studied in primary care 
clinics5-9 with limited data in Emergency Departments and urgent care centers.10-12  
The Emergency Department (ED) is a major provider of medical care in the United 
States and, in turn, an important partner in efforts to improve antimicrobial-prescribing. The use 
of EDs has been increasing steadily over the past decade.13 In 2016, there were 145.6 million 
visits to EDs in the United States, or 46 visits per 100 persons.14 During these visits, ED 
clinicians prescribed 29 million antimicrobials.14 Prior studies have demonstrated several 
opportunities to improve antimicrobial-prescribing in EDs.15-19 
In this quasi-experimental pilot study, we evaluated whether the use of audit-and-
feedback with peer-to-peer comparisons could reduce unnecessary antimicrobial use at 2 
intervention EDs compared to 2 matched control EDs.  
METHODS 
Study design: We performed a quasi-experimental pilot study with an interrupted time-series 
design and a matched-pair non-equivalent control group to evaluate the effect of a pilot 
intervention on clinicians at 2 participating EDs versus 2 control EDs. All EDs were affiliated with 
Veterans Affairs Medical Centers (VAMCs) in the midwestern United States.  
Ethics approval: The clinical trial was registered on ClinicalTrias.gov, NCT03349567. The 
Institutional Review Boards (IRB) at the University of Iowa and Indiana University as well as the 

3 
 
Research & Development Committee of the Iowa City and Indianapolis VAMCs approved this 
study and waived written informed consent. All other sites were not engaged in research 
activities, so local IRB approval was not sought. 
Site selection: First, we contacted VAMCs within two midwestern Veterans Integrated Service 
Networks (VISNs) to assess their interest in participating in this study. These 2 VISNs were 
chosen because they were among the few VISNs that had not implemented regional initiatives 
to improve antimicrobial use in ambulatory care.  
Once two intervention sites were identified, we pulled a year of antimicrobial-prescribing 
data for all EDs within these same two midwestern VISNs. Eligible control sites had a baseline 
antimicrobial-prescribing rate (outpatient antimicrobial prescriptions dispensed per total ED 
patient-visits) that was similar to the intervention sites. Selection of controls on the pre-
intervention outcome of interest minimized bias and reduced the impact of regression to the 
mean, which is a key threat when selecting poor performers in non-randomized trials.20 To 
further ensure that intervention and control sites were as similar as possible, control and 
intervention sites were also matched on their hospital complexity level, as defined by the VHA. 
Intervention: The entire study period was 2-years in duration. The baseline period covered 1-
year between 10/01/2017 through 09/30/2018, the implementation phase lasted one month 
(10/2018), and the intervention period spanned another 11 months, 11/01/2018 through 
09/30/2019. 
In 10/2018, the study team had meetings with all willing ED clinicians at the intervention 
sites. At intervention site 1, these meetings were face-to-face and took place during a two-day 
time frame when D.J.L., an Infectious Disease (ID) physician, was present on-site. These one-
on-one meetings with each clinician lasted 10-15 minutes and included a review of the 
clinician’s individual antimicrobial-prescribing data (see feedback metrics), an anonymous 
comparison to his/her local peers, and a review of antimicrobial-prescribing guidelines. At 
intervention site 2, D.J.L. enrolled ED clinicians by telephone while an on-site ID physician 

4 
 
(A.D.) met with clinicians in-person. The content of these meetings was identical to those at 
intervention site 1.  
After the initial meeting, the study team e-mailed updated data to each participating 
clinician on his/her antimicrobial use. These e-mails were sent once every quarter. Overall, 
three e-mails were sent over the 12-month study period. E-mails were not sent after the fourth 
quarter, as the study had concluded by that point.  
Beginning in late September 2018, a clinical pharmacist was assigned to the ED at 
intervention site 2 from 10a-10p every day. One of the clinical pharmacist’s responsibilities was 
to ensure adequate antimicrobial selection. These pharmacists were not part of this trial, and 
the study team was only made aware of their presence as the intervention period was about to 
begin. To account for this, separate analyses were performed for intervention sites 1 and 2. 
Feedback metrics: Feedback at the enrollment visit and in each quarterly e-mail update 
consisted of two metrics: 1) the clinician’s total antimicrobial-prescribing volume adjusted for the 
total number of ED patient-visits, i.e. a total antimicrobial prescribing rate, and 2) the clinician’s 
rate of prescribing antibiotics for uncomplicated viral acute respiratory infections (ARI), based on 
data from the VA’s internal Academic Detailing website. The ARI metric included visits for acute 
bronchitis and non-specific upper respiratory tract infections but excluded visits for patients who 
were immunosuppressed, had chronic lung disease, or had a concurrent infection.  
For the enrollment meeting with clinicians, both metrics were based on the year 
preceding the initiation of the study. Once the intervention period began, data on the clinician’s 
total antimicrobial-prescribing rate was only drawn from the intervention period, which could 
include more than one quarter. Data on the ARI metric was only provided if the patient had seen 
≥8 eligible ARI cases during the prior quarter; if there were not 8 eligible visits during the prior 
quarter, then more than one quarter of data could be included as long as the data was all from 
the intervention period.  

5 
 
At enrollment, clinicians were compared to their peers for both metrics. In subsequent e-
mail communications, clinicians were compared to their peers and to their own prior 
performance for the ARI metric. For the total antimicrobial-prescribing rate, e-mails only 
compared clinicians to their personal baseline performance. The wording used in all feedback 
reports was based on the MITIGATE toolkit (Supplemental Table 1).21 All peer-to-peer 
comparisons were also demonstrated graphically in an attachment to the e-mail (Supplemental 
Figure 1).  
To assess whether the clinician received the e-mail feedback, we requested a read 
receipt for all e-mail communication. The percentage of read receipts that were acknowledged 
was tracked over time. 
We did not educate or provide feedback to ED clinicians at the two control sites. The 
control sites were not aware that there antimicrobial-prescribing was being monitored. 
Outcomes: The primary outcome was the cumulative frequency of an outpatient antimicrobial 
prescription within 24 hours of the ED visit. ED visits that resulted in an inpatient admission 
within 24 hours after the visit were not eligible for inclusion in the outcome. Antimicrobials were 
defined as all agents included in the National Healthcare Safety Network’s (NHSN) Antimicrobial 
Use and Resistance Module.22 On a monthly basis, the cumulative number of outpatient 
antimicrobials prescribed by the ED and the total number of eligible ED visits were used as the 
numerator and denominator, respectively. This outcome was chosen as it would not be 
influenced by diagnostic shifting.12, 23 
The secondary outcome was the proportion of ED visits associated with an outpatient 
antibiotic prescription within 24 hours of the ED visit; it was calculated similar to antimicrobial 
use. Antibiotics were defined as all antibacterial agents, as defined by the NHSN’s Antimicrobial 
Use and Resistance Module.22 
The ARI metric, as defined above, was captured across all clinicians at the control and 
intervention sites. Guideline-concordant management was assessed using manual chart 

6 
 
reviews (see below). Other secondary outcomes were measured from 24 hours after the ED 
visit up to 30 days after the visit. These secondary outcomes included testing for the presence 
of Clostridioides difficile, confirmed C. difficile infection (CDI), return ED visit, late antimicrobial 
use, late antibiotic use, and admission to a VHA acute-care facility. CDI linked to a visit was 
confirmed if the patient had a positive C. difficile test and a qualifying antibiotic (metronidazole, 
oral vancomycin, or fidaxomicin) was dispensed to the patient within 7 days of the positive C. 
difficile test. Visits that occurred from 24 hours up to 30 days after the prior visit were recorded 
as a return ED visit. ED visits that occurred after 30 days from the prior ED visit were 
considered as independent visits. Primary and secondary outcomes, except return ED visits, 
included all eligible visits in the denominator.   
Data on oral antimicrobial prescriptions, demographics, comorbidities, infections, and 
laboratory data for Clostridioides difficile for ED visits during the study period were extracted 
from the Corporate Data Warehouse (CDW) using the VA Informatics and Computing 
Infrastructure (VINCI). The International Classification of Diseases, Tenth Revision (ICD-10) 
codes were used to identify existing comorbidities and infection diagnoses linked to the ED visit.  
Statistical Analysis: An interrupted times series analysis was used to evaluate the effect of 
audit-and-feedback between the intervention and control sites. The crude differences in patient 
characteristics from baseline to intervention periods were assessed for intervention and control 
sites using Fisher’s exact test and Wilcoxon rank-sum test. Monthly proportions of antimicrobial 
and antibiotic use were calculated at each site for the baseline and intervention time periods. 
Segmented regression analysis was conducted using generalized linear models to estimate 
change in monthly antimicrobial and antibiotic prescription rates between baseline and 
intervention periods for intervention and control sites, respectively. Models were adjusted for 
continuous variables for time (months before, during, and after implementation). The estimate 
for time during implementation was used to calculate the immediate change in prescription rates 
and was reported as incident rate ratios (IRR) with 95% confidence intervals (CI). Indicator 

7 
 
variables for the four seasons were also included in each model to adjust for seasonal trends. 
An R-side random statement was included in each model to vary the degree of overdispersion 
for each facility. Finally, a difference-in-difference estimate was reported for the change in 
monthly prescription rates between intervention and control sites in the intervention period while 
controlling for changes in the baseline period. A Poisson regression model with log link was 
used to estimate the rate of secondary outcomes. Models were adjusted for time as a 
continuous covariate, an indicator for month of implementation of intervention (October 2018), 
and included random effects to account for repeated measures among sites each month during 
the study period. An interaction variable for the study period (baseline or intervention) and site 
(intervention or control) was included in the model to estimate the IRR and 95% CIs. 
Manual chart reviews to assess guideline-concordant management: Patient-visits were eligible 
for manual chart review if the visit met the following criteria: 1) a patient’s first ED visit during the 
study period; 2) a qualifying infection diagnosis, as defined by ICD-10 codes, and 3) no 
evidence of immunosuppression. Qualifying diagnoses were acute bronchitis, acute 
exacerbations of chronic obstructive pulmonary disease (AE-COPD), acute sinusitis, cystitis, 
pharyngitis, and upper respiratory tract infection (URI) (Supplemental Table 2). 
Immunosuppression was defined as having a diagnosis of lymphoma, leukemia, HIV/AIDs, or 
organ transplantation during the 12 months prior to presentation or receipt of an 
immunosuppressive medication, which was defined as follows: prednisone or steroid equivalent 
at a dose ≥20 mg/day during the 30 days prior to admission, chemotherapy within the 30 days 
prior to admission, or an anti-rejection medication, biologic agent or a disease-modifying anti-
rheumatic drug within the 3 month prior to admission. At each of the 4 EDs, eligible patient-visits 
were randomly chosen for review. Two independent, blinded reviewers evaluated selected 
medical records while using defined algorithms (Supplemental Figures 2-7). Reviewers 
assessed the accuracy of the diagnosis and, if necessary, re-classified the infection type; they 
also evaluated a decision to prescribe or not prescribe antibiotics as well as the appropriateness 

8 
 
of antibiotic selection and duration. Reviewers consisted of a fourth-year medical student, an 
Internal Medicine resident physician, and an Infectious Disease fellow physician. If the initial 
reviewers’ assessment were discordant, a third reviewer, who was not blinded, reviewed the 
medical record. The third reviewer (D.J.L.) had to evaluate 128 (26.7%) charts during the 
baseline period and 135 (29.7%) during the intervention period. The goal was to include at least 
110 adjudicated charts per site for the baseline and intervention periods. 
Guideline-concordant management was defined as either prescribing an antibiotic when 
indicated or not prescribing an antibiotic when not indicated. Guideline-concordant antibiotic 
selection was defined as prescribing an antibiotic recommended by professional guidelines, as 
defined by our protocols (Supplemental Figures 2-7). Guideline-concordant duration was 
defined as prescribing an antibiotic for a duration that was ≤2 days shorter and ≤2 days longer 
than what is recommended by professional guidelines. 
References 
 
1. 
Suda KJ, Hicks LA, Roberts RM, Hunkler RJ, Matusiak LM, Schumock GT. Antibiotic 
Expenditures by Medication, Class, and Healthcare Setting in the United States, 2010-
2015. Clin Infect Dis. 2018;66:185-190. 
2. 
Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. Prevalence of Inappropriate Antibiotic 
Prescriptions Among US Ambulatory Care Visits, 2010-2011. JAMA. 2016;315:1864-
1873. 
3. 
Cosgrove SE, Seo SK, Bolon MK, et al. Evaluation of postprescription review and 
feedback as a method of promoting rational antimicrobial use: a multicenter intervention. 
Infect Control Hosp Epidemiol. 2012;33:374-380. 
4. 
Barlam TF, Cosgrove SE, Abbo LM, et al. Implementing an Antibiotic Stewardship 
Program: Guidelines by the Infectious Diseases Society of America and the Society for 
Healthcare Epidemiology of America. Clin Infect Dis. 2016;62:e51-77. 
5. 
Meeker D, Linder JA, Fox CR, et al. Effect of Behavioral Interventions on Inappropriate 
Antibiotic Prescribing Among Primary Care Practices: A Randomized Clinical Trial. 
JAMA. 2016;315:562-570. 
6. 
Gjelstad S, Hoye S, Straand J, Brekke M, Dalen I, Lindbaek M. Improving antibiotic 
prescribing in acute respiratory tract infections: cluster randomised trial from Norwegian 
general practice (prescription peer academic detailing (Rx-PAD) study). BMJ. 
2013;347:f4403. 
7. 
Vinnard C, Linkin DR, Localio AR, et al. Effectiveness of interventions in reducing 
antibiotic use for upper respiratory infections in ambulatory care practices. Population 
health management. 2013;16:22-27. 

9 
 
8. 
Madridejos-Mora R, Amado-Guirado E, Perez-Rodriguez MT. Effectiveness of the 
combination of feedback and educational recommendations for improving drug 
prescription in general practice. Med Care. 2004;42:643-648. 
9. 
Buehrle DJ, Shively NR, Wagener MM, Clancy CJ, Decker BK. Sustained Reductions in 
Overall and Unnecessary Antibiotic Prescribing at Primary Care Clinics in a Veterans 
Affairs Healthcare System following a Multi-faceted Stewardship Intervention. Clin 
Infect Dis. 2019. 
10. 
Yadav K, Meeker D, Mistry RD, et al. A Multifaceted Intervention Improves Prescribing 
for Acute Respiratory Infection for Adults and Children in Emergency Department and 
Urgent Care Settings. Acad Emerg Med. 2019;26:719-731. 
11. 
Cummings PL, Alajajian R, May LS, et al. Utilizing Behavioral Science to Improve 
Antibiotic Prescribing in Rural Urgent Care Settings. Open Forum Infect Dis. 
2020;7:ofaa174. 
12. 
Madaras-Kelly KJ HC, Townsend M, Potter EM, Spivak ES, Hall SK, Goetz MB, Nevers 
M, Ying J, et al. Impact of Implementation of the Core Elements of Outpatient Antibiotic 
Stewardship within Veterans Health Administration Clinics on Antibiotic Prescribing and 
Patient Outcomes Clinical Infectious Diseases. Clin Infect Dis. 2020. 
13. 
Marcozzi D, Carr B, Liferidge A, Baehr N, Browne B. Trends in the Contribution of 
Emergency Departments to the Provision of Hospital-Associated Health Care in the USA. 
Int J Health Serv. 2018;48:267-288. 
14. 
Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care 
Survey: 2017 Emergency Department Summary Tables. 
https://www.cdc.gov/nchs/data/nhamcs/web_tables/2017_ed_web_tables-508.pdf. 
Published 2017. Accessed January 18, 2021. 
15. 
Meyer HE, Lund BC, Heintz BH, Alexander B, Egge JA, Livorsi DJ. Identifying 
Opportunities to Improve Guideline-Concordant Antibiotic Prescribing in Veterans with 
Acute Respiratory Infections or Cystitis. Infect Control Hosp Epidemiol. 2017;38:724-
728. 
16. 
Timbrook TT, Caffrey AR, Ovalle A, et al. Assessments of Opportunities to Improve 
Antibiotic Prescribing in an Emergency Department: A Period Prevalence Survey. Infect 
Dis Ther. 2017;6:497-505. 
17. 
Denny KJ, Gartside JG, Alcorn K, Cross JW, Maloney S, Keijzers G. Appropriateness of 
antibiotic prescribing in the Emergency Department. J Antimicrob Chemother. 
2019;74:515-520. 
18. 
Chua KP, Fischer MA, Linder JA. Appropriateness of outpatient antibiotic prescribing 
among privately insured US patients: ICD-10-CM based cross sectional study. BMJ. 
2019;364:k5092. 
19. 
Lowery JL, Alexander B, Nair R, Heintz BH, Livorsi DJ. Evaluation of antibiotic 
prescribing in emergency departments and urgent care centers across the Veterans' Health 
Administration. Infect Control Hosp Epidemiol. 2020:1-8. 
20. 
Shadish WR Cook TD, Campbell DT. Experimental and quasi-experimental designs for 
generalized causal inference. Boston, MA: Wadsworth Cengage learning; 2002. 
21. 
May L, Yadav K, Gaona SD, Mistry R, Stahmer A, Meeker D, Doctor J, Fleischman R. 
MITIGATE Antimicrobial Stewardship Toolkit: a guide for practical implementation in 
adult and pediatric emergency department and urgent care settings. http://shea-

10 
 
online.org/images/priority-topics/MITIGATE_TOOLKIT_final.pdf. Published 2018. 
Accessed January 18, 2021. 
22. 
National Healthcare Safety Network. Antimicrobial Use and Resistance Module. Atlanta, 
GA: Centers for Disease Control and Prevention. 
https://www.cdc.gov/nhsn/pdfs/pscmanual/11pscaurcurrent.pdf. Published 2021. 
Accessed January 18, 2021. 
23. 
Sanchez GV, Fleming-Dutra KE, Roberts RM, Hicks LA. Core Elements of Outpatient 
Antibiotic Stewardship. MMWR Recomm Rep. 2016;65:1-12. 
 

11