# Hospital antimicrobial stewardship team perceptions and usability of a computerized clinical decision support system

Alexandre Baudet, Marie-Jo Brennstuhl, Alexandre Charmillon, Florence Meyer, Céline Pulcini, Nathalie Thilly, Béatrice Demoré, Arnaud Florentin

<figure><img src="image_1.png" /></figure>

## To cite this version:

Alexandre Baudet, Marie-Jo Brennstuhl, Alexandre Charmillon, Florence Meyer, Céline Pulcini, et al.. Hospital antimicrobial stewardship team perceptions and usability of a computerized clinical decision support system. International Journal of Medical Informatics, 2024, 192, pp.105653. 10.1016/j.ijmedinf.2024.105653. hal-04756064

HAL Id: hal-04756064

https://hal.univ-lorraine.fr/hal-04756064v1

Submitted on 28 Oct 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Hospital antimicrobial stewardship team perceptions and usability of a computerized clinical decision support system

Alexandre Baudet<sup>a,b,*</sup>, Marie-Jo Brennstuhl<sup>a</sup>, Alexandre Charmillon<sup>c,d</sup>, Florence Meyer<sup>e</sup>, Céline Pulcini<sup>a,d</sup>, Nathalie Thilly<sup>a,f</sup>, Béatrice Demoré<sup>a,e</sup>, Arnaud Florentin<sup>a,g</sup>

## ARTICLE INFO

### Keywords:

Antimicrobial stewardship

Computerized decision support system

Mixed-method study

Semistructured interview

Usability

## ABSTRACT

**Background:** Antimicrobial stewardship (AMS) programs aim to optimize antibiotic use through a panel of interventions. The implementation of computerized clinical decision support systems (CDSSs) offers new opportunities for semiautomated antimicrobial review by AMS teams. This study aimed to evaluate the perceived facilitators, barriers and benefits of end-users related to a commercial CDSS recently implemented in a hospital and to assess its usability.

**Methods:** A mixed-method approach was used among AMS team members nine months after the implementation of the CDSS in a university hospital in northeastern France. A qualitative analysis based on individual semi-structured interviews was conducted to collect end-users' perceptions. A quantitative analysis was performed using the System Usability Scale (SUS).

**Results:** Eleven AMS team members agreed to participate. The qualitative analysis revealed technical, organizational and human barriers and facilitators of CDSS implementation. Effective collaboration with information technology teams was crucial for ensuring the installation and configuration of the software. CDSS adoption by the AMS team required time, human resources, training, adaptation and a clinical leader. Moreover, the CDSS had to be well designed, user-friendly and provide benefits to AMS activities. The quantitative analysis indicated that the CDSS was a "good" system in terms of perceived ease of use (median SUS score: 77.5/100).

**Conclusions:** This study shows the value of the studied CDSS to support AMS activities. It reveals barriers, facilitators and benefits to the implementation and adoption of the CDSS. These barriers and facilitators could be considered to facilitate the implementation of the software in other hospitals.

## 1. Introduction

Responsible use of antimicrobials is crucial for the treatment of infectious diseases and for reducing side effects at the individual level and antimicrobial resistance (AMR) at the population level [1]. AMR can complicate the treatment of infections, prolong the length of hospital stay, and increase treatment costs and mortality rates. Without actions to slow the increase in AMR, by 2050, the impact of drug-resistant infections will likely cause 10 million people to die every year and cost up to US$100 trillion across the world [2]. Antimicrobial misuse and

overuse are among the main modifiable drivers of AMR [3]. Between 20 % and 50 % of antibiotic use is estimated to be inappropriate [4]. A coherent set of actions—named antimicrobial stewardship (AMS)— should be implemented, i.e., promoting the use of an optimal antimicrobial drug, dose, duration and route of administration when an antibiotic is indicated to achieve the best desired clinical outcomes with low toxicity and minimum AMR, to improve the use of antimicrobials [5]. In our hospital, despite a long-standing and effective AMS policy [6,7], room for improvement still exists. As an example, we indeed found that in our intensive care units in 2020–2021, antibiotic treatment dosages

<footnote><sup>a</sup> Université de Lorraine, Inserm, INSPIRE, F-54000 Nancy, France</footnote>

<footnote><sup>b</sup> Université de Lorraine, CHRU-Nancy, Service d'Odontologie, F-54000 Nancy, France</footnote>

<footnote><sup>c</sup> CHRU-Nancy, Service de Maladies Infectieuses et Tropicales, F-54000 Nancy, France</footnote>

<footnote><sup>d</sup> CHRU-Nancy, Centre Régional en Antibiothérapie de la Région Grand-Est, France</footnote>

<footnote><sup>e</sup> CHRU-Nancy, Pharmacie, F-54000 Nancy, France</footnote>

<footnote><sup>f</sup> Université de Lorraine, CHRU-Nancy, Département Méthodologie Promotion Investigation, F-54000 Nancy, France</footnote>

<footnote><sup>g</sup> Université de Lorraine, CHRU-Nancy, Département Territorial d'Hygiène et Prévention du Risque Infectieux, F-54000 Nancy, France</footnote>

<footnote>* Corresponding author at: Faculté d'Odontologie de Lorraine, 7 Avenue de la Forêt de Haye, 54500 Vandœuvre-lès-Nancy Cedex, France.</footnote>

<footnote>E-mail address: alexandre.baudet@univ-lorraine.fr (A. Baudet).</footnote>
were only adapted for 45 % of patients with excessive antibiotic plasma concentrations and 14 % of patients with underconcentrations [8]. We felt that considering new actions to be implemented was necessary to strengthen our AMS strategy.

Over the past few decades, the rise of electronic medical records and other electronic patient data in hospitals has permitted the emergence of computerized clinical decision support systems (CDSSs) for AMS. CDSSs match electronic patient data, such as clinical, microbiological, therapeutic and administrative patient records, with a computerized knowledge base (rules based on clinical guidelines and expert opinions) to organize and present relevant and updated information to end-users (prescribers or AMS team members according to the type of CDSS) for a decision [9]. Many CDSSs have been developed to optimize antimicrobial use in hospitals [4,10]. The impact of CDSSs on antimicrobial use in hospitals has been studied in several systematic reviews and meta-analyses. In these studies, CDSSs were associated with improvements in the appropriate use of antimicrobials from 50 to 100 % and reduced patient mortality from 10 to 18 % [10–13]. In our hospital, a commercial CDSS named APSS® (Lumed, Sherbrooke, Québec, Canada) was recently implemented by the AMS team to support antimicrobial management. It was initially implemented in a Canadian hospital, where its impact was assessed through a quasiexperimental study using an interrupted time series between 2008 and 2013. This study revealed that the routine use of APSS was associated with improvements in appropriate antimicrobial prescriptions and decreased antimicrobial consumption, the average hospital length of stay and antimicrobial spending [14].

While CDSSs may improve antimicrobial use and patient outcomes and reduce health care costs, the challenges associated with their implementation, optimization, and adoption cannot be ignored [15]. As an approach to overcome these challenges, methodological pluralism is needed to evaluate CDSSs and to develop a rich and deep understanding of their implementation and use [16]. Both qualitative and quantitative mixed-method approaches are recommended for providing in-depth insights into end-users' perceptions and identifying benefits, facilitators and barriers related to CDSS implementation and use [16–18]. These approaches should address not only technical issues but also organizational and human factors [15]. Indeed, evaluating the impact of CDSSs requires not only an investigation of computerized systems but also an investigation of the social and behavioral processes that affect and are affected by the introduction of CDSSs into daily practice [19]. If end-users have negative perceptions of a CDSS, they may be reluctant to incorporate it into their routine practice [20,21]. Therefore, investigating end-users' perceptions to provide useful insights into the determinants of successful implementation and use of CDSSs is interesting [22,23]. Facilitators and barriers that affect the implementation and use of CDSSs should be identified to anticipate difficulties and to improve their adoption by end-users.

This study aimed to evaluate the perceived facilitators, barriers and benefits of end-users of the AMS team regarding a commercial CDSS recently implemented in our hospital to support antimicrobial management and to assess the usability of this CDSS named APSS®.

## 2. Methods

This study was part of a larger quasiexperimental study that evaluated the impact of a CDSS for antimicrobial usage and electronic software to support inpatient infection surveillance at the University Hospital of Nancy, France (ClinicalTrials.gov Identifier: NCT04976829) [24]. This study used the same methodology already described in our study on end-users' perceptions and usability of the electronic surveillance software [25].

### 2.1. Study design

A mixed-method approach was adopted (i) to qualitatively explore the perceptions of facilitators, barriers and potential benefits of the

CDSS using individual semistructured interviews and (ii) to quantitatively evaluate the potential benefits and usability of the CDSS using a structured questionnaire. This study followed the consolidated criteria for reporting qualitative research (COREQ) guidelines (Supplementary Table S1) [26].

### 2.2. Setting and participants

The study was conducted at Nancy University Hospital in north-eastern France, a 1500-bed hospital where an AMS policy was initiated in the mid-1990s and reinforced in 2006. The AMS team started to use the CDSS in May 2023, and the study started in January 2024 after 9 months of routine use to give the staff time to adapt to the software.

All CDSS end-users within the AMS team were invited to participate in the study by email. The target population consisted of hospital practitioners and residents of the AMS team (clinical pharmacists and infectious disease specialists) who used the CDSS for at least one month and who agreed to participate in the study.

### 2.3. CDSS implementation and use

The CDSS, named APSS® (Lumed, Sherbrooke, Quebec, Canada) for the Antimicrobial Prescription Surveillance System, was developed in 2007 in Quebec and was first deployed at the Sherbrooke University Hospital in 2010. In a previous study, Beaudoin et al. reported the development and implementation of APSS®, as well as its use in this university hospital [27].

APSS® was first installed in France in 2021 at our university hospital. Between 2021 and 2023, the AMS team members of our hospital collaborated with the society Lumed to adapt the initial knowledge base of APSS® ("Lumed Canada") according to French rules ("Lumed France") and then to the local rules and guidelines of our hospital ("Lumed Nancy"). APSS® was progressively connected to the data flow from other software already in use at the hospital, such as: the electronic medical records DxCare® (Dedalus France, Le Plessis-Robinson, France), the electronic medical records dedicated to intensive care units ICCA® (Philips, Amsterdam, Netherlands) and the laboratory information system Glims® (Clinisys, Vincennes, France). APSS® was actively tested at the beginning of 2023 by AMS team members to validate the reliability of operations and to correct the last main problems before its practical use.

In May 2023, AMS surveillance with APSS® began in our hospital, and the AMS team members used it in their daily activities. This software monitors the relevant clinical information of inpatients to facilitate the evaluation of the appropriateness of antimicrobial prescriptions (antibiotics and antifungals) by AMS team members and to assist them in the postprescription review process. As new information becomes available, the system automatically checks in real time whether the ongoing antimicrobial treatment seems appropriate according to predefined rules in its knowledge base. APSS® generates automatic graduated alerts sent to the AMS team when a prescription seems inappropriate (a molecule not appropriate for treating the microorganism identified, redundant molecules, inappropriate dose or duration of therapy, possible switch of the route of administration, etc.) and when no antimicrobial agent is prescribed to treat an identified microorganism. A pharmacist resident on the AMS team screens these alerts daily (08:30 to 18:30)-except during the weekends-using different filters. He or she accepts or refuses the alerts based on clinical relevance after a review of the electronic medical records, if needed. Once a day, the pharmacist resident reviews the complex medical files with an infectious disease specialist of the AMS team to accept or refuse the alerts. If the alert is found to be relevant, the AMS team contacts the prescribers to discuss their prescriptions and advise them on potential improvements.
## 2.4. Data Collection

All interviews were conducted by a researcher (AB) trained for qualitative research and who had previously met the AMS team members to see how they worked. The interviews were conducted face-to-face at the hospital or by videoconference using Microsoft® Teams (Microsoft Corporation, Redmond, WA, USA), ensuring confidentiality (no one else was listening to the interviews). The semistructured interviews were conducted using an interview guide (Supplementary Table S2). The interview guide was adapted from the study by Jung et al. [28] and was already used in the previously mentioned study [25]. It consisted of open-ended questions to explore the perceptions of AMS team members about the use of the implemented CDSS and to understand barriers and facilitators related to its use.

All the AMS team members who used the CDSS were interviewed, and data saturation was reached (i.e., no new information that could inform the domains of discussion emerged from the last interview). All of them completed a short questionnaire including sociodemographic questions, six questions investigating the potential benefits of CDSS use with Likert scales, and the French version of the System Usability Scale (F-SUS), a translated and validated version of the System Usability Scale [29]. It is a ten-item scale developed by Brooke to quickly and easily provide a global view of the end-user's subjective assessments of usability [30].

## 2.5. Data analysis

All interviews were recorded using an audio recorder, transcribed verbatim and deidentified. NVivo 12 (QSR International, Burlington, MA) was used to analyze the interviews and to guide the thematic analysis. The analysis of transcripts was subject to a grounded and inductive thematic analysis [31,32]. A preliminary coding scheme was subsequently developed using the interview guide questions after data familiarization with the first few transcripts. Then, line-by-line coding was performed by adding emerging codes. Finally, the coding scheme was refined, and the last cycle of coding was performed. The coding scheme was performed by a researcher (AB) and reviewed by another researcher who is an expert in qualitative research (MJB). Two authors (AC and FM), who are AMS team members, checked the accuracy of the results and analysis.

The F-SUS score was calculated as described previously [25] and interpreted as indicated by Bangor et al. according to the seven-item adjective rating scale [33,34].

The quantitative data were analyzed using RStudio® (RStudio Inc., Boston, MA, USA). The data are described as numbers and percentages for categorical variables and as medians with ranges (min-max) for continuous variables.

## 2.6. Ethical Considerations

This study was approved by the Ethics Committee of the University Hospital of Nancy in August 2021 (approval no. 322). The participants were informed about the purpose of the study, and all participants signed an informed consent form. Participation was voluntary and not compensated.

## 3. Results

The semistructured individual interviews lasted 28 ± 12 min (min: 21 min, max: 56 min). Data saturation was reached at the 11th and last interviews.

### 3.1. Participant characteristics

Table 1 presents the demographic and professional characteristics of the included AMS team members. All the AMS team members were

<caption>Table 1</caption>

Characteristics of the antimicrobial stewardship (AMS) team participants (N = 11).

<table><thead><tr><th>Characteristics</th><th>n</th><th>%</th></tr></thead><tbody><tr><td>Sex</td><td></td><td></td></tr><tr><td>Female</td><td>8</td><td>73</td></tr><tr><td>Male</td><td>3</td><td>27</td></tr><tr><td>Age (years)</td><td></td><td></td></tr><tr><td>25-30</td><td>8</td><td>73</td></tr><tr><td>30-45</td><td>3</td><td>27</td></tr><tr><td>Position</td><td></td><td></td></tr><tr><td>Resident in pharmacy</td><td>7</td><td>64</td></tr><tr><td>Hospital practitioner in pharmacy</td><td>2</td><td>18</td></tr><tr><td>Hospital practitioner in infectious diseases</td><td>2</td><td>18</td></tr><tr><td>AMS activity</td><td></td><td></td></tr><tr><td>Antibiotics</td><td>5</td><td>46</td></tr><tr><td>Antifungals</td><td>3</td><td>27</td></tr><tr><td>Antibiotics and antifungals</td><td>3</td><td>27</td></tr><tr><td>Hospital experience (years)</td><td></td><td></td></tr><tr><td>&lt;3</td><td>4</td><td>36</td></tr><tr><td>3-5</td><td>4</td><td>36</td></tr><tr><td>c&gt;5</td><td>3</td><td>27</td></tr><tr><td>Experience on the AMS team (years)</td><td></td><td></td></tr><tr><td>&lt;1</td><td>7</td><td>64</td></tr><tr><td>1-5</td><td>1</td><td>9</td></tr><tr><td>&gt;5</td><td>3</td><td>27</td></tr><tr><td>Experience with the CDSS (months)</td><td></td><td></td></tr><tr><td>2-6</td><td>5</td><td>46</td></tr><tr><td>6-12</td><td>4</td><td>36</td></tr><tr><td>&gt;12</td><td>2</td><td>18</td></tr><tr><td>Use of the CDSS during working days</td><td></td><td></td></tr><tr><td>Every day</td><td>10</td><td>91</td></tr><tr><td>Every week</td><td>1</td><td>9</td></tr><tr><td>&lt;1/week</td><td>0</td><td>0</td></tr><tr><td>Use of electronic tools at home</td><td></td><td></td></tr><tr><td>Every day</td><td>10</td><td>91</td></tr><tr><td>Every week</td><td>1</td><td>9</td></tr><tr><td>&lt;1/week</td><td>0</td><td>0</td></tr></tbody></table>

included (11/11, participation rate: 100 %) and participated in the qualitative and quantitative study. Seven were pharmacy residents (R1 to R7), and four were hospital practitioners: two clinical pharmacists (HP1 and HP2) and two infectious disease specialists (HP3 and HP4). The participants had a median age of 28 years (range: 25-45 years), and they had worked on the AMS team for a median of 6 months (range: 2 months-14 years). They used the CDSS for a median of 6 months (range: 2-24 months), five participated in its installation in the hospital, and 10 used the CDSS every working day.

### 3.2. Perceived barriers

The AMS team members perceived several barriers to the adoption of the CDSS; they mainly reported technical barriers but also organizational and human barriers (Supplementary Table S3). At the time of installation, hospital practitioners involved in CDSS implementation highlighted technical barriers due to information technology (IT) teams, data flow interfacing and the CDSS configuration. First, the lack of involvement or the reluctance of IT teams could slow the implementation of the CDSS as follows:

* The hospital IT department.

> "It was due to our IT department; it didn't seem to be a priority, so the responses were often long and/or unsuitable, so it took a long time." (HP3)

* The CDSS developer.
"We don't have complete control of the software. To change a parameter, we always need to ask and justify the modification to the Lumed® team; they always have a stranglehold." (HP1)

"Our vision was not identical to that of the Lumed® team, so they were a little reluctant." (HP2)

* The companies that supply other hospital software providing data flows to the CDSS.

"We had to get feedback from (...) the company that manages DxCare® (...) to unblock the problem because (...) the interfacing was stuck." (HP3)

"We asked for some things in February, and a year later, it was still the same (...) it was a configuration with DxCare®." (HP3)

The interfacing of data flow between the CDSS and other hospital software was crucial for allowing the CDSS to operate by linking clinical, microbiological, drug and administrative data. In our hospital, the main difficulty was the interfacing of the antibiotic prescriptions' data flow from the prescription software.

"The interfacing between software, with Glims®, DxCare® and ICCA® (...) it was when it comes to antibiotic prescriptions that things get stuck." (HP3)

"The data flow (...) was a huge obstacle, given that not all the medical prescriptions (...) were coming through. We had inconsistent prescriptions with a major lack of data." (HP1)

Finally, the CDSS configuration was particularly complex because, for the first time, this CDSS was implemented in a European country. Therefore, it had to be set according to national and local guidelines. Moreover, the CDSS configuration also encountered organizational barriers because it required time and human resources, and the initial communication between the CDSS developer and the AMS team via email was not efficient.

"We had to adapt their Quebec database to our French way of doing things. That was a lot of work." (HP3)

"In the preimplementation and implementation phases, before use, it takes time." (HP3)

"For us, the other barrier to implementing this software was human resources, because there weren't really any dedicated people (...). So that, too, was a limitation that meant it could take some time." (HP1)

"Rather than just sending an email every X amount of time with a huge list of problems. Looking back, that's not the way to manage something over the long term." (HP2)

The CDSS configuration also required adaptations during the first months of use because of technical, organizational and human difficulties. Indeed, AMS team members identified missing and irrelevant alerts, which contributed to wasting time, increasing the already high number of alerts and leading to alert fatigue.

"We still have a significant number [of alerts] that are not relevant (...). I find it wastes time" (HP4)

"It generates a huge number of alerts for us, and we'd like to do a bit less of that." (R1)

"There are a lot of alerts (...), but it's probably also because of the size of the hospital that it can scare off those who use it." (HP1)

"It's a bit frustrating when you update, you have a lot of alerts that come back, and you feel like you can't get your head above water (...). You get the impression that you're being swamped a bit too quickly." (HP2)

Furthermore, the CDSS configuration needed to be adapted to solve any detected problems and to facilitate the work of AMS team members. Moreover, the AMS team had to reorganize its workflow to include CDSS use and to harmonize the practices of all the AMS team members.

"It's not artificial intelligence, so it doesn't learn from its mistakes, so (...) it takes time for the errors that are noted and reported to Canadians to be finally modified in the software." (HP4)

"For us, what took the longest was managing to combine practical use [of the software] with our pharmaceutical validation, knowing which alerts would be processed and being able to work successfully with infectious disease specialists. For us, what took the longest was knowing how we could change our basic way of work and adapt it to the use of the software." (HP1)

"There are several people involved in the antibiotic workstation (...); getting everyone to do the same thing was also a question of harmonizing practices." (HP3)

Finally, another barrier to CDSS adoption was the skepticism of some end-users about the impact on patients' health and reliability.

"At the University Hospital, there has been a workstation dedicated to validating antibiotics for a very long time now (...). [The CDSS] has changed practice, but has it improved it?" (R1)

"For drug interactions, I looked at APSS®, what it told me, what it suggested. But I always checked with another drug interaction analysis software." (R4)

### 3.3. Perceived facilitators

AMS team members perceived many technical and organizational facilitators and some human facilitators of the adoption of the CDSS (Supplementary Table S4). The adoption of the CDSS required two technical prerequisites: responsive technical support from the developer team and an adapted workstation. For example, the AMS team highlighted the importance of a dual screen.

"The company Lumed® did everything to keep things moving (...) they were reactive (...) they responded quickly." (HP3)

"When I had a problem (...) the IT guy [from Lumed®] would always get back to me." (R3)

"For me, it's great because I have two screens: on the left I have DxCare® where I do my validations and consult the patient file I'm working on, and on the right I have APSS®. Frankly, it's really fluid, it's great. When you only have one screen, it's a bit of a pain." (R1)

In addition, technical facilitators included the well-thought design of the CDSS and useful tools such as notes and reminders. Moreover, the CDSS should be pleasant and user-friendly to facilitate its use; it should be intuitive, easy to learn, easy to use, smooth to navigate, visually nice

<table><caption>**Table 2**

Technical characteristics that facilitated CDSS use.</caption><thead><tr><td>Facilitators</td><td>Verbatim statements</td></tr></thead><tbody><tr><td>Intuitive and easy to learn</td><td>"The software is easy to use, clearly (...). It's easy to learn." (HP4)</td></tr><tr><td>Easy to use</td><td>"It's very simple. It's cool. It's intuitive." (R2)<br>"It's a pretty good software: practical, ergonomic, fairly easy to use and to learn." (HP3)<br>"It's very easy to use (...). You quickly understand how it works." (R1)</td></tr><tr><td>Smooth to navigate</td><td>"The navigation of the software is fluid, it's fast, there are no problems." (HP1)<br>"The time it takes to load data (...), refresh windows, etc., goes really fast." (HP1)</td></tr><tr><td>Good visual appearance</td><td>"We've got a great visual, so we can see things very quickly and easily." (HP4)<br>"Visually, it's very good. On APSS®, we have a good visual record of prescriptions, duration of prescriptions (...), all dosages, we have renal function, in other words all the clinical and biological data is well displayed." (R2)</td></tr><tr><td>Customizable</td><td>"That's what's really interesting, customizing my screen to suit what I want and what I'm looking for in terms of information." (R6)<br>"You can change your window." (R3)</td></tr></tbody></table>
and clear, and customizable (Table 2).

"It's interesting to use, easy to navigate, very clear, well designed and quite intuitive." (R3)

"What I also like about the software is that it allows us to make notes." (R3)

"The reminder system allows us to set our reassessment dates." (R6)

The organizational facilitators reported by the AMS team included effective communication with the CDSS developer and organizational factors that contributed to the adoption of the CDSS by the AMS team. Regarding communication between AMS and developer teams, two effective strategies were highlighted: using a collaborative online platform to report and manage difficulties and planning regular meetings via videoconference to discuss and solve problems.

"Going through the Azur® platform (...), you make a work item, you send them a problem with a screenshot (...) and then they treat the problem, they assign it to themselves, etc. And then (...), you see if it's being processed or if it's still pending, you see if they've solved the problem and then they tell you how they've solved it." (HP3)

"With the weekly meetings, I think we've made a lot of progress (...), we could have exchanged information more easily." (HP2)

Regarding the adoption of the CDSS by the AMS team, several organizational facilitators were reported as follows: dedicated time to use the CDSS, a clinical leader to train the end-users and to coordinate the CDSS adoption, and progressive implementation to facilitate the new distribution and prioritization of tasks among the AMS team.

"To deploy and improve, you still need to have human time." (HP1)

"[The clinical leader] gave us a short one-hour training course." (R6)

"Little by little, we lightened the load (...), gradually making adjustments." (HP3)

"[Pharmacy residents] have alerts, they sort alerts, so they remove the alerts that they consider irrelevant, they only present us [infectious disease specialists] things that they think are relevant and for which we can take action or have an impact." (HP4)

"We have chosen a list of alert categories that we prioritize for review during the day." (HP2)

Finally, some human facilitators helped to adopt new software, such as the experience of end-users with other software and their proficiency in using a computer.

"I was quite familiar with classifying interventions because I've already worked with artificial intelligence software." (R4)

"I like anything that involves computers so, for me, it was pretty quick." (R3)

### 3.4. Perceived benefits

CDSS adoption was facilitated by all the perceived benefits highlighted by AMS team end-users (Supplementary Table S5). First, all members reported that the CDSS improved the clinical practices of the AMS team, and hospital practitioners thought they called prescribers more to discuss antimicrobial prescriptions. Some concrete examples of improved practices are presented in Supplementary Table S6.

"It really is an improvement in patient care. It probably won't reduce antibiotic consumption, but it will have an impact on patient care and the quality of patient care." (HP3)

"The impact it has on improving the quality of antibiotic stewardship." (HP3)

"I think we (...) call [the wards] more because we have additional data (...), and what's really good about the software is that it keeps us constantly on our toes." (HP1)

Moreover, the end-users appreciated using the CDSS because it centralized the information needed for their work in a digital format and

saved time checking antimicrobial prescriptions and tracking patients.

"It has provided a single system for antibiotics and antifungals (...) with something centralized in the same place." (R6)

"You have everything on the same screen: you have the prescriptions, you have all the microbiological samples, and you can open them, you just have to move your cursor over." (HP2)

"It allows us to be fast and efficient (...). In terms of prescription analysis (...), it saves a lot of time." (R7)

Finally, the CDSS was perceived as a helpful and reassuring tool for performing AMS, especially by residents who had less experience in AMS activities than did senior practitioners.

"For new residents who begin (...), I think [APSS®] is a great help." (HP2)

"I think it makes for greater safety (...), I think it's clearly an additional back-up in terms of alert (...). It adds another safety barrier." (HP1)

"It helps to secure my validation. Yes, it's very reassuring." (R4)

Regarding the six possible benefits investigated with the Likert scale (Fig. 1), the two main benefits reported by the AMS team (majority of "strongly agree") were the ability of the CDSS to improve the appropriate use of antimicrobials and the quality of care.

### 3.5. Usability of the CDSS

The overall median score was 77.5/100 (min: 65-max: 92.5), indicating a 'good' system in terms of perceived ease of use (Fig. 2). The details of each usability item are presented in Table 3. The three items obtained an optimal median score were the items 1, 3 and 4.

## 4. Discussion

This study used a mixed-method approach to identify the perceived facilitators, barriers and benefits related to the implementation and use of a CDSS by a hospital AMS team. Technical, organizational and human factors were identified. On the one hand, barriers were mainly technical difficulties due to IT teams' involvement and availability, data flow interfacing and the CDSS configuration. The reorganization of teamwork was also an important barrier, with difficulties in terms of human resources, dedicated time and the harmonization of practices associated with alert fatigue at the personal level. On the other hand, the end-users highlighted several facilitators. Having responsive technical support from the developer team, an adapted workstation and a clinical leader who assisted and trained his or her colleagues appeared crucial. CDSS adoption was also strongly associated with CDSS usability and perceived benefits. Both qualitative and quantitative analyses confirmed this last finding.

Two types of CDSSs exist: in-house CDSSs built by hospital teams for their own hospital practices or commercial CDSSs created by a developer team and sold to several hospitals. Commercial CDSSs have the advantage of proposing practical solutions, including medication-related knowledge bases and rules for pharmacy and prescribing applications, which do not require expertise or important resources from hospital teams to create such knowledge bases [35]. Other advantages of commercial rule bases include comprehensiveness, regular updates including new drugs and a larger pool of experts to draw on in authoring rules [36]. In our hospital, we chose a commercial CDSS, but this software was implemented for the first time in Europe and thus in France. A partnership between the CDSS developer and our hospital was established to allow the development of new knowledge bases adapted to our national rules. Thus, the configuration and implementation of the CDSS required substantial hospital human resources and time in our hospital. CDSS development indeed requires mixed clinical and IT teams to ensure user-friendliness, ergonomics, workflow integration and transparency of the decision-making process [37].

The development of a well-designed CDSS is important because poor
<figure><img src="image_4.png" /><figcaption>Fig. 1. Quantitative results for the six possible benefits of CDSS use.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. System usability scale (SUS) scores in relation to the adjective ratings.</figcaption></figure>

<table><caption>**Table 3**
Usability test results.</caption><thead><tr><th>Items rated on a 5-point Likert scale ranging from "1 = strongly disagree" to "5 = strongly agree".</th><th>Median</th><th>Min-Max</th></tr></thead><tbody><tr><td>I would like to continue using this system frequently.</td><td>5</td><td>3-5</td></tr><tr><td>I found this system unnecessarily complex.</td><td>2</td><td>1-3</td></tr><tr><td>I thought this system was easy to use.</td><td>5</td><td>4-5</td></tr><tr><td>I need the support of a technical person to be able to use this system.</td><td>1</td><td>1-4</td></tr><tr><td>I found the various functions in this system were well integrated.</td><td>4</td><td>3-5</td></tr><tr><td>I thought there was too much inconsistency in this system.</td><td>2</td><td>1-4</td></tr><tr><td>I would imagine that most people would learn to use this system very quickly.</td><td>4</td><td>3-5</td></tr><tr><td>I found the system very cumbersome to use.</td><td>2</td><td>1-3</td></tr><tr><td>I felt very confident using the system.</td><td>4</td><td>2-5</td></tr><tr><td>I needed to learn a lot of things before I could get going with this system.</td><td>2</td><td>1-4</td></tr></tbody></table>

system design has been identified in the literature as a barrier to CDSS adoption, while good and user-friendly designs have been reported as facilitators [38]. Therefore, well-designed software is an important determinant of its adoption, and a developer team must be attentive, reactive and engaged in the development and improvement of its software [25]. While several studies reported the poor user interface design of the studied CDSSs [28,38], the design of the CDSS implemented in our hospital was appreciated by end-users. Being a well-designed CDSS was identified as a facilitator during our interviews, and, in the quantitative analysis using SUS, the majority of end-users indicated that they would “strongly agree” to continue using this system that they found easy to use. Moreover, another positive point regarding the CDSS implemented in our hospital was its customizability, which allowed end-users to reorganize the software windows with the information they considered most relevant. Other studies have shown that a lack of customizability, flexibility and adaptability are barriers to CDSS adoption [28,39].

CDSS adoption is also dependent on the computer literacy of end-users. Poor computer skills are identified as barriers [38]. The learning period that follows the implementation of a new system depends on the end-user's technological competence and on training sessions [40]. A lack of technical knowledge and training is an important barrier to CDSS adoption [39]. However, CDSSs considered “easy to use”, such as the CDSS implemented in our hospital, provide a system that requires minimal training and computer skills [18]. For example, in our previous study on electronic surveillance software, end-users indicated poor computer skills, but they easily adopted the newly installed software because they reported good usability and adequate training sessions with a clinical leader [25]. On the one hand, ease of use is a strong facilitator of CDSS adoption [20,39]. On the other hand, a clinical leader among the hospital team is an adequate strategy for providing training sessions that are easily accessible for end-users [40]. Training sessions with clinical leaders could also contribute to the standardization of clinical practice.

The end-users frequently reported an excessive number of alerts when using the CDSS. This difficulty is particularly reported with closed commercial knowledge bases that generate alerts that are clinically unhelpful due to insufficient end-user configuration tools that allow local customization and the ability to turn on and off certain decision support features to be more suited to the local clinical environment [35]. In our study, end-users explained that the high number of alerts was due to the size of the hospital, with many patients receiving antibiotics, and to irrelevant alerts, especially in the first months of use. Prioritizing and selecting alerts that are critically important and that will have the greatest impact on patients are recommended to facilitate the management of a large number of alerts [40]. Concerning irrelevant alerts, they should be progressively corrected because they are frustrating for end-users and are a source of alert fatigue, which occurs when clinicians feel that there are too many alerts [39,41,42]. Regarding CDSSs, irrelevant alerts were described for drug-allergy interactions [43], drug-drug interactions [44], drug dosage limits [36], and intravenous/oral
switches in our study. Irrelevant alerts are not helpful and are problematic in other ways: they may reduce clinician confidence in the CDSS, and they can lead clinicians to ignore clinically relevant alerts [35]. The relevance of the information and alerts provided by the CDSS are impacted by the interfacing of the data flow and the quality of the data feeding the CDSS [45]. The use of open standards for structural and semantic interoperability and exchange, such as Health Level 7 International (HL7), is crucial for ensuring optimal connectivity between systems [40].

The accuracy of the content was identified as an important determinant for end-users to trust the CDSS, while doubts or skepticism concerning the relevance and reliability of the content reduce their overall trust in the system and negatively impact their perceptions of the CDSS [39]. In our study, despite the skepticism of some end-users, numerous perceived benefits of the implemented CDSS were reported by end-users. Perceived benefits are crucial for reinforcing the use of newly installed medical software, and perceived usefulness is a key characteristic in CDSS adoption [20]. Performance expectancy is an important facilitator of CDSS adoption and includes not only how end-users perceive the system to be useful but also how they perceive it to improve an individual's performance and how the system will improve outcomes and job fit [38]. For example, Albahar et al. highlighted that their system for AMS management was perceived as a system that may notably improve the ability to deliver AMS, improve work efficiency and improve the safety of antibiotic use [22]. In our study, similar perceived benefits were highlighted, such as the ability to improve antimicrobial prescriptions and the use of antimicrobials and to decrease AMR. Finally, several end-users reported that the CDSS was helpful and reassuring because it was perceived as a “safety net” that provides additional support for the validation of antimicrobial prescriptions. Chow et al. reported similar results with junior physicians, who trusted the credibility of the CDSS’s recommendations and used them to “cross-reference” their antibiotic choices [46]. In a study by Justinia et al., physicians reported that the CDSS is a valuable medication dispensing safety tool that can reduce medication errors [47]. CDSSs were effectively shown to reduce medication errors [48] and improve the quality and efficiency of medication use [49].

Regarding potential limitations, our study investigated the perceptions of the benefits, facilitators and barriers of one CDSS named APSS® by only one AMS team in a university hospital. This single-center design may lead to low external validity and a possible lack of generalizability of the results to other hospitals. Our study focusing on one team of users without any other diversification, it is more a descriptive study than a generalizable study [50]. Nevertheless, many of our results are consistent with those of previous studies, and thus we believe that our results have validity for applications in other hospitals that consider implementing a CDSS for AMS. The inclusion of all eligible AMS team members allowed us to reach data saturation in the qualitative analysis with 11 individual interviews. Moreover, the inclusion of these 11 end-users also seems sufficient for quantitative analysis because Tullis and Stetson showed that the accuracy of the SUS is 80 to 100 % with a sample size of 10 to 12 people [51]. No prescriber was included in our study because the CDSS was only used by AMS team members and not by prescribers in our hospital. Other studies had shown that prescribers using CDSS could be reluctant to its use because they judged that CDSS recommendations conflicting with their professional autonomy [52].

This mixed-methods approach is the first step in the evaluation of the CDSS implemented in our hospital. In line with our protocol study [24], further studies using interrupted time-series analysis will be conducted to evaluate the impact of the CDSS on the antibiotic use in adult inpatients, and to assess the safety of the intervention, and its impact on the appropriateness of antibiotic prescriptions and on antimicrobial resistances.

# 5. Conclusions

This study provides evidence showing how technical, organizational and human factors influence end-users' perceptions about the implementation and adoption of a CDSS for AMS. The main technical barriers were the IT teams' involvement and availability, data flow interfacing and the CDSS configuration during the implementation phase, and the organizational barriers were related to human resources, dedicated time, reorganization of the teamwork and harmonization of practices. The responsiveness of the technical support of the CDSS developer and the presence of a clinical leader within the hospital AMS team were crucial for assisting the end-users and improving the software. Functionality-related characteristics such as the speed and ease of use of the CDSS, as well as the relevance of its alerts, were important facilitators that contributed to the decrease in alert fatigue. Finally, perceived benefits and usefulness seemed to be decisive for CDSS use.

# 6. Summary table

What was already known on the topic:

* Commercial computerized clinical decision support systems (CDSSs) propose more practical solutions for antimicrobial stewardship team than in-house CDSSs built by hospital teams.

* Adoption of new health information systems in hospital depends on technical, organizational and human factors.

* CDSS development requires involvement of mixed clinical and information technology teams.

What this study added to our knowledge:

* CDSSs are valuable systems to support antimicrobial stewardship activities.

* The antimicrobial stewardship team highlighted benefits about the CDSS use which improved their clinical practices and the quality of care of patients.

* The adoption of a CDSS requires time, involvement, availability, training and adaptability of the user team.

# 7. Ethical approval Statement

The study was approved by the Ethics Committee of the University Hospital of Nancy.

## CRediT authorship contribution statement

**Alexandre Baudet:** Writing – review & editing, Writing – original draft, Visualization, Methodology, Investigation, Formal analysis, Data curation, Conceptualization.
**Marie-Jo Brennstuhl:** Writing – review & editing, Validation, Methodology, Formal analysis, Conceptualization.

**Alexandre Charmillon:** Writing – review & editing, Conceptualization, Software.
**Florence Meyer:** Writing – review & editing, Software.

**Céline Pulcini:** Writing – review & editing, Conceptualization.

**Nathalie Thilly:** Writing – review & editing, Methodology, Conceptualization.
**Béatrice Demoré:** Writing – review & editing, Supervision, Project administration, Conceptualization.
**Arnaud Florentin:** Writing – review & editing, Supervision, Project administration, Methodology, Conceptualization.

## Funding

BioMérieux—owner of Lumed—funded the installation and maintenance of the CDSS during the evaluation period. The company was,
however, not involved in the different phases of this study.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors thank all the interviewees who participated in the study.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.
org/10.1016/j.ijmedinf.2024.105653.

References

[1] C. Magalhães, M. Lima, P. Trieu-Cuot, P. Ferreira, To give or not to give antibiotics is not the only question, Lancet Infect. Dis. 21 (2021) e191-e201, https://doi.org/10.1016/S147873-3099(20)30602-2.

[2] O'Neill J. Tackling Drug-Resistant Infections Globally: Final Report and Recommendations. 2016.

[3] A.H. Holmes, L.S.P. Moore, A. Sundsfjord, M. Steinbakk, S. Regmi, A. Karkey, et al., Understanding the mechanisms and drivers of antimicrobial resistance, Lancet Lond. Engl. 387 (2016) 176-187, https://doi.org/10.1016/S0140-6736(15)00473-0.

[4] B. Rittmann, M.P. Stevens, Clinical decision support systems and their role in antibiotic stewardship: a systematic review, Curr. Infect. Dis. Rep. 21 (2019) 29, https://doi.org/10.1007/s11908-019-0683-8.

[5] Dyar OJ, Huttner B, Schouten J, Pulcini C, ESGAP (ESCMID Study Group for Antimicrobial stewardship). What is antimicrobial stewardship? Clin Microbiol Infect Off Publ Eur Soc Clin Microbiol Infect Dis 2017;23:793-8. DOI: 10.1016/j.cmi.2017.08.026.

[6] S. Bevilacqua, B. Demoré, E. Boschetti, T. Doco-Lecompte, I. May, T. May, et al., 15 years of antibiotic stewardship policy in the Nancy Teaching Hospital, Med. Mal. Infect. 41 (2011) 532-539, https://doi.org/10.1016/j.medmal.2011.08.001.

[7] S. Bevilacqua, B. Demoré, M.-L. Erpelding, E. Boschetti, T. May, I. May, et al., Effects of an operational multidisciplinary team on hospital antibiotic use and cost in France: a cluster controlled trial, Int. J. Clin. Pharm. 33 (2011) 521-528, https://doi.org/10.1007/s11096-011-9499-9.

[8] A. Baudet, M. Regad, S. Gibot, É. Conrath, J. Lizon, B. Demoré, et al., Pseudomonas aeruginosa infections in patients with severe COVID-19 in intensive care units: a retrospective study, Antibiotics 13 (2024) 390, https://doi.org/10.3390/antibiotics1305390.

[9] K. Thursky, Use of computerized decision support systems to improve antibiotic prescribing, Expert Rev. Anti Infect. Ther. 4 (2006) 491-507, https://doi.org/10.1586/14787210.4.3.491.

[10] E. Carracedo-Martinez, C. Gonzalez-Gonzalez, A. Teixeira-Rodrigues, J. Prego-Dominguez, B. Takkouche, M.T. Herdeiro, et al., Computerized clinical decision support systems and antibiotic prescribing: a systematic review and meta-analysis, Clin. Ther. 41 (2019) 552-581, https://doi.org/10.1016/j.clinthera.2019.01.018.

[11] M.T. Baysari, E.C. Lehnbom, L. Li, A. Hargreaves, R.O. Day, J.I. Westbrook, The effectiveness of information technology to improve antimicrobial prescribing in hospitals: A systematic review and meta-analysis, Int. J. Med. Inf. 92 (2016) 15-34, https://doi.org/10.1016/j.ijmedinf.2016.04.008.

[12] C.E. Curtis, F. Al Bahar, J.F. Marriott, The effectiveness of computerised decision support on antibiotic use in hospitals: A systematic review, PLoS One 12 (2017) e0183062.

[13] M. Laka, A. Milazzo, T. Merlin, Can evidence-based decision support tools transform antibiotic management? A systematic review and meta-analyses, J. Antimicrob. Chemother. 75 (2020) 1099-1111, https://doi.org/10.1093/jac/dkz543.

[14] V. Nault, J. Pepin, M. Beaudoin, J. Perron, J.-M. Moutquin, L. Valiquette, Sustained impact of a computer-assisted antimicrobial stewardship intervention on antimicrobial use and length of stay, J. Antimicrob. Chemother. 72 (2017) 933-940, https://doi.org/10.1093/jac/dkz468.

[15] Z. Chen, N. Liang, H. Zhang, H. Li, Y. Yang, X. Zong, et al., Harnessing the power of clinical decision support systems: challenges and opportunities, Open Heart 10 (2023) e002432.

[16] B. Kaplan, Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism, Int. J. Med. Inf. 64 (2001) 39-56, https://doi.org/10.1016/s1386-5056(01)00184-8.

[17] I. Sim, P. Gorman, R.A. Greenes, R.B. Haynes, B. Kaplan, H. Lehmann, et al., Clinical decision support systems for the practice of evidence-based medicine, J. Am. Med. Inform. Assoc. JAMIA 8 (2001) 527-534, https://doi.org/10.1136/jamia.2001.0080527.

[18] K. Zheng, R. Padman, M.P. Johnson, H.S. Diamond, Understanding technology adoption in clinical care: clinician adoption behavior of a point-of-care reminder

system, Int. J. Med. Inf. 74 (2005) 535-543, https://doi.org/10.1016/j.ijmedinf.2005.03.007.

[19] J.G. Anderson, C.E. Aydin, Evaluating the impact of health care information systems, Int. J. Technol. Assess. Health Care 13 (1997) 380-393, https://doi.org/10.1017/s0266462300010436.

[20] T. Korteistö, J. Komulainen, M. Mäkelä, I. Kunnamo, M. Kaila, Clinical decision support must be useful, functional is not enough: a qualitative study of computer-based clinical decision support in primary care, BMC Health Serv. Res. 12 (2012) 349, https://doi.org/10.1186/1472-6963-12-349.

[21] A. Moxey, J. Robertson, D. Newby, I. Hains, M. Williamson, S.-A. Pearson, Computerized clinical decision support for prescribing: provision does not guarantee uptake, J. Am. Med. Inform. Assoc. JAMIA 17 (2020) 25-33, https://doi.org/10.1197/jamia.M3170.

[22] F. Albahar, R.K. Abu-Farha, O.Y. Alshogran, H. Alhamad, C.E. Curtis, J.F. Marriott, Healthcare professionals' perceptions, barriers, and facilitators towards adopting computerised clinical decision support systems in antimicrobial stewardship in jordanian hospitals, Healthc. Basel. Switz. 11 (2023) 836, https://doi.org/10.3390/healthcare11060836.

[23] S.T.R. Zaidi, J.L. Marriott, Barriers and facilitators to adoption of a web-based antibiotic decision support system, South Med. Rev. 5 (2012) 42-50.

[24] A. Baudet, N. Agrinier, A. Charmillon, C. Pulcini, A. Lozniewski, N. Aissa, et al., Evaluating antibiotic stewardship and healthcare-associated infections surveillance assisted by computer: protocol for an interrupted time series study, BMJ Open 12 (2022) e056125.

[25] A. Baudet, M.-J. Brennstuhl, J. Lizon, M. Regad, N. Thilly, B. Demoré, et al., Perceptions of infection control professionals toward electronic surveillance software supporting inpatient infections: A mixed methods study, Int. J. Med. Inf. 186 (2024) 105419, https://doi.org/10.1016/j.ijmedinf.2024.105419.

[26] B.C. O'Brien, I.B. Harris, T.J. Beckman, D.A. Reed, D.A. Cook, Standards for reporting qualitative research: a synthesis of recommendations, Acad. Med. 89 (2014) 1245, https://doi.org/10.1097/ACM.0000000000000388.

[27] M. Beaudoin, F. Kabanza, V. Nault, L. Valiquette, An antimicrobial prescription surveillance system that learns from experience, AI Mag. 35 (2014) 15, https://doi.org/10.1609/aimag.v35i1.2500.

[28] S.Y. Jung, H. Hwang, K. Lee, H.-Y. Lee, E. Kim, M. Kim, et al., Barriers and facilitators to implementation of medication decision support systems in electronic medical records: mixed methods approach based on structural equation modeling and qualitative analysis, JMIR Med. Inform. 8 (2020) e18758, https://doi.org/10.2196/18758.

[29] G. Gronier, A. Baudet, Psychometric evaluation of the F-SUS: creation and validation of the french version of the system usability scale, Int. J. Human-Computer Interact. 37 (2021) 1571-1582, https://doi.org/10.1080/10447318.2021.1898828.

[30] Brooke J. SUS - A quick and dirty usability scale. Usability Eval. Ind., CRC Press; 1996.

[31] J.M. Corbin, A. Strauss, Grounded theory research: Procedures, canons, and evaluative criteria, Qual. Sociol. 13 (1990) 3-21, https://doi.org/10.1007/BF00988593.

[32] C. Pope, S. Ziebland, N. Mays, Qualitative research in health care, Analysing Qualitative Data. BMJ 320 (2000) 114-116, https://doi.org/10.1136/bmj.320.7227.114.

[33] A. Bangor, P.T. Kortum, J.T. Miller, An empirical evaluation of the system usability scale, Int. J. Hum-Comput Interact. 24 (2008) 574-594, https://doi.org/10.1080/10447310802205776.

[34] A. Bangor, P. Kortum, J. Miller, Determining what individual SUS scores mean: adding an adjective rating scale, JUX - J. User Exp. 4 (2009) 114-123.

[35] G.J. Kuperman, R.M. Reichley, T.C. Bailey, Using commercial knowledge bases for clinical decision support: opportunities, hurdles, and recommendations, J. Am. Med. Inform. Assoc. JAMIA 13 (2006) 369-371, https://doi.org/10.1197/jamia.M2055.

[36] R.M. Reichley, T.L. Seaton, E. Resetar, S.T. Micek, K.L. Scott, V.J. Fraser, et al., Implementing a commercial rule base as a medication order safety net, J. Am. Med. Inform. Assoc. JAMIA 12 (2005) 383-389, https://doi.org/10.1197/jamia.M1783.

[37] G. Catho, N.S. Centemero, H. Catho, A. Ranzani, C. Balmelli, C. Landelle, et al., Factors determining the adherence to antimicrobial guidelines and the adoption of computerised decision support systems by physicians: A qualitative study in three European hospitals, Int. J. Med. Inf. 141 (2020) 104233, https://doi.org/10.1016/j.ijmedinf.2020.104233.

[38] S. Devaraj, S.K. Sharma, D.J. Fausto, S. Viernes, H. Kharrazi, Barriers and facilitators to clinical decision support systems adoption: a systematic review, J. Bus. Adm. Res. 3 (2014) 36, https://doi.org/10.5430/jbar.v3n2p36.

[39] M. Laka, A. Milazzo, T. Merlin, Factors that impact the adoption of clinical decision support systems (CDSS) for antibiotic management, Int. J. Environ. Res. Public Health 18 (2021) 1901, https://doi.org/10.3390/ijerph18041901.

[40] R.T. Sutton, D. Pincock, D.C. Baumgart, D.C. Sadowski, R.N. Fedorak, K.I. Kroeker, An overview of clinical decision support systems: benefits, risks, and strategies for success, NPJ. Digit Med. 3 (2020) 17, https://doi.org/10.1038/s41746-020-0221-y.

[41] J.S. Ash, D.F. Sittig, E.M. Campbell, K.P. Guappone, R.H. Dykstra, Some unintended consequences of clinical decision support systems, AMIA Annu. Symp. Proc. AMIA Sympos. 2007 (2007) 26-30.

[42] G.N. Forrest, T.C. Van Schooneveld, R. Kullar, L.T. Schulz, P. Duong, M. Postelnick, Use of electronic health records and clinical decision support systems for antimicrobial stewardship, Clin. Infect. Dis. off Publ. Infect. Dis. Soc. Am. 59 (Suppl 3) (2014) S122-S133, https://doi.org/10.1093/cid/ciu565.
[43] M. Luri, L. Leache, G. Gastaminza, A. Idoate, A. Ortega, A systematic review of drug allergy alert systems, Int. J. Med. Inf. 159 (2022) 104673, https://doi.org/10.1016/j.ijmedinf.2021.104673.

[44] A. Alanazi, W. Alalawi, B. Aldosari, An evaluation of drug-drug interaction alerts produced by clinical decision support systems in a tertiary hospital, Cureus 15 (2023) e43141, https://doi.org/10.7759/cureus.43141.

[45] W. Van Biesen, D. Van Cauwenberge, J. Decruyenaere, T. Leune, S. Sterckx, An exploration of expectations and perceptions of practicing physicians on the implementation of computerized clinical decision support systems using a Qsort approach, BMC Med. Inf. Decis. Making 22 (2022) 185, https://doi.org/10.1186/s12911-022-01933-3.

[46] A. Chow, D.C.B. Lye, O.A. Arah, Psychosocial determinants of physicians' acceptance of recommendations by antibiotic computerised decision support systems: A mixed methods study, Int. J. Antimicrob. Agents 45 (2015) 295-304, https://doi.org/10.1016/j.ijantimicag.2014.10.009.

[47] T. Justinia, W. Qattan, A. Almenhali, A. Abo-Khatwa, O. Alharbi, T. Alharbi, Medication errors and patient safety: evaluation of physicians' responses to medication-related alert overrides in clinical decision support systems, Acta Inform

Medica AIM J Soc Med Inform Bosnia Herzeg Cas Drustva Za Med Inform BiH 29 (2021) 248-252, https://doi.org/10.5455/aim.2021.29.248-252.

[48] P. Jia, L. Zhang, J. Chen, P. Zhao, M. Zhang, The effects of clinical decision support systems on medication safety: an overview, PLoS One 11 (2016) e0167683.

[49] D.W. Bates, M. Cohen, L.L. Leape, J.M. Overhage, M.M. Shabot, T. Sheridan, Reducing the frequency of errors in medicine using information technology, J. Am. Med. Inform. Assoc. JAMIA 8 (2001) 299-308, https://doi.org/10.1136/jamia.2001.0080299.

[50] J. Daly, K. Willis, R. Small, J. Green, N. Welch, M. Kealy, E. Hughes, A hierarchy of evidence for assessing qualitative health research, J. Clin. Epidemiol. 60 (1) (2007) 43-49, https://doi.org/10.1016/j.jclinepi.2006.03.014.

[51] Tullis T, Stetson J. A comparison of questionnaires for assessing website usability, 13th Annual UPA Conference. Minneapolis, Minnesota, USA: 2004.

[52] B.A. Van Dort, W.Y. Zheng, M.T. Baysari, Prescriber perceptions of medication-related computerized decision support systems in hospitals: A synthesis of qualitative research, Int. J. Med. Inf. 129 (2019) 285-295, https://doi.org/10.1016/j.ijmedinf.2019.06.024.