# Impact on clinical guideline adherence of Orient-COVID, a clinical decision support system based on dynamic decision trees for COVID19 management: A randomized simulation trial with medical trainees

Mouin Jammal, Antoine Saab, Cynthia Abi Khalil, Charbel Mourad, Rosy Tsopra, Melody Saikali, Jean-Baptiste Lamy

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mouin Jammal, Antoine Saab, Cynthia Abi Khalil, Charbel Mourad, Rosy Tsopra, et al. Impact on clinical guideline adherence of Orient-COVID, a clinical decision support system based on dynamic decision trees for COVID19 management: A randomized simulation trial with medical trainees. International Journal of Medical Informatics, 2025, 195, pp.105772. 10.1016/j.ijmedinf.2024.105772. hal-04855598

HAL Id: hal-04855598

https://hal.science/hal-04855598v1

Submitted on 25 Dec 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Impact on clinical guideline adherence of Orient-COVID, a clinical decision support system based on dynamic decision trees for COVID19 management: A randomized simulation trial with medical trainees

Mouin Jammal<sup>a</sup>, Antoine Saab<sup>b,g, ID</sup>, Cynthia Abi Khalil<sup>c,g</sup>, Charbel Mourad<sup>d, ID</sup>, Rosy Tsopra<sup>e,f</sup>, Melody Saikali<sup>b, ID</sup>, Jean-Baptiste Lamy<sup>g, ID,*</sup>

## ARTICLE INFO

### Keywords:

Clinical decision support system
Decision tree
Simulation trial
COVID-19

## ABSTRACT

**Background:** The adherence of clinicians to clinical practice guidelines is known to be low, including for the management of COVID-19, due to their difficult use at the point of care and their complexity. Clinical decision support systems have been proposed to implement guidelines and improve adherence. One approach is to permit the navigation inside the recommendations, presented as a decision tree, but the size of the tree often limits this approach and may cause erroneous navigation, especially when it does not fit in a single screen.

**Methods:** We proposed an innovative visual interface to allow clinicians easily navigating inside decision trees for the management of COVID-19 patients. It associates a multi-path tree model with the use of the fisheye visual technique, allowing the visualization of large decision trees in a single screen. To evaluate the impact of this tool on guideline adherence, we conducted a randomized controlled trial in a near-real simulation setting, comparing the decisions taken by medical trainees using Orient-COVID with those taken with paper guidelines or without guidance, when performing on six realistic clinical cases.

**Results:** The results show that paper guidelines had no impact (p = 0.97), while Orient-COVID significantly improved the guideline adherence compared to both other groups (p < 0.0003). A significant impact of Orient-COVID was identified on several key points during the management of COVID-19: ordering troponin lab tests, prescribing anticoagulant and oxygen therapy. A multifactor analysis showed no difference between male and female participants.

**Conclusions:** The use of an interactive decision tree for the management of COVID-19 significantly improved the clinician adherence to guidelines. Future works will focus on the integration of the system to electronic health records and on the adaptation of the system to other clinical conditions.

## 1. Introduction

The US Institute of Medicine's influential report "To Err Is Human" [1] created awareness that medical error is a major cause of avoidable mortality, morbidity and inappropriate use of resources. With the

increasing recognition of shortcomings of healthcare systems, practice guidelines were widely advocated as a means of encouraging compliance with evidence-based practice, leading to the "guidelines movement" [2]. Clinical Practice Guidelines (CPGs) are text documents summarizing recommended practices for a specific condition, with the rationale

<footnote><sup>a</sup> Department of Internal Medicine, Lebanese Hospital Geitaoui-UMC, Beirut, Lebanon</footnote>

<footnote><sup>b</sup> Quality and Patient Safety Department, Lebanese Hospital Geitaoui-UMC, Beirut, Lebanon</footnote>

<footnote><sup>c</sup> Nursing Administration, Lebanese Hospital Geitaoui-UMC, Beirut, Lebanon</footnote>

<footnote><sup>d</sup> Department of Medical Imaging, Lebanese Hospital Geitaoui-UMC, Beirut, Lebanon</footnote>

<footnote><sup>e</sup> Université Paris Cité, Sorbonne Université, Inserm, Centre de Recherche des Cordeliers, F-75006 Paris, France</footnote>

<footnote><sup>f</sup> Department of Medical Informatics, AP-HP, Hôpital Européen Georges-Pompidou, F-75015 Paris, France</footnote>

<footnote><sup>g</sup> INSERM, Université Sorbonne Paris Nord, Sorbonne Université, LIMICS, 75006 Paris, France</footnote>

<footnote>* Corresponding author.
*E-mail addresses:* mouinjammal@yahoo.fr (M. Jammal), antoineSaab@outlook.com (A. Saab), Cynthiakjammal@outlook.com (C. Abi Khalil), charbel.j.mourad@hotmail.com (C. Mourad), rosytsopra@gmail.com (R. Tsopra), melodysaikali@hotmail.com (M. Saikali), jean-baptiste.lamy@inserm.fr (J.-B. Lamy).</footnote>
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Screenshot of the interactive decision tree for the management of hospitalized COVID-19 patients, before any user interaction. It gives an overview of the entire decision process, at a glance. Most nodes are yes/no questions, and use checked/unchecked radio buttons as symbols on the edge. To interact with the tree, the user can either click on the button at the bottom of a current node (e.g. "Yes" or "No"), or directly click on any node, for performing a faster or backward navigation.</figcaption></figure>

and supporting evidence. CPGs may include flowchart clinical algorithms.

There is evidence that CPGs can improve clinical outcomes, but also that the level of adherence is low in practice [3]. Paper guidelines provide limited support to clinicians for finding patient-specific recommendations [4]. The adherence to CPGs is impaired by many factors [5–9] including: (1) inaccessibility of guidance at the point of care: CPGs are long documents, difficult to read during medical consultations, (2) difficulties of application to local settings, (3) oversimplification: most CPGs address a single disease while many patients have multiple comorbidities, (4) ambiguity: guidelines are not written in a formal language, and (5) lack of integration of patient values and goals.

Clinical Decision Support Systems (CDSSs) are designed to implement CPGs and help clinicians make a decision about individuals [10–12]. Evidence suggests that CDSSs can positively impact care processes [13] and guideline adherence [14]. CDSSs can mitigate to a considerable degree the criticisms frequently made about CPGs.

The first step in CDSS design is to formalize the medical knowledge that is informally described in CPGs, using a variety of computer-interpretable formats. The second step is to develop a computer application that presents the knowledge conveniently, e.g. by triggering alerts [15] or allowing interactive navigation in the recommendations, presented as a tree or as a sequence of questions [16]. However, trees are often too large to be presented in their entirety on the screen, the navigation is then laborious and can be a significant cause of errors (up to 44% error in complex situations [17]).

The present study is part of a larger project, Orient-COVID, aimed at designing a CDSS to support the management of patients with COVID-19. The CDSS relies on an innovative visual interface for navigating in a decision tree. The present work aims to measure the impact of Orient-COVID on physician adherence, through a randomized controlled trial methodology in a simulated setting, versus paper guidelines, and versus the absence of support. The paper will follow the amendments to the STROBE guidelines for simulation-based research [18].

## 2. Materials and methods

### 2.1. Brief description of Orient-COVID

Orient-COVID is based on medical decision trees established from a review of international best practice guidelines for the management of COVID19 and formalized by a multidisciplinary team, including doctors, nurses and specialists in medical informatics. The decision trees have been structured in a formal ontology, and stored in an RDF quadstore.

The CDSS proposes an interactive navigation through the decision trees. It includes two innovative features for reducing the size of the tree and permitting its presentation in its entirety on the screen: (1) the use of the fisheye technique reduces the space devoted to the uns elected parts of the tree, and (2) the use of a multi-path decision tree model [19] allows the user selecting several paths at specific nodes. This is particularly useful when the CPG considers several risk factors or fol lowup elements, and proposes a specific independent response for each. In such situations, the multi-path tree model avoids duplication of parts of the tree. Instead, several paths are selected and each leads to a distinct recommendation.

Orient-COVID was developed as a client-server web application in Python using Brython, a JavaScript-compiled version of Python and Owlready, a module for ontology-oriented programming [20]. The role of the server is limited and most of the program is implemented in the client. This allows patient data to remain on the client and thus supports data privacy.

Fig. 1 shows the multi-path decision tree for hospitalization of COVID-19 patients, before user navigation, and Fig. 2 during user in teraction. Orient-COVID also proposes a patient data entry form, where the clinician can optionally enter patient data (Fig. 3). The data is used for triggering a personalized semi-automatic navigation in the decision tree, hence accelerating the navigation.

For more details on Orient-COVID, please refer to [21] and to the demonstration website:

http://www.lesfleursdunormal.fr/appliweb/orient_covid.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Screenshot of the interactive multi-path decision tree for the management of hospitalized COVID-19 patients, after some user interactions. Three nodes are current: two question nodes (labeled “Followup elements”, a multiple-choice question node, and “Abnormal CRP and cytokine storm”) and a recommendation node (“Monitoring + Anticoagulant”). Two other recommendations are still accessible for future navigation (“Additional tests” and “Treatment”). Notice the parts of the tree that have not been selected have been grayed and squeezed, thanks to fisheye.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Screenshot of the patient data entry form.</figcaption></figure>

## 2.2. Study design

This is a single-center, 3-arm parallel group unblinded randomized controlled study performed at the Lebanese Hospital Geitaoui, a 250-bed University Medical Center in Beirut, Lebanon. The study was per-

formed between December 2023 and February 2024. The participants performed on clinical cases, and no real patients were involved.

The protocol and informed consent documents were reviewed and approved by the hospital institutional review board.

## 2.3. Recruitment

Participants were medical students and residents on rotation at the Lebanese Hospital Geitaoui-UMC, a university hospital center. Enrollment was open after a communication about the study through diffusion lists. The participants were equally randomized in 3 groups: group A (no guidance), group B (paper guidance) and group C (Orient-COVID). Participants were remunerated for their participation in the study.

Upon their enrollment, participants received an information notice about the study method and protocol. Each participant performed sequentially all six clinical cases in the presence of the same senior medical professional (internal medicine physician with more than 10 years of postgraduate clinical experience, and thorough experience in managing COVID19 cases). His role was to perform the simulation in total neutrality, including asking the participant to state his decision at all steps, and recording the participant answers.

## 2.4. Clinical cases and gold standard

Six COVID19 clinical cases were created by a panel of medical experts, inspired by retrospective anonymized data of real patients admitted to the hospital between January and December 2022. This ensured near-to-real patient data for the simulation. The cases covered a number of common COVID19 hospital scenarios in terms of severity, with different outcomes (healed, deceased, transferred to higher level of care). For each case, experts defined through consensus and in accordance
with CPGs a set of time-dependent (upon admission, post-24h, upon discharge) diagnostic, clinical and therapeutic decisions, and then analyzed the patient medical file to verify patient clinical pathways, outcomes and conformity with the CPGs. These decisions relative to each case constituted the gold standard for the study. Cases are labeled thereafter 1, 2, 3, 4, 5, 6 (Supplementary Material #1).

## 2.5. Protocol

The participant signed an informed consent and disclosed demographic data (year of residency, year of birth, sex, university grades in the past academic year, prior experience with management of COVID19 hospital cases, estimated number of COVID19 cases managed, date of last COVID19 case managed). The participant was then given an appointment for the simulation session, which comprised the following steps:

1. The participant received an information notice about the study objectives, methods and steps.

2. If the participant was in Group B, he was presented with the paper CPG.

3. If the participant was in Group C, he was presented with Orient-COVID.

4. The senior evaluator presented the six cases sequentially to the participant, instructed the participant to formulate a decision according to the predefined decision checklist. The participant was invited to use the paper guidelines (group B) or Orient-COVID (group C), but without forcing them. Decisions were recorded by the evaluator and no critical feedback was given to the participant. For each case, the participant had to perform the initial evaluation of the patient (including the ordering of various lab tests), to make the initial decision (including various drug prescriptions, oxygen therapy, etc.) and finally to perform the reevaluation of the patient after a few days.

## 2.6. Data collected

For each clinical case, 22 decision criteria were considered, including:

* 9 criteria regarding the initial evaluation, and consisting of examinations or lab tests that can be ordered: EKG (electrocardiogram), chest CT (computed tomography), general blood tests, CRP (C-reactive protein), LDH (lactate dehydrogenase), troponin, D-Dimers, ferritin, Il 6 (interleukin 6).

* 6 criteria regarding the initial decision, consisting of 2 criteria relative to the decision to hospitalize the patient and the level of care (e.g. ICU or not), and 4 criteria about drug prescriptions: antibiotics, steroids, anticoagulant, oxygen.

* 7 criteria regarding the reevaluation of the patient, mixing medical assessment of the patient status, examination and lab test ordering: clinical status, oxygen need, fever, blood tests, chest CT, reevaluation decision, plan of care.

Each criterion was worth one point: whenever the participant's answer was in concordance with the gold standard for the criteria, 1 point was awarded. When the participant answer was incorrect, 0 point was awarded. The total score, with a maximum of 22 for each case, was the primary outcome. Secondary outcomes were each criterion considered individually, and the impact of the various factors (age, sex,...).

## 2.7. Number of subjects

Based on a power of 0.8, a risk $\alpha = 5\%$, a mean difference in score of 2 points, and a standard deviation of 4 points, the minimum number of clinical cases solved per group is 60, leading to 10 participants per group.

<table><caption>**Table 1**
Demographic characteristics of the participants recruited.</caption><thead><tr><th>Characteristic</th><th>Type</th><th>Modalities / Aggregation</th></tr></thead><tbody><tr><td rowspan="2">Sex</td><td rowspan="2">nominal</td><td>Male</td><td>12 (40%)</td></tr><tr><td>Female</td><td>18 (60%)</td></tr><tr><td rowspan="4">Age (years)</td><td rowspan="4">integer</td><td>Mean</td><td>25.9</td></tr><tr><td>Sd</td><td>2.21</td></tr><tr><td>Min</td><td>24</td></tr><tr><td>Max</td><td>33</td></tr><tr><td rowspan="4">Study year</td><td rowspan="4">integer</td><td>7<sup>th</sup></td><td>10 (33.3%)</td></tr><tr><td>8<sup>th</sup></td><td>9 (30.0%)</td></tr><tr><td>9<sup>th</sup></td><td>6 (20.0%)</td></tr><tr><td>10<sup>th+</sup></td><td>5 (16.7%)</td></tr><tr><td rowspan="2">Status</td><td rowspan="2">nominal</td><td>intern</td><td>10 (33.3%)</td></tr><tr><td>resident</td><td>20 (66.7%)</td></tr><tr><td rowspan="4">University grades (average over the previous year)</td><td rowspan="4">integer (0-20)</td><td>Mean</td><td>15.3</td></tr><tr><td>Min</td><td>14</td></tr><tr><td>Max</td><td>17</td></tr><tr><td></td><td></td></tr><tr><td rowspan="4">Number of COVID-19 cases treated</td><td rowspan="4">ordered nominal</td><td>&lt; 5</td><td>0</td></tr><tr><td>5 - 10</td><td>4 (13.3%)</td></tr><tr><td>11 - 30</td><td>11 (36.7%)</td></tr><tr><td>&gt; 30</td><td>15 (50%)</td></tr><tr><td rowspan="3">Group</td><td rowspan="3">nominal</td><td>A</td><td>10 (33.3%)</td></tr><tr><td>B</td><td>10 (33.3%)</td></tr><tr><td>C</td><td>10 (33.3%)</td></tr></tbody></table>

## 2.8. Statistical analysis

Statistical analysis was performed using R software, with a risk $\alpha = 5\%$ and bilateral tests. For the analysis, the base unit is the clinical case solved by a participant.

First, the mean score obtained in the three groups was compared with Welch Two Sample t-test, in two-by-two comparisons (3 tests).

Second, the impact of Orient-COVID was tested on each of the 22 criteria using Welch Two Sample t-test. Due to the relatively important number of criteria, we applied Bonferroni correction: we considered a significance threshold $\alpha' = \frac{0.05}{22} = 0.0023$.

Third, in addition to the use of Orient-COVID, the following factors were tested: participant sex, age, grades obtained at university, number of COVID-19 cases treated in the last year by the participant, and clinical case ID. For each factor, we performed a linear mixed model (LMM) analysis on the score, considering two fixed-effect factors: Orient-COVID and the factor to test (including the potential interaction between them). The participant ID was added as a random-effect factor.

Supplementary Material #2 includes the dataset and #3 the R sources.

# 3. Results

## 3.1. Recruited participants

Thirty participants were recruited in the study, 10 were allocated in each group, A, B and C. Table 1 shows demographic characteristics of the participants. 10 participants were 7th year students in medicine (pregraduation, called interns in Lebanon) and 20 were 8th year students or more (postgraduation, called residents). Residents were almost equally spread among groups (6 in group A, 7 in B and 7 in C). Despite being invited to do so, some participants in group B did not use the paper guidelines as instructed. On the contrary, in group C, all participants used Orient-COVID. The mean per-participant duration of the study was 45 minutes (for six clinical cases).

## 3.2. Comparison of the three groups

Fig. 4 shows the scores obtained for each of the three groups, A (no guidance), B (paper guideline) and C (Orient-COVID). Groups A and B
<figure><img src="image_7.png" /><figcaption>Fig. 4. Boxplot showing the score obtained for each group, the mean and the 95% confidence intervals, and *p*-values for two-by-two comparisons (\* = significant difference).</figcaption></figure>

obtained almost the same results, while group C obtained a better result (about 1.6 points above). Statistical tests show that there is no significant difference between group A and B, but a significant difference between groups A and C, and groups B and C, respectively (p < 0.0003).

This suggests that paper guideline did not help the participants for solving the clinical cases, while Orient-COVID significantly improved the quality of the decisions. The improvement is relatively modest (1.6 points) but highly significant. This may be related to the fact that several criteria were actually easy to answer: for 6 criteria, the percentage of good answers is above 95% (Chest CT, General blood tests, CRP, Decision to hospitalize, Antibiotics, Reevaluation decision). Consequently, the difference measured is restricted to a limited number of criteria.

As paper guidelines did not provide any support, and group A and B performed equivalently, we grouped them in a group labeled AB (without Orient-COVID) for the rest of the analysis, thus considering an "Orient-COVID" boolean variable instead of three groups. This facilitates the analysis and increases group size.

### 3.3. Per-criterion analysis

Table 2 shows the per-criterion analysis. Five significant differences were observed. For four criteria (Troponin, Anticoagulant, Oxygen and Clinical status), the value was significantly better with Orient-COVID. For the last criteria (Ferritin), the value was significantly lower. Indeed, it appears that ferritin lab test was not considered in the Orient-COVID decision support tool. Finally, some other criteria, e.g. EKG or Blood test, are not significantly impacted by the use of Orient-COVID using the Bonferroni correction, but trends can be observed.

### 3.4. Factor analysis

Table 3 shows the per-factor analysis. No significant difference was observed with regard to sex, age, study year, status (resident or not), grades obtained at university, or the number of COVID-19 cases the participants encountered during their clinical activity, nor any interaction between these factors and Orient-COVID. Nevertheless, this analysis should be considered cautiously because of the low group size.

<caption>**Table 2**</caption>

Per criteria analysis showing the mean score obtained for each criterion without Orient-COVID (AB) or with Orient-COVID (C), and the corresponding *p*-value (\* : significant after Bonferroni correction i.e. *p* < 0.0023, \*\* : *p* < 0.05).

<table><caption>**Table 3**</caption><thead><tr><th>Criteria</th><th>AB</th><th>C</th><th>p-value</th><th></th></tr></thead><tbody><tr><td>EKG</td><td>0.53</td><td>0.73</td><td>0.0053</td><td></td></tr><tr><td>Chest CT</td><td>0.98</td><td>0.98</td><td>0.70</td><td></td></tr><tr><td>General blood test</td><td>1</td><td>1</td><td>-</td><td></td></tr><tr><td>CRP</td><td>0.96</td><td>0.95</td><td>0.81</td><td></td></tr><tr><td>LDH</td><td>0.43</td><td>0.45</td><td>0.75</td><td></td></tr><tr><td>Troponin</td><td>0.31</td><td><strong>0.57</strong></td><td>0.0011</td><td>*</td></tr><tr><td>D-Dimers</td><td>0.61</td><td>0.72</td><td>0.14</td><td></td></tr><tr><td>Ferritin</td><td><strong>0.25</strong></td><td>0</td><td>5.3 × 10<sup>-9</sup></td><td>*</td></tr><tr><td>IL 6</td><td>0.48</td><td>0.63</td><td>0.044</td><td></td></tr><tr><td>Decision to hospitalize</td><td>0.99</td><td>1</td><td>0.32</td><td></td></tr><tr><td>Level of care</td><td>0.86</td><td>0.92</td><td>0.23</td><td></td></tr><tr><td>Antibiotics</td><td>0.98</td><td>0.92</td><td>0.14</td><td></td></tr><tr><td>Steroids</td><td>0.85</td><td>0.87</td><td>0.76</td><td></td></tr><tr><td>Anticoagulant</td><td>0.70</td><td><strong>0.98</strong></td><td>3.6 × 10<sup>-9</sup></td><td>*</td></tr><tr><td>Oxygen</td><td>0.58</td><td><strong>0.82</strong></td><td>0.00074</td><td>*</td></tr><tr><td>Clinical status</td><td>0.73</td><td><strong>0.93</strong></td><td>9.8 × 10<sup>-5</sup></td><td>*</td></tr><tr><td>Oxygen need</td><td>0.73</td><td>0.82</td><td>0.16</td><td></td></tr><tr><td>Fever</td><td>0.18</td><td>0.25</td><td>0.32</td><td></td></tr><tr><td>Blood test</td><td>0.88</td><td>0.97</td><td>0.018</td><td></td></tr><tr><td>Chest CT</td><td>0.98</td><td>0.98</td><td>0.705</td><td></td></tr><tr><td>Reevaluation decision</td><td>0.96</td><td>1</td><td>0.025</td><td></td></tr><tr><td>Plan of care</td><td>0.80</td><td>0.82</td><td>0.79</td><td></td></tr></tbody></table>

Results of the LMM analysis for the various factors considered. For each factor, the *p*-value is given, as well as the *p*-value of the factor interaction with the use of Orient-COVID.

<table><thead><tr><th>Factor</th><th>p-value</th><th>Inter. p-value</th></tr></thead><tbody><tr><td>Clinical case ID</td><td><strong>&lt;2 × 10<sup>-16</sup> *</strong></td><td><strong>0.023 (*)</strong></td></tr><tr><td>University grade</td><td>0.57</td><td>0.65</td></tr><tr><td>#COVID-19 cases treated</td><td>0.31</td><td>0.74</td></tr><tr><td>Age</td><td>0.24</td><td>0.87</td></tr><tr><td>Study year</td><td>0.22</td><td>0.68</td></tr><tr><td>Status (resident or not)</td><td>0.81</td><td>0.19</td></tr><tr><td>Sex</td><td>0.96</td><td>0.94</td></tr></tbody></table>

We noted that the three groups randomly obtained through randomization were not homogeneous regarding the number of COVID-19 cases encountered, i.e. the participants' level of experience with regards to COVID-19. In fact, there were 4 experienced participants, i.e. that have seen more than 30 COVID-19 cases, in group A, 4 in group B and 7 in group C. This might have favored the results of the group C. Therefore, we performed two complementary analyses: the first one was restricted to the most experienced participants (8 participants without Orient-COVID i.e. 48 samples vs 7 participants with Orient-COVID i.e. 42 samples), and the second one was restricted to participants with lower experience (12 participants without Orient-COVID i.e. 72 samples vs 3 participants with Orient-COVID i.e. 18 samples). In both analyses, the Welch t-test shows a significant difference in the score obtained with or without Orient-COVID, with *p* = 0.0095 and *p* = 0.0033, respectively. This stratified analysis confirms that Orient-COVID has a significant impact, independently from the experience level.

A significant difference was observed with clinical case ID (*p* < 2 × 10⁻¹⁶), with an interaction with Orient-COVID (*p* = 0.023). Fig. 5 shows per-case boxplots. It suggests that there was possibly some carryover or learning effect, i.e. the participants improved their performance as they solved clinical cases. The use of Orient-COVID improved the score for all clinical cases, with the exception of the clinical case #6.
<figure><img src="image_8.png" /><figcaption>Fig. 5. Boxplot showing the score obtained for each clinical case (labeled as 1-6 numbers in the box label at the bottom), without or with Orient-COVID (labeled in the box label as AB and C, respectively).</figcaption></figure>

# 4. Discussion

In this study, we have assessed the impact of the use of Orient-COVID, a CDSS based on the advanced visualization of decision trees, on the adherence to COVID19 CPGs during a randomized controlled trial in simulated environment. The results show a significant overall positive impact of the use of the Orient-COVID CDSS versus paper guidelines and absence of guidance. Previously, we performed a preliminary qualitative evaluation of Orient-COVID perceived usability, and we obtained a SUS (System Usability Scale) score of 92.5% [21], which is "excellent" according to the SUS scale [22].

## 4.1. Limitations

A study limitation is that participants were medical trainees (one third participants were 7th year students in medicine, and two thirds were medical residents from postgraduate years 1 till 5). This could have potentially affected the results positively, as trainees are less experienced and younger, and therefore more comfortable with digital tools. If the participants were more experienced physicians, we would expect differences in terms of need of support, as experienced physicians may need less the support of a CDSS, and in terms of facility to use the digital tool. This may lead to participants reluctant to use the proposed tool, either because they do not need it (of feel they do not need it), or because they are not comfortable with digital tools. However, the experience of older physicians may remain limited for recent emergent diseases (such as COVID-19 at the beginning of the outbreak), and more generally, medical students and residents are very involved in decision-making, especially in university medical centers. Another limit is that the study was monocentric. Finally, the study is a simulation trial [18], in which participants may not have acted as they would in real patients; however, simulation trials are often used for CDSSs because of their simplicity to set up, e.g. [23].

The main limitation of Orient-COVID is that it relies on CPGs, and therefore on their clinical quality. However, it has been shown that the quality of COVID-19 guidelines was not encouragingly high [24–26]. Another limitation is that it is designing for presenting the decision tree in its entirety on the screen, which is feasible on a computer, but not on a smaller screen, such as the one of a smartphone.

We encountered difficulties during randomization, the random process leading to non-homogeneous distribution across the "#COVID-19 cases treated" parameter. In future study, the randomization should be stratified according to parameters that may impact the results, especially if the number of participants is limited. We observed that the participant experience with COVID-19 had no significant impact on the results. This might be explained by the fact: (1) that all participants had at least a limited experience (no participants had seen fewer than 5 cases), (2)

that the current practice, leading to experience, may not match the recommendations, or (3) that the recommendations may have evolved.

## 4.2. Comparison to literature

In the literature, many CDSS were proposed for COVID-19. A first review, by A Ameri et al. [27], distinguished two main approaches: (1) expert-system CDSSs that rely on a human-designed knowledge base, such as those implementing CPGs, and (2) CDSSs that rely on machine learning, for which the knowledge is learned from huge patient databases. In the first category, to which belongs Orient-COVID, the most common approach was rule-based systems. Most of the proposed CDSSs (about 75%) belong to the second category.

A second review, by H Ben Khalfallah et al. [28], distinguished four categories of CDSSs: (1) alert systems that raise alerts at the point of care, (2) monitoring systems that track and record various physiological parameters of patients, (3) recommendation systems that support the navigation through CPGs, and (4) prediction systems that aim at making diagnosis or predicting the outcomes of treatment.

## 4.3. Detailed impact on the adherence to COVID19 CPGs

A positive impact was observed for certain criteria pertaining to all three levels of the clinical management (initial assessment, therapeutic decisions and reassessment), namely: Troponin, Anticoagulation treatment, Oxygenation treatment and Clinical status, with additional trends on the following criteria: EKG, Il 6, Blood test and Reevaluation decision. For COVID19, EKG and troponin measurement upon admission were reported as having a potentially high impact on the morbidity and mortality of COVID19 patients [29–32], and were used in prognostic scores [33,34]. Moreover, anticoagulation use has been found to be associated with better clinical outcomes for COVID19 patients [35]. Finally, oxygenation supplementation is critical [36].

Regarding the major decisions relative to patient admission, transfer or discharge, it seems, however, that there is no significant improvement associated with the use of Orient-COVID. This might imply that the real added value of the CDSS might lie more in guiding the clinician for the details of the evaluation and therapeutics, rather than result in a change in the distribution of COVID19 patient across different hospitals and extra-hospital settings.

Regarding ferritin, we have seen that Orient-COVID provided no support, and that it led to a significantly lower adherence on that point, probably because participants expected some guidance. This phenomenon is known as automation bias [37].

## 4.4. Comparison to paper CPG experience

The study results have shown no significant difference between paper guidance and absence of guidance, and a significant difference between the CDSS and both other groups. This is in line with other studies reporting that the paper-based guidelines did not support sufficiently healthcare practitioners in finding patient-specific recommendations [4,38].

Some participants did not use the paper guidelines. Three advantages of Orient-COVID over paper guidelines were reported orally by participants during the session: the intuitive aspect and functionalities of the user interface, the ease of navigation in the decision tree, and the automatic navigation after having entered patient data. This can reduce the time to decision and the cognitive burden [39].

## 4.5. Challenges and perspectives of integration in real clinical workflow

Orient-COVID was constructed using an ontological approach. It makes its update easy, since, in case of change in the recommendations, editing the decision trees modeled in the ontology is sufficient
for updating the system, without any need to modify the implementation. In fact, ontologies facilitate standardization, flexibility for change, and therefore promote sharing and reusability of medical knowledge between CDSS systems implemented in different technologies and standards. Along with the decision support tool, we developed a dedicated decision tree editor as a desktop application.

Further evaluations of the approach are, of course, needed, to assess its usability more in depth, but also to evaluate it in terms of chance of erroneous navigation and time gain for clinicians. The semi-automatic navigation, considering structured patient data available, also has to be connected to EHR from hospitals to reduce the cognitive burden associated with data entry, and properly evaluated. Second, as more experienced physicians may have different needs compared with trainees, in terms of decision-making and human-computer interaction, an evaluation involving a more experienced medical panel, such as attending physicians, is required.

## 5. Conclusion

We presented a simulation-based evaluation of Orient-COVID, a clinical decision support system for COVID19. The results showed that this tool significantly improved the adherence of participants to guidelines when compared to paper-based guidance and absence of guidance. In particular, adherence to a number of important assessment and therapeutic criteria were significantly improved, which might translate into better decisions impacting patient morbidity and mortality. Our main perspectives include (1) the integration of the system with hospital EHR, (2) the application of the dynamic multi-path decision tree visual approach to other clinical guidelines, in order to support clinicians on multiple types of patient diagnostic or therapeutic decisions for other clinical situations beyond COVID-19, and (3) the evaluation of the CDSS in clinical conditions, and/or with attending physicians, rather than medical trainees.

## 6. Summary table

### 6.1. What was already known on the topic

* Clinician adherence to clinical practice guidelines is low for many disorders, including COVID-19.

* Clinical decision support systems implementing guidelines can improve the clinician's adherence to guidelines.

* An approach is to permit the navigation through the guidelines, presented as a decision tree.

* This approach is limited by the size of the tree, which rapidly grows and does not allow its visualization in its entirety on the screen.

### 6.2. What this study added to our knowledge

* Using the fisheye visualization technique and an innovative multipath tree model, we designed Orient-COVID, a clinical decision support system for managing patients with COVID-19.

* We conducted a randomized controlled trial with medical trainees in a near-real simulation setting comparing Orient-COVID to paper guidelines and to the absence of guidance.

* Results showed that Orient-COVID improved significantly guideline adherence compared to paper guidelines or the absence of guidance.

### CRediT authorship contribution statement

**Mouin Jammal:** Writing – original draft, Validation, Methodology, Investigation, Formal analysis. **Antoine Saab:** Writing – original draft, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization. **Cynthia Abi Khalil:** Writing – review & editing, Validation, Methodology, Investigation, Conceptualization. **Charbel Mourad:** Project administration, Funding acquisition. **Rosy Tsopra:** Writing – review & editing, Methodology. **Melody Saikali:** Methodology, Investigation, Conceptualization. **Jean-Baptiste Lamy:** Writing – original draft, Visualization, Supervision, Software, Project administration, Methodology, Funding acquisition, Formal analysis, Conceptualization.

### Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Jean-Baptiste LAMY reports financial support was provided by French National Research Agency. Charbel Mourad reports financial support was provided by Lebanese CNRS. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Acknowledgements

This work was funded by the French Research Agency (ANR) and the Lebanese national research center (CNRS-L) through the Orient-COVID project [Grant No. ANR-21-LIBA-0004].

## Appendix A. Supplementary material

Supplementary material related to this article can be found online at https://doi.org/10.1016/j.ijmedinf.2024.105772.

## References

[1] Committee on Quality of Health Care in America (Institute of Medicine), To Err is Human Building a Safer Health System, National Academy Press, Washington, D.C., 2000.

[2] J. Fox, V. Patkar, I. Chronakis, R. Begent, From practice guidelines to clinical decision support: closing the loop, J. R. Soc. Med. 102 (11) (2009) 464–473, https://doi.org/10.1258/jrsm.2009.090010.

[3] E. Sharmin, M.S. Ahmed, M.R. Amin, M.N. Hasan, S. Papia, S.F. Ahmed, R. Quader, Adherence with the Standard Treatment Regimen for the Treatment of COVID-19 Patients in Bangabandhu Sheikh Mujib, Medical University, 2021.

[4] E. Kilsdonk, L.W. Peute, R.J. Riezebos, L.C. Kremer, M.W.M. Jaspers, Uncovering healthcare practitioners' information processing using the think-aloud method: from paper-based guideline to clinical decision support system, Int. J. Med. Inform. 86 (2016) 10–19, https://doi.org/10.1016/j.ijmedinf.2015.11.011.

[5] M. Lugtenberg, J.S. Burgers, C.F. Besters, D. Han, G.P. Westert, Perceived barriers to guideline adherence: a survey among general practitioners, BMC Fam. Pract. 12 (2011) 98, https://doi.org/10.1186/1471-2296-12-98.

[6] T. Sinuff, K.W. Eva, M. Meade, P. Dodek, D. Heyland, D. Cook, Clinical practice guidelines in the intensive care unit: a survey of Canadian clinicians' attitudes, Can. J. Anaesth. 54 (9) (2007) 728–736.

[7] T.M. Shaneyfelt, M.F. Mayo-Smith, J. Rothwangl, Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature, JAMA 281 (20) (1999) 1900–1905.

[8] A.F. Shaughnessy, L. Cosgrove, J.R. Lexchin, The need to systematically evaluate clinical practice guidelines, Journal of the American Board of Family Medicine: JABFM 29 (6) (2016) 644–648, https://doi.org/10.3122/jabfm.2016.06.160115.

[9] M. Sultan, W. Waganew, M. Kassaye, L. Beza, M. Waleligh, A. Azazh, S. Yifru, B. Redae, D. Ataro, A. Fisseha, A. Ashagre, A. Baru, Facilitators and barriers to national COVID 19 guideline adherence among healthcare providers in Ethiopia, Ethiop. J. Health Sci. 33 (2) (2023) 183–192, https://doi.org/10.4314/ejhs.v33i2.2.

[10] E.S. Berner, Clinical Decision Support Systems: Theory and Practice, Springer Science+ Business Media, LLC, New York, 2007.

[11] F. Hak, T. Guimarães, M. Santos, Towards effective clinical decision support systems: a systematic review, PLoS ONE 17 (8) (2022) e0272846, https://doi.org/10.1371/journal.pone.0272846.

[12] L. Souza-Pereira, N. Pombo, S. Ouhbi, V. Felizardo, N. Garcia, Clinical decision support systems for chronic diseases: a systematic literature review, Comput. Methods Programs Biomed. 195 (2020) 105565, https://doi.org/10.1016/j.cmpb.2020.105565.

[13] J.L. Kwan, L. Lo, J. Ferguson, H. Goldberg, J.P. Diaz-Martinez, G. Tomlinson, J.M. Grimshaw, K.G. Shojania, Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials, BMJ (Clinical Research Ed.) 370 (2020) m3216, https://doi.org/10.1136/bmj.m3216.
[14] M. Gholamzadeh, H. Abtahi, R. Safdari, The application of knowledge-based clinical decision support systems to enhance adherence to evidence-based medicine in chronic disease, Journal of Healthcare Engineering 2023 (2023) 8550905, https://doi.org/10.1155/2023/8550905.

[15] J.B. Lamy, V. Ebrahiminia, C. Riou, B. Séroussi, J. Bouaud, C. Simon, S. Dubois, A. Butti, G. Simon, M. Favre, H. Falcoff, A. Venot, How to translate therapeutic recommendations in clinical practice guidelines into rules for critiquing physician prescriptions? Methods and application to five guidelines, BMC Med. Inform. Decis. Mak. 10 (2010) 31.

[16] B. Séroussi, J. Bouaud, G. Chatellier, Guideline-based modeling of therapeutic strategies in the special case of chronic diseases, Int. J. Med. Inform. 74 (2) (2005) 89–99.

[17] B. Séroussi, J. Bouaud, D. Sauquet, P. Giral, P. Cornet, H. Falcoff, J. Julien, Why GPs do not follow computerized guidelines: an attempt of explanation involving usability with ASTI guiding mode, Stud. Health Technol. Inform. 160 (Pt 2) (2010) 1236–1240.

[18] A. Cheng, D. Kessler, R. Mackinnon, T.P. Chang, V.M. Nadkarni, E.A. Hunt, J. Duval-Arnould, Y. Lin, D.A. Cook, M. Pusic, J. Hui, D. Moher, M. Egger, M. Auerbach, Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements, Adv. Simul. (London, England) 1 (2016) 25, https://doi.org/10.1186/s41077-016-0025-y.

[19] H. Guo, M. Fan, Multi-path decision tree, in: International Conference on Computer Science and Electronics Engineering (ICCSEE 2013), 2013, pp. 1411–1413.

[20] J.B. Lamy, Owlready: ontology-oriented programming in python with automatic classification and high level constructs for biomedical ontologies, Artif. Intell. Med. 80 (2017) 11–28.

[21] J.B. Lamy, M. Jammal, M. Saikali, C. Mourad, C. Abi Khalil, A. Saab, Fisheye visualization and multi-path trees for presenting clinical practice guidelines: methods and application to COVID-19, in: International Conference Information Visualisation (IV), Tampere, Finland, 2023.

[22] A. Bangor, P. Kortum, J. Miller, Determining what individual SUS score mean: adding an adjective rating scale, J.usability Stud. 4 (2009) 114–123.

[23] L. Higi, R. Schmitt, K. Käser, M. Wälti, M. Grotzer, P. Vonbach, Impact of a clinical decision support system on paediatric drug dose prescribing: a randomised within-subject simulation trial, BMJ Paediatrics Open 7 (1) (2023), https://doi.org/10.1136/bmjpo-2022-001726.

[24] Y.Y. Wang, Q. Huang, Q. Shen, H. Zi, B.H. Li, M.Z. Li, S.H. He, X.T. Zeng, X. Yao, Y.H. Jin, Quality of and recommendations for relevant clinical practice guidelines for COVID-19 management: a systematic review and critical appraisal, https://doi.org/10.3389/fmed.2021.630765, 2021.

[25] Y.S. Amer, M.A. Titi, M.W. Godah, H.A. Wahabi, L. Hneiny, M.M. Abouelkheir, M.H. Hamad, G.M. ElGohary, M.B. Hamouda, H. Ouertatani, P. Velasquez-Salazar, J. Acosta-Reyes, S.M. Alhabib, S.A. Esmaeil, Z. Fedorowicz, A. Zhang, Z. Chen, S.J. Liptrott, N. Frungillo, A.A. Jamal, S.A. Almustanyir, N.U. Dieyi, J. Powell, K.J. Hon, R. Alzeidan, M. Azzo, S. Zambrano-Rico, P. Ramirez-Jaramillo, I.D. Florez, International alliance and AGREE-ment of 71 clinical practice guidelines on the management of critical care patients with COVID-19: a living systematic review, J. Clin. Epidemiol. 142 (2022) 333–370, https://doi.org/10.1016/j.jclinepi.2021.11.010.

[26] K.E.A. Burns, M. Laird, J. Stevenson, K. Honarmand, D. Granton, M.E. Kho, D. Cook, J.O. Friedrich, M.O. Meade, M. Duffett, D. Chaudhuri, K. Liu, F. D'Aragon, A. Agarwal, N.K.J. Adhikari, H. Noh, B. Rochwerg, Adherence of clinical practice guidelines for pharmacologic treatments of hospitalized patients with COVID-19 to trustworthy

standards: a systematic review, JAMA Netw. Open 4 (12) (2021) e2136263, https://doi.org/10.1001/jamanetworkopen.2021.36263.

[27] A. Ameri, A. Ameri, F. Salmanizadeh, K. Bahaadinbeigy, Clinical decision support systems (CDSS) in assistance to COVID-19 diagnosis: a scoping review on types and evaluation methods, Health Science Reports 7 (2) (2024) e1919, https://doi.org/10.1002/hsr2.1919.

[28] H. Ben Khalfallah, M. Jelassi, J. Demongeot, N.B. Ben Saoud, Decision support systems in healthcare: systematic review, meta-analysis and prediction, with example of COVID-19, AIMS Bioengineering 10 (1) (2023) 27–52.

[29] A. Wibowo, R. Pranata, M.R. Akbar, A. Purnomowati, J.W. Martha, Prognostic performance of troponin in COVID-19: a diagnostic meta-analysis and meta-regression, Int. J. Infect. Dis. 105 (2021) 312–318, https://doi.org/10.1016/j.ijid.2021.02.113.

[30] K. Changal, S. Veria, S. Mack, D. Paternite, S.A. Sheikh, M. Patel, T. Mir, M. Sheikh, P.K. Ramanathan, Myocardial injury in hospitalized COVID-19 patients: a retrospective study, systematic review, and meta-analysis, BMC Cardiovasc. Disord. 21 (1) (2021) 626, https://doi.org/10.1186/s12872-021-02450-3.

[31] M. Mele, L. Tricarico, E. Vitale, A. Favia, F. Croella, S. Alfieri, M.D. Corbo, F. Mango, G. Casavecchia, N.D. Brunetti, Electrocardiographic findings and mortality in covid-19 patients hospitalized in different clinical settings, Heart Lung 53 (2022) 99–103, https://doi.org/10.1016/j.hrtlng.2022.02.007.

[32] R. Zeijlon, P. Hällgren, V. Le, J. Chamat, J. Wågerman, I. Enabtawi, A. Rawshani, S. Unenge, S. Jha, E. Omerovic, B. Redfors, The role of admission electrocardiogram in predicting outcome in patients hospitalized for COVID-19, J. Electrocardiol. 75 (2022) 10–18, https://doi.org/10.1016/j.jelectrocard.2022.10.005.

[33] K.S. Appel, R. Geisler, D. Maier, O. Miljukov, S.M. Hopff, J.J. Vehreschild, A systematic review of predictor composition, outcomes, risk of bias, and validation of COVID-19 prognostic scores, Clin. Infect. Dis. 78 (4) (2024) 889–899, https://doi.org/10.1093/cid/ciad618.

[34] C. Buttia, E. Llanaj, H. Raeisi-Dehkordi, L. Kastrati, M. Amiri, R. Meçani, P.E. Taneri, S.A.G. Ochoa, P.F. Raguindin, F. Wehrli, F. Khatami, O.P. Espínola, L.Z. Rojas, A.P. de Mortanges, E.F. Macharia-Nimietz, F. Alijla, B. Minder, A.B. Leichtle, N. Lüthi, S. Ehrhard, Y.A. Que, L.K. Fernandes, W. Hautz, T. Muka, Prognostic models in COVID-19 infection that predict severity: a systematic review, Eur. J. Epidemiol. 38 (4) (2023) 355–372, https://doi.org/10.1007/s10654-023-00973-x.

[35] L. Jiang, Y. Li, H. Du, Z. Qin, B. Su, Effect of anticoagulant administration on the mortality of hospitalized patients with COVID-19: an updated systematic, Review and Meta-Analysis (2021), https://doi.org/10.3389/fmed.2021.698935.

[36] F. Mansab, H. Donnelly, A. Kussner, J. Neil, S. Bhatti, D.K. Goyal, Oxygen and mortality in COVID-19 pneumonia: a comparative analysis of supplemental oxygen policies and health outcomes across 26 countries, Front. Public Health 9 (2021) 580585, https://doi.org/10.3389/fpubh.2021.580585.

[37] K. Goddard, A. Roudsari, J.C. Wyatt, Automation bias: empirical results assessing influencing factors, Int. J. Med. Inform. 83 (5) (2014) 368–375, https://doi.org/10.1016/j.ijmedinf.2014.01.001.

[38] Y. O'Connor, D. Ryan, V. Hardy, M. Thompson, J. Tsung-Shu Wu, C. Heavin, J. O'Donoghue, Stakeholders perspectives on paper-based and electronic clinical decision support systems in Malawi Africa, 2016.

[39] B.J. Sanderson, J.D. Field, A.B. Kocaballi, L.J. Estcourt, F. Magrabi, E.M. Wood, E. Coiera, Clinical decision support versus a paper-based protocol for massive transfusion: impact on decision outcomes in a simulation study, Transfusion (Paris) 63 (12) (2023) 2225–2233, https://doi.org/10.1111/trf.17580.