# Integration of Explicit Criteria in a Clinical Decision Support System Through Evaluation of Acute Kidney Injury Events

Laurine Robert, Chloé Rousseliere, Jean-Baptiste Beuscart, Sophie Gautier, Emmanuel Chazard, Bertrand Décaudin, Pascal Odou

<figure><img src="image_1.png" /></figure>

## To cite this version:

Laurine Robert, Chloé Rousseliere, Jean-Baptiste Beuscart, Sophie Gautier, Emmanuel Chazard, et al. Integration of Explicit Criteria in a Clinical Decision Support System Through Evaluation of Acute Kidney Injury Events. Studies in Health Technology and Informatics, 2021, Studies in Health Technology and Informatics, 281, pp.640-644. 10.3233/SHTI210249. hal-04219465

HAL Id: hal-04219465

https://hal.univ-lille.fr/hal-04219465

Submitted on 29 Sep 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Integration of Explicit Criteria in a Clinical Decision Support System Through Evaluation of Acute Kidney Injury Events

Laurine ROBERT<sup>a,b,1</sup>, Chloé ROUSSELIERE<sup>a</sup>, Jean-Baptiste BEUSCART<sup>b</sup>, Sophie GAUTIER<sup>c</sup>, Emmanuel CHAZARD<sup>b</sup>, Bertrand DECAUDIN<sup>a,d</sup> and Pascal ODOU<sup>a,d</sup>

<sup>a</sup>CHU Lille, Institut de pharmacie, F-59000 Lille, France

<sup>b</sup>Univ. Lille, CHU Lille, ULR 2694 - METRICS : Évaluation des technologies de santé et des pratiques médicales, F-59000 Lille, France

<sup>c</sup>Centre régional de PharmacoVigilance, Service de pharmacologie médicale, CHU Lille, F-59000 Lille, France

<sup>d</sup>Univ. Lille, CHU Lille, ULR 7365 - GRITA - Groupe de Recherche sur les formes Injectables et les Technologies Associées, F-59000 Lille, France

**Abstract.** In Clinical Decision Support System (CDSS), relevance of alerts is essential to limit alert fatigue and risk of overriding relevant alerts by health professionals. Detection of acute kidney injury (AKI) situations is of great importance in clinical practice and could improve quality of care. Nevertheless, to our knowledge, no explicit rule has been created to detect AKI situations in CDSS. The objective of the study was to implement an AKI detection rule based on KDIGO criteria in a CDSS and to optimize this rule to increase its relevance in clinical pharmacy use. Two explicit rules were implemented in a CDSS (basic AKI rule and improved AKI rule), based on KDIGO criteria. Only the improved rule was optimized by a group of experts during the two-month study period. The CDSS provided 1,125 alerts on AKI situations (i.e. 643 were triggered for the basic AKI rule and 482 for the improved AKI rule). As the study proceeds, the pharmaceutically and medically relevance of alerts from the improved AKI rule increased. A ten-fold increase was shown for the improved AKI rule compared to the basic AKI rule. The study highlights the usefulness of a multidisciplinary review to enhance explicit rules integrated in CDSS. The improved AKI is able to detect AKI situations and can improve workflow of health professionals.

**Keywords.** Explicit criteria, CDSS, AKI, Detection

## 1. Introduction

Clinical Decision Support Systems (CDSS) can reduce the incidence of adverse events, optimize prescription safety and improve patient's care [1]. Most of the rules integrated in CDSS are explicit criteria based on validated tools, summary of product characteristics or based on recommendations from expert consensus [2]. Direct implementation of explicit criteria often lead to irrelevant alert rules [2]. The problem with non-specific rules is an over-alerting with the risk of overriding the warning message, even relevant alerts, and leading to alert fatigue from health professional [3,4]. Acute Kidney Injury

<footnote n="1">¹ Corresponding Author; E-mail: laurine.robert@chru-lille.fr</footnote>
(AKI) is a frequent and severe disease, associated with more frequent adverse drug reactions, hospitalizations and deaths [5]. Identification of AKI in CDSS appears therefore highly relevant, as such situations often require drug adaptation. But, to our knowledge, CDSS is used to detect AKI using encoded diagnoses, that are not always available nor reliable. Such implementation can lead to a lack of sensitivity, or a lack of specificity (over-alerting). Definition of AKI is based on explicit criteria of the Kidney Disease Improving Global Outcome (KDIGO) recommendations [6].

The objective of the study was to implement an AKI detection rule based on KDIGO criteria in a CDSS and to optimize this rule to increase its relevance in clinical pharmacy use.

## 2. Methods

### 2.1. *Study design and data collection*

The study was observational and retrospective. It was conducted in the teaching hospital of Lille, in the North of France, during two months (January 1st 2020 to February 28th 2020). All computerized wards analyzed by clinical pharmacists were integrated (1,350 beds). Since July 1st 2019, clinical pharmacist use the CDSS PharmaClass®. It is a standalone tool, developed by the Keenturtle company (Paris, France).

For each hospital stay, clinical pharmacist can access patients' EHR through the computerized physician order entry Sillage® (SIB, Rennes, France). Data available for CDSS were prescribed drugs, laboratory results, and administrative data of the patient (age, sex, and hospital ward). Drug dosage and diagnostic codes were not available at the time of the study. All alerts were treated by clinical pharmacist, even redundant alerts (i.e. following a new signature of the prescriptions by physicians).

### 2.2. *Definition of the AKI rules*

We defined two set of rules based on KDIGO criteria. On these criteria, only two consecutive creatinine values could be comparable in CDSS. At first, we created a basic AKI rule. This rule did not change and remained the same during the study, as shown in Figure 1.

The second rule was the improved AKI rule, based on KDIGO criteria with limitations on estimated glomerular filtration rate (eGFR). Additional information regarding eGFR implementation is of major importance for drug dosage adaptation. Throughout the study, this last AKI rule was subject to revision by a group of experts (comprising clinical pharmacists (BD, CR, LR, PO), a physician (JBB), and a pharmacologist (SG)). Over the two months period, improved AKI rule was refined twice (at T1 and T2) by the expert's review based on analysis of alerts, as shown in Figure 1.
<figure><img src="image_3.png" /><figcaption>Figure 1. Construction of the AKI rules through the two months period.</figcaption></figure>

## 2.3. Evaluation of alerts

The analysis of AKI alerts triggered by the CDSS was performed every day of the week (except weekends) by the clinical pharmacist team (18 pharmacists and 8 interns), through the interface of the CDSS. For each alert, the clinical team followed a standardized approach and collected following information: date of analysis, descriptive data of patient (hospital ward and identification number), and the type of rule triggered. Then, they assessed, in a standardized manner, the following items of the alert:

* Technical validity of the alert: an alert was deemed valid when criteria required to trigger the rule were followed (administrative and biological data). If the terms of the rules were correctly followed, alerts were qualified as “technically valid”.

* Pharmaceutically relevance of alert: only technical valid alert could conduct to a pharmaceutical intervention (PI) by clinical pharmacist. A PI was conducted only if the clinical pharmacist found relevant to re-evaluate the medical care following the alert. Alerts leading to a PI were qualified as “pharmaceutically relevant”.

* Acceptation of the PI: pharmacists formulated PI, which was intended for physicians. The PI was deemed to be accepted when the therapeutic or monitoring suggestion was followed by physicians. Alerts were qualified as “medically accepted”.

Statistical analysis were performed using R statistical programming language (version 3.3.3). Qualitative variables were described in terms of frequency and percentage. Numerical variables were described in terms of means and standard deviations (SD).

# 3. Results

During two months, the CDSS provided 1,125 alerts on AKI situations. The number of alerts triggered for the basic AKI rule was 643, and for the improved AKI rule, it was 482 alerts. Results of the alerts are summarized in the Table 1.

<table><caption>**Table 1.** Analysis of alerts from the CDSS by clinical pharmacist according to the 2 sets of rules and during the three periods</caption><thead><tr><th></th><th>Basic AKI rule</th><th>Improved AKI rule</th></tr></thead><tbody><tr><td>1<sup>st</sup> period – 7 days</td><td>KDIGO criteria</td><td>KDIGO criteria and eGFR≤60</td></tr><tr><td>Alerts, N</td><td>92</td><td>81</td></tr><tr><td>Technically valid alert, n (%)</td><td>72 (78.3%)</td><td>66 (81.5%)</td></tr></tbody></table>
<table><tbody><tr><td>Pharmaceutically relevant alert, n (%)</td><td>8 (8.7%)</td><td>9 (11.1%)</td></tr><tr><td>Medically relevant alert, n (%)</td><td>1 (1.1%)</td><td>3 (3.7%)</td></tr><tr><td><strong>2<sup>nd</sup> period – 7 days</strong></td><td><strong>KDIGO criteria</strong></td><td><strong>KDIGO criteria and eGFR≤45</strong></td></tr><tr><td>Alerts, N</td><td>158</td><td>118</td></tr><tr><td>Technically valid alert, n (%)</td><td>147 (93.0%)</td><td>102 (86.4%)</td></tr><tr><td>Pharmaceutically relevant alert, n (%)</td><td>0</td><td>13 (11.0%)</td></tr><tr><td>Medically relevant alert, n (%)</td><td>0</td><td>8 (6.8%)</td></tr><tr><td><strong>3<sup>rd</sup> period – 43 days</strong></td><td><strong>KDIGO criteria</strong></td><td><strong>KDIGO criteria and 15&lt;eGFR≤45</strong></td></tr><tr><td>Alerts, N</td><td>393</td><td>283</td></tr><tr><td>Technically valid alert, n (%)</td><td>353 (89.8%)</td><td>259 (91.5%)</td></tr><tr><td>Pharmaceutically relevant alert, n (%)</td><td>6 (1.5%)</td><td>40 (14.1%)</td></tr><tr><td>Medically relevant alert, n (%)</td><td>4 (1.0%)</td><td>30 (10.6%)</td></tr></tbody></table>

PI: Pharmaceutical Intervention; eGFR: estimated Glomerular Filtration Rate (ml/min/1.73m²)

For the basic AKI rule, the number of technically valid alerts increased, but the rate of relevant alerts (pharmaceutically and medically) was very low. About 10% of irrelevant PI were due to the lack of drugs to be re-evaluated, or because drugs had already been adapted by physician, requiring only monitoring. For the improved AKI rule, the number of technically valid alerts increased with an increase in the pharmaceutically and medically relevant. Main reasons for pharmaceutically irrelevance were either no prescribed drug at risk during kidney injury (i.e. no change in the medical care was needed), or the AKI situation was deemed non relevant and only monitoring was required. When analyzing the third period, we notice that about one alert out of seven (14.1%) of the improved AKI rule led to a PI, and that three quarter of them were accepted by the physicians. PI were more accepted than during the first period (10.6% vs 3.7%), and compared to the basic AKI rule (10.6% vs 1.0%).

During the third period and among the 450 technically valid alerts, 191 (42.4%) alerts were triggered only with the basic AKI rule. These last alerts mainly concerned creatinine values below 1.0 mg/dl, or an eGFR below 15 ml/min/1.73m². Only 6 (3.1%) conducted to a PI and 4 (2.1%) of them were followed up by the physicians.

## 4. Discussion and conclusion

Detection of AKI situation is of importance for drug adaptation. Among alerts conducting to a PI, most were accepted by physicians. Over the 3 periods, the number of discrepancies between basic and improved AKI alerts increased as definition of AKI evolved. We observed an increase in the number of pharmaceutically and medically relevant alerts with the increasingly detailed definition of the improved AKI rule. Moreover, few alerts, triggered from the basic AKI rule led to PI. Our results showed that the improved AKI rule seems sufficient to detect relevant AKI situation. Main AKI situations not detected by this rule were patients with nephrectomy, or patients with a mild AKI but in these situations very few PI were done and only monitoring was recommended. Detection of AKI situation was not as easy as it sounds, and a step by step method was needed. Through three periods of analysis by experts, it was deemed difficult to integer explicit recommendation directly among CDSS. Rule needs to be refined by experts according to the objective of the alerts (drug adaptation, prevention of adverse event). The study revealed the amount of time needed to evaluate explicit rule
as many alerts triggered during the two months. Some criteria were too large and resulted
in too much alert fatigue of clinical pharmacist with no PI to physicians. The redundancy
of alerts has also led to an increase in alerts and a decrease in the rate of PI, although the
pharmacist analyzed actually all these alerts. Using supervised machine learning for
optimization or for creation of rules demonstrated promising results in improving
specificity of alerts [7,8]. During the first period, the number of alerts changed
significantly, from 7 to 40 alerts to be assessed per day (all rules included). At the end of
February, the number of alerts decreased to 22 alerts per day. Studies highlighted the
problem of over-alerting encountered with irrelevant rules implemented into CDSS and
the risk of fatigue of health professionals [4]. To refine rules, we have analyzed feedback
of health professionals and their justification for overriding alerts. Another interesting
methodology to enhance effectiveness of alerts is an autonomous reevaluation of alerts
[8]. Finally, involvement of clinical pharmacists is essential in the integration of alerts
in CDSS and to reduce inappropriate alerts [5,6]. In most studies, CDSS aimed at alerting
physicians on risk situations, but alerts to clinical pharmacist and their recommendations
may be acceptable [9]. In fact, they translate data of the automated alert into an
information to physicians.

In conclusion, the study showed that integration of explicit criteria into CDSS is
feasible but qualification of AKI situations rules cannot be carried exclusively by CDSS
and requires an expert opinion and evaluation. A multi-disciplinary team for the
definition of rules into CDSS and the analysis of alert by clinical pharmacists are two
main factors to be relevant. Indeed, the translation of an epidemiological situation of AKI
into clinical situation is not simple. Continuous re-evaluation of rules implemented into
CDSS can avoid over-alerting and optimize the security of drug prescription. As CDSS
are real-time tools and aimed to be more centered on the patient, the perspective of this
project is creation of a more specific rule with drugs to re-evaluate during AKI situation.

References

[1] Jia P, Zhang L, Chen J, Zhao P, Zhang M. The Effects of Clinical Decision Support Systems on Medication Safety: An Overview. PLOS ONE 2016;11:e0167683.

[2] de Wit HA, Mestres Gonzalvo C, Cardenas J, Derijks HJ, Janknegt R, van der Kuy PH, Winkens B, Schols JM. Evaluation of clinical rules in a standalone pharmacy based clinical decision support system for hospitalized and nursing home patients. Int J Med Inform. 2015 Jun;84(6):396-405.

[3] Desmedt S, Spinewine A, Jadoul M, Henrard S, Wouters D, Dalleur O. Impact of a clinical decision support system for drug dosage in patients with renal failure. Int J Clin Pharm 2018;40:1225-33.

[4] Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf 2014;23:773-80.

[5] Pontefract SK, Coleman JJ, Vallance HK, Hirsch CA, Shah S, Marriott JF, Redwood S. The impact of computerised physician order entry and clinical decision support on pharmacist-physician communication in the hospital setting: A qualitative study. PLoS One. 2018 Nov 16;13(11):e0207450.

[6] Kellum JA, Lameire N. Diagnosis, evaluation, and management of acute kidney injury: a KDIGO summary (Part 1). Crit Care 2013;17:204.

[7] Corny J, Rajkumar A, Martin O, Dode X, Lajonchère JP, Billuart O, Bézie Y, Buronfosse A. A machine learning-based clinical decision support system to identify prescriptions with a high risk of medication error. J Am Med Inform Assoc. 2020 Nov 1;27(11):1688-1694. [8] Díaz AS, et al. Drug prescribing in patients with renal impairment optimized by a computer-based, semi-automated system. Int J Clin Pharm 2013;35:1170-7.

[9] Choi KS, Lee E, Rhie SJ. Impact of pharmacists' interventions on physicians' decision of a knowledge-based renal dosage adjustment system. Int J Clin Pharm 2019;41:424-33.