# Clinical Decision Support System to Detect the Occurrence of Ventilator-Associated Pneumonia in Pediatric Intensive Care

Jerome Rambaud, Masoumeh Sajedi, Sally Al Omar, Maryline Chomtom, Michael Sauthier, Simon de Montigny, Philippe Jouvet

► To cite this version:

Jerome Rambaud, Masoumeh Sajedi, Sally Al Omar, Maryline Chomtom, Michael Sauthier, et al. Clinical Decision Support System to Detect the Occurrence of Ventilator-Associated Pneumonia in Pediatric Intensive Care. Diagnostics, 2023, 13 (18), pp.2983. 10.3390/diagnostics13182983. inserm-04225808

HAL Id: inserm-04225808

https://inserm.hal.science/inserm-04225808

Submitted on 3 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Article

# Clinical Decision Support System to Detect the Occurrence of Ventilator-Associated Pneumonia in Pediatric Intensive Care

Jerome Rambaud<sup>1,2,*</sup> <sup>1</sup>, Masoumeh Sajedi<sup>3</sup>, Sally Al Omar<sup>3</sup>, Maryline Chomtom<sup>4</sup>, Michael Sauthier<sup>1,5</sup>, Simon De Montigny<sup>3,5</sup> and Philippe Jouvet<sup>1</sup>

1. Pediatric Intensive Care Unit, Sainte-Justine Hospital, Montreal, QC H3T 1C5, Canada; michael.sauthier.med@ssss.gouv.qc.ca (M.S.); philippe.jouvet.med@ssss.gouv.qc.ca (P.J.)

2. Pediatric and Neonatal Intensive Care Unit, Armand-Trousseau Hospital, Sorbonne University, 75012 Paris, France

3. Research Center, Sainte-Justine Hospital, Montreal, QC H3T 1C5, Canada; masoumeh.sajedi.hsj@ssss.gouv.qc.ca (M.S.); sally.alomar@hotmail.com (S.A.O.); simon.de.montigny@umontreal.ca (S.D.M.)

4. Pediatric Intensive Care Unit, Caen University Hospital, 14000 Caen, France; chomtom-m@chu-caen.fr

5. School of Public Health, Montréal University, Montreal, QC H2X 3E4, Canada

* Correspondence: jerome.rambaud@aphp.fr; Tel.: +33-1-71-73-85-27

<figure><img src="image_2.png" /></figure>

check for
updates

**Citation:** Rambaud, J.; Sajedi, M.; Al Omar, S.; Chomtom, M.; Sauthier, M.; De Montigny, S.; Jouvet, P. Clinical Decision Support System to Detect the Occurrence of Ventilator-Associated Pneumonia in Pediatric Intensive Care. *Diagnostics* **2023**, *13*, 2983. https://doi.org/10.3390/diagnostics13182983

Academic Editor: Zhongheng Zhang

Received: 24 August 2023

Revised: 15 September 2023

Accepted: 15 September 2023

Published: 18 September 2023

<figure><img src="image_3.png" /></figure>

**Copyright:** © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

**Abstract:** Objectives: Ventilator-associated pneumonia (VAP) is a severe care-related disease. The Centers for Disease Control defined the diagnosis criteria; however, the pediatric criteria are mainly subjective and retrospective. Clinical decision support systems have recently been developed in healthcare to help the physician to be more accurate for the early detection of severe pathology. We aimed at developing a predictive model to provide early diagnosis of VAP at the bedside in a pediatric intensive care unit (PICU). Methods: We performed a retrospective single-center study at a tertiary-care pediatric teaching hospital. All patients treated by invasive mechanical ventilation between September 2013 and October 2019 were included. Data were collected in the PICU electronic medical record and high-resolution research database. Development of the clinical decision support was then performed using open-access R software (Version 3.6.1®). Measurements and main results: In total, 2077 children were mechanically ventilated. We identified 827 episodes with almost 48 h of mechanical invasive ventilation and 77 patients who suffered from at least one VAP event. We split our database at the patient level in a training set of 461 patients free of VAP and 45 patients with VAP and in a testing set of 199 patients free of VAP and 20 patients with VAP. The Imbalanced Random Forest model was considered as the best fit with an area under the ROC curve from fitting the Imbalanced Random Forest model on the testing set being 0.82 (95% CI: (0.71, 0.93)). An optimal threshold of 0.41 gave a sensitivity of 79.7% and a specificity of 72.7%, with a positive predictive value (PPV) of 9% and a negative predictive value of 99%, and with an accuracy of 79.5% (95% CI: (0.77, 0.82)). Conclusions: Using machine learning, we developed a clinical predictive algorithm based on clinical data stored prospectively in a database. The next step will be to implement the algorithm in PICUs to provide early, automatic detection of ventilator-associated pneumonia.

**Keywords:** pneumonia; ventilator associated; clinical decision system; PICU

## 1. Introduction

Ventilator-associated pneumonia (VAP) is a common and severe complication in intensive care units. VAP, as a care-related complication leads to a worsening prognosis for the affected patients and its early diagnosis remain an ongoing challenge in intensive care. In an attempt to enhance VAP detection, the Centers for Disease Control (CDC) issued diagnosis criteria allowing the identification of VAP after 48h of clinical alteration (defined by worsening gas exchange, fever >38 °C or hypothermia, leukocytosis >15,000/mm³ or leukopenia <4000/mm³, new onset of purulent sputum, apnea or tachypnea, wheezing/rales/rhonchi,
cough and bradycardia <100/min or tachycardia >170/min) [1]. However, delays in VAP diagnosis and, to some extent, in initiating anti-infectious therapy are observed and associated with worse outcomes [2–4]. Furthermore, subjective criteria included in the CDC pediatric definition for VAP results in a variability of VAP diagnosis and incidence (changes in the appearance and amount of sputum, worsening of an existing cough) [5–7]. To help physicians to prospectively diagnose VAP, the CDC developed the concept of Ventilator-Associated Events (VAE) in adults, but children have long been excluded from this definition [1]. It is usual that for adult recommendations, children are excluded mainly because of physiological differences between populations (normal respiratory parameters for an adult are very different from those of a child). Cirulis et al. [8] proposed a pediatric modified VAE definition. Chomton et al. [9] evaluated the pediatric modified VAE definition to detect VAP, but the sensitivity (66%) to identify this ICU-related complication remained disappointing.

In recent years, the number of publications dealing with the development of computerized clinical decision support systems (CDSS) to improve disease diagnosis increased and was shown to be useful for several disease in ICUs [10–14]. The emergence of high-resolution databases supports these developments [15] which allow for a precise and continuous analysis of clinical and biological parameters. Leisman et al. [16] recently reported several recommendations for the development and reporting of predictive models. They identified two categories of predictive models: (1) clinical prediction models for bedside use, and (2) other prediction models intended for deployment across populations for research, benchmarking, and administrative purposes. The usefulness of CDSS had already been highlighted by Mack et al. [17] but no reports on VAP are available currently. To that effect, our project has been developed with the main objective of developing a predictive model to provide early diagnosis of VAP at the bedside in a pediatric intensive care unit (PICU).

## 2. Materials and Methods

This single-center retrospective study was performed using the data collected in the PICU electronic medical record (Intelligence Critical Care and Anesthesia (ICCA®); Philips Medical, version F0.1) of a tertiary-care pediatric teaching hospital (Sainte-Justine Hospital, Montréal, QC, Canada). To improve data quality, ICCA® was configured with drop-down menus and critical values alerts. Furthermore, all data entered in ICCA® benefited from a medically-endorsed validation.

The hospital database was queried using SQL Server Management Studio 18® (Microsoft, Redmond, WA, USA) to select patients who were aged from 1 day to 18 years at PICU admission and were mechanically ventilated for more than 48 h, between September 2013 and October 2019. We analyzed the first 30 days of invasive mechanical ventilation.

During the first step of the study, all medical files were reviewed by two senior pediatric intensive care experts (JR and PJ) to classify patients into two groups: VAP patients and free-of-VAP patients. VAP was defined according to the 2021 CDC criteria [1]: The 1st context criteria: invasive mechanical ventilation for more than 48 h, 2nd radiological criteria: new or progressive and persistent infiltrate/consolidation/cavitation, 3rd clinical criteria: worsening gas exchange, fever >38 °C or hypothermia, leukocytosis >15,000/mm³ or leukopenia <4000/mm³, new onset of purulent sputum, apnea or tachypnea, wheezing/rales/rhonchi, cough and bradycardia <100/min or tachycardia >170/min.

The second step of the study consisted in the extraction of data coming from the electronic medical record (ICCA®, Philips, Toronto, ON, Canada) and high-resolution database (database collecting and storing data from medical devices in real time) [15]. The queried data were date, time, weight (kg), white blood cell count (/mm³), neutrophil count (/mm³), partial pressure of carbon dioxide (PaCO₂ in mmHg), partial pressure of oxygen (PaO₂ in mmHg), inspired fraction of oxygen (FiO₂ in %), positive end-expiratory pressure (PEEP in cmH₂O), peak inspiratory pressure (PIP in cmH₂O), mean airway pressure (MAwP in cmH₂O), respiratory rate (/rpm), tidal volume (mL), subjective amount
of respiratory tract secretion (0, +, ++, +++) oxygenation (OI) and oxygen saturation index (OSI) [18], calculated pulmonary dynamic compliance (in barometric ventilation mode: tidal volume/(PIP-PEEP); and in volumetric ventilation mode: tidal volume/(peak pressure-PEEP)). We also gathered PIM 2 [19] and PELOD-2 scores [20,21].

***Data formatting.*** The data was formatted using R (version 3.6.1) as a preparation step to train the prediction models based on different algorithms.

All times were expressed as a relative duration since ICU admission.

***Data cleaning and Missing data.*** Incoherent data were identified and corrected according to the scheme described in Supplementary Data S1 “Data Cleaning”. Variables consisting of data streams of continuous values were imputed following the last observation carried forward method. For missing data at the beginning of the stream, the first valid observation was carried backward.

***Segmenting Variables in Time Blocks.*** The variables data streams were first segmented into time blocks of 6 h and then for each variable the median (mode for the discrete variable) was calculated over each 6 h time block to avoid aberrant or missing data. Then, the 6 h blocks were aggregated into 48 h time blocks. We chose to aggregate into 48 h time blocks to be as close as possible to the actual VAP timing definition. For each variable, two columns were generated. One consisted of the first non-missing value among the 6 h time blocks and the other one the last non-missing value among the 6 h time blocks, if there was any, in each 48 h time block (if there was no observation, the data was considered missing). For the development of the algorithms, for each variable, the first non-missing values and the actual difference or relative change of the values of the two columns were considered (more details are available in Supplementary Data S2 “Segmenting variables in time blocks”).

***Stratified train-test split at a patient level.*** VAP patients and non-VAP patients were split into the training set (70% of each class) and the testing set (remaining 30% of each class). Since some patients had more than one stay in the PICU, all stays of a patient in the training set were kept in the training set (and the same for patients in the testing set). All details for the train-test split are available in Supplementary Data S3 “train-test split”.

***Imputation.*** Preliminary inspection of the dataset showed that around 50% of data was missing for the variables “pulmonary dynamic compliance” and “minute ventilation”. Missing values imputation in the training dataset was performed by ‘randomForest’ (v4.6-14) with the function ‘rflmpute’ [22]. The imputed values were the weighted average of the non-missing observations, where the weights were the proximities from randomForest. For data in the testing set, the missing values in each variable were replaced by the mean of the imputed values for the variables with missing values in the training set (more details are available in Supplementary Data S4 “Imputation”).

***Predictive models.*** We applied six different learning algorithms to generate predictive models. The algorithms were: Random Forest with the function ‘rfsrc’ and error rate as the measure of performance, Imbalanced Random Forest with the function ‘imbalanced’ and G-means as the measure of performance, Stepwise Regression and Random Forest using 5-fold cross validation (5-CV) with the ‘train’ function; ‘glmStepAIC’ and ‘rf’ methods and accuracy were used to select the optimal model using the largest value [23]. Finally, we implemented Elastic Net Regression (5-CV) and Weighted Elastic Net Regression (5-CV) with the ‘glmnet’ method and ROC was used to select the optimal model using the largest value. The hyperparameters for the Random Forest, Imbalanced Random Forest and stepwise regression (5-CV) algorithms were ‘ntree’ (number of trees used at the tuning step) and ‘mtry’ (number of variables randomly selected as candidates for the division of a node) [24]. The parameters in Elastic Net regression were alpha, which controls the relative balance between the lasso and ridge regularization, and lambda, which controls the amount of the penalty. All these models used readily available implementations in R [25,26]. Here, cross-validation was performed inside the training set only (more details are available in Supplementary Data S5 “Predictive models”).

***Performance measure and model choice.*** Models resulting from the different algorithms were evaluated, at the level of 48 h time blocks, on the train and the test set by calculating
their AUC score and by determining classification thresholds reaching predetermined levels of sensitivity (80%, 85%, 90%, 95%). The final model was chosen based on the capacity to [1] maximize specificity under these sensitivity levels, and [2] generalize the sensitivity and specificity from the test set. The area under the ROC curve (AUC) was considered as the primary measure of performance to choose the best model.

*Per patient validation.* The final model was evaluated on its capacity to correctly assess the infection status of patients over time. The predictions' results obtained after setting different classification thresholds were taken. The number of patients with accurate predictions (i.e., predicted class = observed VAP status) and inaccurate predictions (i.e., predicted class ≠ observed VAP status) were computed over time. The number of patients for whom the predictions contained at least one error were identified. We looked at the accuracy of predictions by stratifying patients into two groups. We identified the patients for whom the predictions contained at least one error for each subgroup. The global error rates were calculated for each subgroup.

### Statistics

Development of the clinical decision support was performed using open-access R software, Version 3.6.1® (R Foundation for Statistical Computing, Vienna, Austria). Statistical analysis of patients' characteristics was performed using Prism X® software (version 7.05) (GraphPad Inc. San Diego, CA, USA). Kolmogorov analysis was performed to test the normal distribution of continuous variables. Population description used categorical variables expressed as frequency with corresponding proportion and quantitative variables presented as mean and standard deviation. Performance evaluation was conducted using ROC curves, AUC and their confidence intervals, and derived measures of sensitivity and specificity. The ethical committee of Sainte-Justine University hospital approved the study and waived the need for informed consent given the retrospective design.

The Saint-Justine ethical committee approved the study as a retrospective study and waived the need for written consent (n°2020-2454).

## 3. Results

### 3.1. General Description of the Population

A total of 5153 children had been hospitalized in Saint-Justine PICU during the study period of which 40% (2077) were mechanically ventilated and 1235 episodes with more than 48 h of mechanical invasive ventilation were identified (Figure 1). Seventy-seven patients had at least one VAP event. Seventy-eight VAP events (6%) were diagnosed by two experts. The patients' general characteristics are described in Table 1.

<table><caption>**Table 1.** Population characteristics.</caption><thead><tr><th>Population Characteristics</th><th>Global Population (N: 827)</th><th>VAP Patients (N: 77)</th><th>No VAP Patients (N: 750)</th><th>p:</th></tr></thead><tbody><tr><td>Weight (kg)</td><td>15.8 ± 1.6</td><td>20.99 ± 2.7</td><td>15.25 ± 0.7</td><td>0.01</td></tr><tr><td>Age (days)</td><td>1308 ± 1904</td><td>1806 ± 250</td><td>1256 ± 69</td><td>0.02</td></tr><tr><td>Gender male (%)</td><td>475 (57%)</td><td>41 (53%)</td><td>434 (58%)</td><td>0.4</td></tr><tr><td>Pelod 2 score</td><td>10.1 ± 4.8</td><td>10.4 ± 0.6</td><td>9.9 ± 0.2</td><td>0.47</td></tr><tr><td>Pelod 2 mortality risk (%)</td><td>0.3 ± 0.3</td><td>0.3 ± 0.1</td><td>0.2 ± 0.01</td><td>0.15</td></tr><tr><td>Bronchoscopie (%)</td><td>70 (8%)</td><td>14 (18%)</td><td>56 (8%)</td><td>0.04</td></tr><tr><td>Neuromuscular blocker (%)</td><td>279 (34%)</td><td>43 (55%)</td><td>236 (31%)</td><td>&lt;0.0001</td></tr><tr><td>Mechanical Ventilation duration (days)</td><td>12.5 ± 30.9</td><td>29.3 ± 5.1</td><td>10.9 ± 1.5</td><td>&lt;0.0001</td></tr><tr><td>PICU length of stay (days)</td><td>26.1 ± 52.5</td><td>48.3 ± 7.1</td><td>23.4 ± 1.8</td><td>&lt;0.0001</td></tr><tr><td>Survival rate (%)</td><td>740 (90%)</td><td>65 (84%)</td><td>675 (90%)</td><td>0.16</td></tr></tbody></table>

PICU: Pediatric intensive care unit; VAP: Ventilator-associated pneumonia.
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Flow chart. VAP: Ventilator-associated event.</figcaption></figure>

Patients with less than 4 days of mechanical ventilation were removed (see Supplementary Data S2 "Segmenting variables") to achieve 811 episodes of invasive mechanical ventilation. The training set (70% of each class) and testing set (remaining 30% of each class), resulted in a training set of 461 patients free of VAP and 45 patients with VAP and in a testing set of 199 patients free of VAP and 20 patients with VAP. Since some patients had more than one stay in the ICU, there could be different events for the same patient. The training set thus had 513 stays with no VAP event and 45 stays with a VAP event, and the testing set had 231 stays with no VAP event and 22 stays with a VAP event. The segmenting of variables in 48 h non-overlapping time blocks generated, from these datasets, 1852 time blocks free of VAP and 45 time blocks with VAP in the training set, and 788 time blocks free of VAP and 22 time blocks with VAP in the testing set.

We observed similar characteristics in the train and test groups (Table 2).

<table><caption>**Table 2.** Train and test groups' characteristics.</caption><thead><tr><th>Train and Test<br>Groups Characteristics</th><th>Test Group (n: 261)</th><th>Train Group (n: 572)</th><th>p:</th></tr></thead><tbody><tr><td>Weight (kg)</td><td>16.9 ± 1.3</td><td>15.6 ± 0.8</td><td>0.40</td></tr><tr><td>Age (days)</td><td>1387 ± 129</td><td>1268 ± 84</td><td>0.43</td></tr><tr><td>Gender male, (n, %)</td><td>146 (60)</td><td>284 (58)</td><td>0.69</td></tr><tr><td>Pelod 2 score</td><td>10.4 ± 0.2</td><td>9.7 ± 0.5</td><td>0.16</td></tr></tbody></table>
<table><caption>**Table 2.** *Cont.*</caption><thead><tr><th>Train and Test<br>Groups Characteristics</th><th>Test Group (n: 261)</th><th>Train Group (n: 572)</th><th>p:</th></tr></thead><tbody><tr><td>Pelod 2 mortality risk (%)</td><td>0.3 ± 0.1</td><td>0.2 ± 0.1</td><td>0.13</td></tr><tr><td>Proportion of VAP patients (n, %)</td><td>25 (10)</td><td>50 (10)</td><td>0.99</td></tr><tr><td>Length of mechanical ventilation<br>before VAP (days)</td><td>9.9 ± 2.7</td><td>9.6 ± 1.9</td><td>0.66</td></tr><tr><td>Length of mechanical ventilation<br>duration (days)</td><td>12.1 ± 1.6</td><td>11.2 ± 1.1</td><td>0.64</td></tr><tr><td>PICU length of stay (days)</td><td>21.3 ± 2.4</td><td>22.3 ± 2.0</td><td>0.81</td></tr></tbody></table>

Pelod: Pediatric logistic organ dysfunction, PICU: Pediatric intensive care unit; VAP: Ventilator-associated pneumonia.

## 3.2. Missing Data

We observed two missing values for "sf ratio" and "oxygen saturation index (OSI)" in the test set (0.1% of total observations). For the variable "pulmonary dynamic compliance" the proportion of missing values in the train and test sets were 0.49 and 0.54, respectively. For the variable "minute ventilation", the proportion of missing values in the train and test sets were 0.49 and 0.54, respectively.

## 3.3. Results of Training Algorithm

The Imbalanced Random Forest model was considered as the best fit with an area under the ROC curve of 0.86 from the train set.

Thresholds and specificities corresponding to the predetermined levels of sensitivity are presented in Table 3. Variable importance obtained from the Imbalanced Random Forest model are presented in Figure 2.

<table><caption>**Table 3.** Imbalanced Random Forest model. Threshold and specificity from predetermined sensitivity for the train set.</caption><thead><tr><th>Threshold</th><th>Specificity</th><th>Sensitivity</th></tr></thead><tbody><tr><td>0.41</td><td>0.79</td><td>0.80</td></tr><tr><td>0.29</td><td>0.64</td><td>0.87</td></tr><tr><td>0.25</td><td>0.58</td><td>0.91</td></tr><tr><td>0.22</td><td>0.52</td><td>0.96</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>**Figure 2.** Variable importance used in the clinical decision system.</figcaption></figure>
### 3.4. Performance on Test Dataset

The area under the ROC curve from fitting the Imbalanced Random Forest model on the test set was 0.82 (95% CI: (0.71, 0.93)) (Figure 3).

<figure><img src="image_6.png" /><figcaption>**Figure 3.** ROC Curve. Black curve represent the efficiency of the training of the algorithm on 2/3 of the dataset. Red curve represents the efficiency of the test of the algorithm on the rest of the data set.</figcaption></figure>

The specificity and sensitivity obtained after setting different classification thresholds are presented in Table 4. An optimal threshold of 0.41 gave a sensitivity of 79.7% and a specificity of 72.7%, with a positive predictive value (PPV) of 9% and a negative predictive value of 99%, with an accuracy of 79.5% (95% CI: (0.77, 0.82)).

<table><caption>**Table 4.** Imbalanced Random Forest model. Sensitivity and specificity for the test set corresponding to different thresholds.</caption><thead><tr><td>Threshold</td><td>Specificity</td><td>Sensitivity</td></tr></thead><tbody><tr><td>0.41</td><td>0.797</td><td>0.73</td></tr><tr><td>0.28</td><td>0.66</td><td>0.77</td></tr><tr><td>0.25</td><td>0.59</td><td>0.77</td></tr><tr><td>0.22</td><td>0.53</td><td>0.82</td></tr></tbody></table>

### 3.5. Per Patient Validation

Performance of the final model was evaluated over different time periods. Time periods were defined starting from the first time block and going up to a given time block in the future. The confusion matrices for all the time periods were constructed. False positive rates (FPR), true positive rates (TPR), and area under the curve (AUC) were calculated. The results are presented in Figure 4. The procedure is explained in detail in Supplementary Data S6 "Per patient validation".

The global error rate is presented in Table 5. We observed a lower error rate for patients with at most three time blocks of observations, compared to the ones with at least four time blocks of observations.
<figure><img src="image_7.png" /><figcaption>**Figure 4.** False positive rate and true positive rate over different time periods for different thresholds. Th Default: default threshold of the model; Th 80%: threshold correspond to the 80% sensitivity; Th 85%: threshold correspond to the 85% sensitivity.</figcaption></figure>

<table><caption>**Table 5.** Error rates (%) for predicted classes.</caption><thead><tr><th rowspan="2"></th><th colspan="3">G1</th><th colspan="3">G2</th></tr><tr><th>ER.Pred</th><th>ER.Pred.th80</th><th>ER.Pred.th85</th><th>ER.Pred</th><th>ER.Pred.th80</th><th>ER.Pred.th85</th></tr></thead><tbody><tr><td>All</td><td>11.56</td><td>19.60</td><td>31.66</td><td>79.59</td><td>83.67</td><td>95.92</td></tr><tr><td>VAP</td><td>23.08</td><td>23.08</td><td>23.08</td><td>66.67</td><td>66.67</td><td>88.89</td></tr><tr><td>NoVAP</td><td>10.75</td><td>19.35</td><td>32.26</td><td>82.50</td><td>87.50</td><td>97.50</td></tr></tbody></table>

G1: Patients with at most 3 time blocks of observations; G2: Patients with at least 4 time-blocks of observations. E.Pred: Error rate for prediction; E.Pred.th80%: Error rate for prediction with threshold correspond to the 80% sensitivity; E.Pred.th85%: Error rate for prediction with threshold correspond to the 85% sensitivity.

## 4. Discussion

Using an electronic medical record, an algorithm supporting clinicians in the early diagnosis of ventilator-associated pneumonia in PICU had a sensitivity of 80% and specificity of 73%, with the threshold of 0.41. To date, it is the most accurate sensitivity achieved by a CDSS system to provide early detection of VAP.

Ventilator-associated pneumonias is a severe health care disease [2,27,28]. To improve the delay and accuracy of this challenging diagnosis, Cirulis et al. [8] evaluated the accuracy of adults' ventilator-associated events (VAE) to early diagnose pediatric VAP and developed modified pediatric criteria for VAE (increase in FiO₂ by 20% or PEEP by 2 cm H₂O sustained for more than one day). VAE and modified pediatric VAE both had a disappointing sensitivity of 23% and 56% for Cirulis et al. [8] and 56% and 66% for Chomton et al. [9], respectively. Our algorithm was based on machine learning methods and improved the sensitivity in this study and could be implemented to screen in real-time patient's data to provide early detection of VAP in children. The prediction of the test set using the Imbalanced Random Forest model is stored in a file and is available on Github [29].

Implementation of a clinical decision system to help physicians is a promising technology aimed at helping the physician to take medical decision [10,30], to analyze chest X-rays [31], or to increase diagnosis sensitivity [32]. The development methodology starts with a retrospective classification of analyzed patients to define whether they develop
the studied conditions (e.g.,VAP). This step is crucial to develop an accurate algorithm and rely on the quality of the classification method. In a large review of published CDSS, Ostropolets et al. [33] highlight that only one manuscript addressed confounding and bias due to misclassification. Our classification methodology included all the relevant data from the electronic medical record clinically collected and is the best accuracy that can be obtained currently.

In addition to the classification methodology, the main strength of this study includes the use of continuous vital signs and the ventilatory parameters monitoring database, limiting the number of missing data and allowing the use of the algorithm in real time in the future [15]. The variables extracted from this monitoring included the OSI ratio, the variation of pulmonary compliance, minute ventilation, and ventilatory median pressures. However, the algorithm identified the variation of PEEP during the last 48 h preceding the VAP as the most important criteria as suggested by the CDC definition. Nevertheless, the variation of the ventilatory mean airway pressure was the second most important variable. This result seems crucial because the ventilatory mean airway pressure that not only includes PEEP but also the PIP, I/E ratio and instantaneous gas flow is not included in the CDC diagnosis criteria for VAP.

Nonetheless, we noticed that our algorithm has a better efficacy to diagnose early VAP (before day 6 of the PICU stay) versus late VAP (after day 6 of the PICU stay), with the error rate in prediction of 23.08% vs. 66.67%, respectively. We can hypothesize that the more time the patient stays in the PICU, the more discrete are the variations to be detected due to the potential alteration of the patient's condition.

This study has several strengths. First, this is the first study with a CDSS system reaching over 80% sensitivity. Second, despite this being a single-center study, we report one of the largest number of patients included in a study in children. Finally, we report the highest sensitivity and specificity to diagnose VAP.

Despite these promising results, this work suffers from several limitations. First, the invasive procedures were not considered in our algorithm (bronchoscopy, transportations) due to the lack of data concerning the timing between these procedures and the VAP. Second, data on the reason for invasive mechanical ventilation were not reported in all medical files, although it is well known that brain injury and neurological disorder with impaired swallowing predispose more to pneumonia. Third, the treatment of missing data was conducted using data-focused approaches (last observation carried forward for missing data mid-stream, first observation carried backward for data missing at the beginning of a stream) which did not model the missing data process; the classifications between the VAP and non-VAP patients were retrospectively performed which may have resulted in some misinterpretation of the clinical data. Fourth, for generalizability, a prospective validation of the algorithm in several PICU needs to be conducted.

**5. Conclusions**

We developed the first clinical predictive system dedicated to VAP diagnosis in PICUs using a high-fidelity database. The implementation of such an algorithm in PICUs could allow physicians to be alerted early in cases of respiratory function impairment and to decide whether to perform respiratory tract analysis and start anti-infective treatment. Although this algorithm achieves a promising sensitivity and specificity level, it is still lacking power and cannot be implanted in PICUs. Additionally, it still needs to be prospectively validated in other PICUs to confirm its reproductivity and external power.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/diagnostics13182983/s1, Supplementary Data S1 “Data Cleaning”; Supplementary Data S2 “Segmenting variables in time blocks”; Supplementary Data S3 “train-test split”; Supplementary Data S4 “Imputation”; Supplementary Data S5 “Predictive models”; Supplementary Data S6 “Per patient validation”.
**Author Contributions:** Conceptualization, M.C., P.J. and J.R.; Methodology, J.R., S.A.O. and M.S. (Michael Sauthier); Software, S.A.O., M.S. (Michael Sauthier) and S.D.M.; Formal analysis, M.S. (Michael Sauthier) and S.D.M.; Investigation, J.R.; Data curation, M.S. (Masoumeh Sajedi) and S.D.M.; Writing—original draft, J.R.; Writing—review & editing, J.R. and M.C.; Supervision, P.J.; Project administration, P.J.; Funding acquisition, P.J. All authors have read and agreed to the published version of the manuscript.

**Funding:** J.R. received a studentship from the scholarship supplement program from the Quebec Respiratory Health Research Network. This work was supported in part by the Quebec Respiratory Health Research Network. J.P.'s scientific research funds from the Canadian Foundation for Innovation, Fonds de Recherche Québec, Quebec Ministry of Health and Sainte-Justine Hospital.

**Institutional Review Board Statement:** The Saint-Justine ethical committee approved the study as a retrospective study and waived the need for written consent (n°2020-2454).

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are available on demand.

**Acknowledgments:** The authors want to thank the research center of Sainte-Justine University hospital and for their support.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Abbreviations**

<table><tr><td>AUC</td><td>area under the curve</td></tr><tr><td>CDC</td><td>centers for disease control</td></tr><tr><td>CDSS</td><td>clinical decision support systems</td></tr><tr><td>FiO<sub>2</sub></td><td>inspired fraction of oxygen</td></tr><tr><td>FPR</td><td>false positive rates</td></tr><tr><td>ICCA</td><td>Intelligence Critical Care and Anesthesia®</td></tr><tr><td>MAwP</td><td>mean airway pressure</td></tr><tr><td>OI</td><td>oxygenation index</td></tr><tr><td>OSI</td><td>oxygenation and saturation index</td></tr><tr><td>PaCO<sub>2</sub></td><td>partial pressure of carbon dioxide</td></tr><tr><td>PaO<sub>2</sub></td><td>partial pressure of oxygen</td></tr><tr><td>PELOD-2</td><td>pediatric logistic organ dysfunction score</td></tr><tr><td>PEEP</td><td>positive end-expiratory pressure</td></tr><tr><td>PICU</td><td>pediatric intensive care unit</td></tr><tr><td>PIM 2</td><td>pediatric index of mortality</td></tr><tr><td>PIP</td><td>peak inspiratory pressure</td></tr><tr><td>ROC</td><td>receiving operator curve</td></tr><tr><td>TPR</td><td>true positive rates</td></tr><tr><td>VAE</td><td>ventilator-associated event</td></tr><tr><td>VAP</td><td>ventilator-associated pneumonia</td></tr></table>

**References**

1. Center for Disease Control. 2018. Available online: https://www.cdc.gov/nhsn/PDFs/pscManual/6pscVAPcurrent.pdf (accessed on 14 September 2023).

2. Chastre, J.; Fagon, J.-Y. Ventilator-associated pneumonia. Am. J. Respir. Crit. Care Med. **2002**, *165*, 867–903. [CrossRef] [PubMed]

3. Gutiérrez, J.M.M.; Borromeo, A.R.; Dueño, A.L.; Paragas, E.D.; Ellasus, R.O.; Abalos-Fabia, R.S.; Abriam, J.A.; Sonido, A.E.; Hernandez, M.A.; Generale, A.J.A.; et al. Clinical epidemiology and outcomes of ventilator-associated pneumonia in critically ill adult patients: Protocol for a large-scale systematic review and planned meta-analysis. Syst. Rev. **2019**, *8*, 180. [CrossRef] [PubMed]

4. Papazian, L.; Klompas, M.; Luyt, C.-E. Ventilator-associated pneumonia in adults: A narrative review. Intensive Care Med. **2020**, *46*, 888–906. [CrossRef]

5. Tullu, M.S.; Balasubramanian, P. Ventilator-Associated Pneumonia in Pediatric Intensive Care Unit: Correspondence. Indian J. Pediatr. **2015**, *82*, 662–663. [CrossRef]

6. Shaath, G.A.; Jijeh, A.; Faruqui, F.; Bullard, L.; Mehmood, A.; Kabbani, M.S. Ventilator-associated pneumonia in children after cardiac surgery. Pediatr. Cardiol. **2014**, *35*, 627–631. [CrossRef]
7. Ericson, J.E.; McGuire, J.; Michaels, M.G.; Schwarz, A.; Frenck, R.; Deville, J.G.; Agarwal, S.; Bressler, A.M.; Gao, J.; Spears, T.; et al. Hospital-acquired Pneumonia and Ventilator-associated Pneumonia in Children: A Prospective Natural History and Case-Control Study. *Pediatr. Infect. Dis. J.* **2020**, *39*, 658–664. [CrossRef] [PubMed]

8. Cirulis, M.M.; Hamele, M.T.; Stockmann, C.R.; Bennett, T.D.; Bratton, S.L. Comparison of the New Adult Ventilator-Associated Event Criteria to the Centers for Disease Control and Prevention Pediatric Ventilator-Associated Pneumonia Definition (PNU2) in a Population of Pediatric Traumatic Brain Injury Patients. *Pediatr. Crit. Care Med.* **2016**, *17*, 157–164. [CrossRef]

9. Chomton, M.; Brossier, D.; Sauthier, M.; Vallières, E.; Dubois, J.; Emeriaud, G.; Jouvet, P. Ventilator-Associated Pneumonia and Events in Pediatric Intensive Care: A Single Center Study. *Pediatr. Crit. Care Med.* **2018**, *19*, 1106–1113. [CrossRef] [PubMed]

10. Giannini, H.M.; Ginestra, J.C.; Chivers, C.; Draugelis, M.; Hanish, A.; Schweickert, W.D.; Fuchs, B.D.; Meadows, L.; Lynch, M.; Donnelly, P.J.; et al. A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock: Development, Implementation, and Impact on Clinical Practice. *Crit. Care Med.* **2019**, *47*, 1485–1492. [CrossRef]

11. Chen, C.-H.; Lee, Y.-W.; Huang, Y.-S.; Lan, W.-R.; Chang, R.-F.; Tu, C.-Y.; Chen, C.-Y.; Liao, W.-C. Computer-aided diagnosis of endobronchial ultrasound images using convolutional neural network. *Comput. Methods Programs Biomed.* **2019**, *177*, 175–182. [CrossRef] [PubMed]

12. Roggeveen, L.F.; Guo, T.; Driessen, R.H.; Fleuren, L.M.; Thoral, P.; van der Voort, P.H.J.; Girbes, A.R.J.; Bosman, R.J.; Elbers, P. Right Dose, Right Now: Development of AutoKinetics for Real Time Model Informed Precision Antibiotic Dosing Decision Support at the Bedside of Critically Ill Patients. *Front. Pharmacol.* **2020**, *11*, 646. [CrossRef] [PubMed]

13. Lauritsen, S.M.; Kalør, M.E.; Kongsgaard, E.L.; Lauritsen, K.M.; Jørgensen, M.J.; Lange, J.; Thiesson, B. Early detection of sepsis utilizing deep learning on electronic health record event sequences. *Artif. Intell. Med.* **2020**, *104*, 101820. [CrossRef]

14. Wulff, A.; Haarbrandt, B.; Tute, E.; Marschollek, M.; Beerbaum, P.; Jack, T. An interoperable clinical decision-support system for early detection of SIRS in pediatric intensive care using openEHR. *Artif. Intell. Med.* **2018**, *89*, 10–23. [CrossRef]

15. Brossier, D.; El Taani, R.; Sauthier, M.; Roumeliotis, N.; Emeriaud, G.; Jouvet, P. Creating a High-Frequency Electronic Database in the PICU: The Perpetual Patient. *Pediatr. Crit. Care Med.* **2018**, *19*, e189–e198. [CrossRef] [PubMed]

16. Leisman, D.E.; Harhay, M.O.; Lederer, D.J.; Abramson, M.; Adjei, A.A.; Bakker, J.; Ballas, Z.K.; Barreiro, E.; Bell, S.C.; Bellomo, R.; et al. Development and Reporting of Prediction Models: Guidance for Authors From Editors of Respiratory, Sleep, and Critical Care Journals. *Crit. Care Med.* **2020**, *48*, 623–633. [CrossRef]

17. Mack, E.H.; Wheeler, D.S.; Embi, P.J. Clinical decision support systems in the pediatric intensive care unit. *Pediatr. Crit. Care Med.* **2009**, *10*, 23–28. [CrossRef]

18. DesPrez, K.; McNeil, J.B.; Wang, C.; Bastarache, J.A.; Shaver, C.M.; Ware, L.B. Oxygenation Saturation Index Predicts Clinical Outcomes in ARDS. *Chest* **2017**, *152*, 1151–1158. [CrossRef]

19. Slater, A.; Shann, F.; Pearson, G.; Paediatric Index of Mortality (PIM) Study Group. PIM2: A revised version of the Paediatric Index of Mortality. *Intensive Care Med.* **2003**, *29*, 278–285. [CrossRef] [PubMed]

20. Leteurtre, S.; Duhamel, A.; Salleron, J.; Grandbastien, B.; Lacroix, J.; Leclerc, F.; Groupe Francophone de Réanimation et d'Urgences Pédiatriques (GFRUP). PELOD-2: An update of the PEdiatric logistic organ dysfunction score. *Crit. Care Med.* **2013**, *41*, 1761–1773. [CrossRef]

21. Sauthier, M.; Landry-Hould, F.; Leteurtre, S.; Kawaguchi, A.; Emeriaud, G.; Jouvet, P. Comparison of the Automated Pediatric Logistic Organ Dysfunction-2 Versus Manual Pediatric Logistic Organ Dysfunction-2 Score for Critically Ill Children. *Pediatr. Crit. Care Med.* **2020**, *21*, e160–e169. [CrossRef]

22. Breiman, L. Breiman and Cutler's Random Forests for Classification and Regression, R package version 4.6-14 [Internet]. 2018. Available online: https://cran.r-project.org/web/packages/randomForest/randomForest.pdf (accessed on 14 September 2023).

23. Available online: https://cran.r-project.org/web/packages/randomForestSRC/randomForestSRC.pdf (accessed on 14 September 2023).

24. Chen, C.; Liaw, A.; Breiman, L. *Using Random Forest to Learn Imbalanced Data*; University of California: Los Angeles, CA, USA, 2004.

25. Ishwaran, H. Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC). R Package Version 2.9.3. [Internet]. 2020. Available online: https://cran.rproject.org/web/packages/randomForestSRC/randomForestSRC.pdf (accessed on 14 September 2023).

26. Kuhn, M. Training on Classification and Regression, R Package Version 6.0.86. Available online: https://cran.r-project.org/web/packages/caret/caret.pdf (accessed on 14 September 2023).

27. Cernada, M.; Aguar, M.; Brugada, M.; Gutiérrez, A.; López, J.L.; Castell, M.; Vento, M. Ventilator-associated pneumonia in newborn infants diagnosed with an invasive bronchoalveolar lavage technique: A prospective observational study. *Pediatr. Crit. Care Med.* **2013**, *14*, 55–61. [CrossRef] [PubMed]

28. Elward, A.M.; Warren, D.K.; Fraser, V.J. Ventilator-associated pneumonia in pediatric intensive care unit patients: Risk factors and outcomes. *Pediatrics* **2002**, *109*, 758–764. [CrossRef] [PubMed]

29. Sajedi, M. VAP-Predictive-Model, GitHub repository [Internet]. 2021. Available online: https://github.com/SajediM/VAP-Predictive-Model (accessed on 30 September 2020).

30. Jouvet, P.A.; Payen, V.; Gauvin, F.; Emeriaud, G.; Lacroix, J. Weaning children from mechanical ventilation with a computer-driven protocol: A pilot trial. *Intensive Care Med.* **2013**, *39*, 919–925. [CrossRef]

31. Zaglam, N.; Jouvet, P.; Flechelles, O.; Emeriaud, G.; Cheriet, F. Computer-aided diagnosis system for the Acute Respiratory Distress Syndrome from chest radiographs. *Comput. Biol. Med.* **2014**, *52*, 41–48. [CrossRef] [PubMed]
32. Mazo, C.; Kearns, C.; Mooney, C.; Gallagher, W.M. Clinical Decision Support Systems in Breast Cancer: A Systematic Review. *Cancers* **2020**, *12*, 369. [CrossRef] [PubMed]

33. Ostropolets, A.; Zhang, L.; Hripcsak, G. A scoping review of clinical decision support tools that generate new knowledge to support decision making in real time. *J. Am. Med. Inform. Assoc.* **2020**, *27*, 1968–1976. [CrossRef]

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.