AI-HEAT: A Clinical Decision Support System for pediatrics febrile conditions

Authors: abraham  bautista-castillo
rocio a padilla-medina
jessica  nguyen
chisato  shimizu
adriana h tremoulet
jane c burns
ananth v annapragada
tiphanie p vogel
ioannis a kakadiaris
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-15 00:31:43

Abstract: In recent years, several diagnostic challenges have developed due to the COVID-19 pandemic, including the post-infectious sequelae multisystem inflammatory syndrome in children (MIS-C). This syndrome shares several clinical features with other entities, such as Kawasaki disease (KD) and endemic typhus, among other febrile diseases. Endemic typhus, or murine typhus, is an acute infection treated much differently than MIS-C and KD. Early diagnosis and appropriate treatment are crucial to a favorable outcome for patients with these disorders. To address these challenges, a Clinical Decision Support System (CDSS) designed to support the decision-making of medical teams can be implemented to differentiate between these disorders. We developed and evaluated a CDSS based on a Triplet Loss Siamese Network to distinguish between patients presenting with clinically similar febrile illnesses, KD, MIS-C, or typhus, using eight clinical and laboratory features typically available within six hours of presentation. The performance assessment for AI-HEAT, Logistic Regression, Support Vector Machine, XGBoost, and the TabPFN machine learning models, was performed by computing Balanced Accuracy. AI-HEAT is a CDSS capable of obtaining performance similar to a state-of-the-art Transformer-type deep learning model such as TabPFN, with advantages such as being almost a thousand times smaller.

Main paper: 
 i. introduction: 
 In April 2020, children began to be hospitalized for fever and multisystem inflammation [3, 4, 5, 6] , and one of the most critical clinical challenges to arise during the pandemic appeared: multisystem inflammatory syndrome in children (MIS-C). In May 2020, the Centers for Disease Control and Prevention (CDC) published a case definition for this syndrome [7] , where the clinical similarity with other febrile diseases was already evident. Fever, rash, conjunctivitis, oromucosal changes, abdominal pain, vomiting, diarrhea, myocarditis, and hematological abnormalities are just some of the symptoms frequently found in MIS-C [3, 4, 5, 6, 7, 8] and that can be found in Kawasaki Disease (KD) [9] , toxic shock syndrome (TSS) [10] , and typhus [11, 12] , generating a clinical challenge to distinguish MIS-C from these pathologies.To address other high-level clinical challenges, computer systems designed to support the decision-making of medical teams, such as Clinical Decision Support Systems (CDSS), have been implemented. These computing systems have been implemented in a wide range of clinical challenges, such as antibiotic management [13] , heart disease prediction [14] , and even cancer detection [15] , so the implementation of a CDSS capable of distinguishing between KD, MIS-C, typhus among other non-specific febrile illnesses would be of significant impact for medical teams in the emergency department for timely-decision making which is essential for better outcomes in these febrile conditions.The main contributions of this paper are:• Developed and evaluated an AI-based CDSS for distin-guishing between KD, MIS-C, typhus, and other nonspecific febrile illnesses. • Developed and evaluated a new downsampling approach to face the problems of small imbalanced datasets.CDSSs have been used during the COVID-19 pandemic as support tools for the prognosis of disease severity [16] or predicting mortality [17] , and most of them have focused only on MIS-C-related clinical challenges [18, 19] . One of the only CDSS that has extended its approach beyond MIS-C prediction is the one presented by Lam et al. [20] , where they built a two-stage model of feedforward neural networks intended to differentiate between MIS-C, KD, and children with non-specific febrile illnesses, considering the importance of timely prediction using features obtained within the first 24 hours.This research can be considered an extension of Bautista-Castillo, et al. work [21] , which, to our knowledge, is the only one considering endemic typhus as one of the possible overlapping febrile diseases for MIS-C and that also incorporates a score that can be used in the Emergency Department (ED) without using electronic devices and only using features obtained during the first six hours after patient's arrival. AI-HEAT incorporates KD and children with nonspecific febrile illnesses to be distinguished, being the first CDSS that considers these febrile conditions only using eight features obtained during the first six hours after Emergency Department arrival and predicting the four febrile conditions.This section will discuss the methods used to create AI-HEAT, a CDSS based on a Triplet Loss Siamese Network that distinguishes between patients with Kawasaki, MIS-C, Typhus, and non-specific febrile illnesses.Multiple Imputation by Chained Equations (MICE) [22] with LightGBM was used to address the missing values for data imputation. This is an iterative statistical technique where values are imputed several times and performed chained using LightGBM to perform the predictions in every iteration. The Python package implemented in this work can be found in [23] .To avoid biases during the training and testing phases, we created four cohorts, considering three characteristics: age, sex, and the patient's condition, to obtain the most homogeneous distribution of patients possible in each cohort. This process can also be referred to as a 4-set cross-validation with matching conditions, where the matching conditions are the age, delimited in one-year intervals, the sex, and the patient's febrile condition. To address the dataset class imbalance problem, we propose an algorithm for downsampling the majority classes. First, we perform a bi-dimensional projection of all patients from a majority class that will be used for training in the trial using tdistributed Stochastic Neighbor Embedding (t-SNE). Next, we used k-means as our clustering algorithm, defining the number of clusters (K) to find using the "elbow method," where we initialize k-means from one and iteratively augment that number until the sum of square distances, or distortion stops being considerably smaller for the next iteration, compared to the previous one to find the optimal number of clusters (Fig. 1 ). Finally, we compute the Euclidean distance of all points from the cluster's centroid to which they belong to determine which points are closer to the centroid and which are further from it, keeping those that are further due to they possess the less common pattern within that cluster and, therefore, they will be more informative to our model (Fig. 2 ). This process is iterative and is applied to all majority classes one at a time. The core of our deep learning model is the Triplet Loss Siamese Network. Two of the network's main characteristics are that it has three branches that share weights and require as input a positive example, a negative example, and an anchor or baseline (Fig. 3 ). This input is usually referred to as a triplet. This neural network aims to enforce a desired distance between the triplets that make up its input in a meaningful embedding space, reducing the distance between the anchor or baseline and the positive example and maximizing the distance between the anchor or baseline and the negative one, using the Triplet Loss function defined as:L(A, P, N ) = max{d(A, P ) -d(A, N ) + α, 0}where d(A, P ) is the distance between the anchor or baseline and the positive example, d(A, N ) is the distance between the anchor or baseline and the negative example, and α is the minimum desired difference between the distances (Fig. 4 ).AI-HEAT consists of an embedding stage and a classification stage. In the embedding stage, the patient information enters the four Siamese Triplet Loss Networks, where the representation of that patient is obtained for each of the embedding spaces in which the four febrile conditions are represented. Subsequently, the resulting embedded vectors are concatenated to form a single vector with said embedded representations. This vector will feed the classification stage, consisting of densely connected layers with a final softmax layer, whose output will be a probability distributed among the four febrile conditions given the patient's clinical and laboratory features (Fig. 5 ). This section will discuss the characteristics related to the dataset used to train and test the AI-HEAT model and the models used as baselines, among them TabPFN [24] , a priordata fitted network that uses a transformer to classify small tabular data, XGBoost [25] , Support Vector Machine (SVM), and Logistic Regression (LR), in addition to the characteristics of the cohorts and trials, the metrics used as performance assessment, and all the settings for the experimental results.The dataset used for training and testing both models included 943 patients admitted with non-specific febrile illnesses and 1,105 patients admitted with KD to Rady Children's Hospital and its satellite locations, 135 patients admitted with MIS-C and 87 patients admitted with murine typhus admitted to Texas Children's Hospital and its two satellite campuses within the greater Houston area. Medical records were reviewed, with eight demographic, clinical, and laboratory features available within six hours of the presentation for all febrile conditions (Table I ). Within the dataset, 1,347 patients are males, and 923 are females. Maximum, minimum, mean, prevalence, and missing values for all the features of the dataset are shown in Table II .After applying the 4-set cross-validation with matching conditions to create the four cohorts for training and testing, the distributions shown in Table III cohorts were created, we defined the training and testing patients for each trial based on these cohorts, where for each trial, the cohort with the same Trial number will be used as a test cohort. For Trial 1, the cohort used as a test set was Cohort 1; for Trial 2, the cohort used as the test was Cohort 2, and so on. Finally, for the cohorts used as training for each trial, the Informative Samples Selection Method (Sec. III-C) was applied, obtaining the distribution of the febrile conditions for training for every trial shown in Table IV .The Balanced Accuracy metric was used to evaluate the performance of AI-HEAT and the baseline models. This decision was made based on the evident imbalance of the cohort used for testing in each trial, which is not subject to the Informative Sample Selection Method and thus retains the characteristics shown in Table III . The Balanced Accuracy metric was defined as follows:Balanced Accuracy = 1 N N i=1 Sensitivity i ,where N is the number of febrile conditions.The training and testing of all models were performed with Python 3.9.17, Tensorflow 2.13.0, Pandas 2.0.3, and Keras 2.13.1 running on a LINUX-based computer equipped with an AMD Ryzen 5 5600g CPU and a NVIDIA GeForce RTX3060 GPU. For TabPFN, we used the Python package downloaded directly from the authors' GitHub repository [26] . Similarly, XGBoost was installed and ran following the steps outlined in the developers' documentation [27] . SVM and LR were implemented directly from the scikit-learn package for Python [28] . To train AI-HEAT, our CDSS, we began by training each Triplet Loss Siamese network. These networks were designed to create an embedding space for each of the four febrile conditions. In this setup, each febrile condition was treated as a positive instance within its embedding space, while the other conditions were considered negative. Once the four Triplet Loss Siamese Networks had been trained, their outputs were fed into a concatenation layer that combines the vectors from each network and pass them through a series of densely connected layers that form the classification stage of our CDSS. Each Triplet Loss Siamese Network consists of four layers with 16, 32, 16, and 8 neurons, respectively. Training was performed using the Triplet Loss function with a learning rate of 0.002, an alpha parameter of 0.2, and a batch size of 30. The classification module uses three densely connected layers, with 32, 64, and 32 neurons, respectively, and employs a ReLU activation function. For training this module, we used mean square error as the loss function and an Adam optimizer with a learning rate of 0.001 over 25 epochs, with ten steps per epoch. The training was conducted for all models with and without the Informative Sample Selection method implementation to compare the impact of the downsampling method proposed in III-C. In the case of TabPFN, due to the Transformer's restrictions that do not admit more than 1024 samples for training, a random sampling of 400 samples was carried out for non-specific febrile illnesses and KD patients. The experimental results for all trials are shown in Table V, while experimental results for all classes can be seen in Fig. 6 .In the results shown in Table V and Figure 6 , the positive impact of implementing the Informative Samples Selection Method is evident, where the models most benefited by its implementation were SVM and AI-HEAT, and the least benefited was TabPFN. This may be due to TabPFN's well-known robustness even when dealing with imbalanced datasets thanks to its Transformer-type architecture. AI-HEAT achieves a performance similar to that of the TabPFN, with some specific advantages such as being a model that allows visualization of the embedding spaces (Fig. 7 , 8, 9, and 10) that can result in a better understanding of the similarity of some of the febrile conditions treated here, as well as the significant difference in size of both architectures, where TabPFN has 25.8 M parameters. In comparison, AI-HEAT only has 24.4 K parameters.In these bidimensional projections of the multi-embedding space, it can be observed how AI-HEAT interprets the existence of a certain similarity in febrile conditions such as Kawasaki (Fig. 8 ) and typhus (Fig. 10 ), where it observes that several patients occupy the same regions of this multiembedding space. At the same time, it can be seen that patients with MIS-C (Fig. 9 ) are distributed throughout the multi-embedding space, which suggests that they are the most challenging patients to distinguish between the four febrile conditions. These conclusions were confirmed by medical staff who are in constant contact with these febrile conditions. VI. CONCLUSIONS AI-HEAT is a CDSS capable of obtaining performance similar to that of a state-of-the-art Transformer-type deep learning model such as TabPFN with advantages such as being almost a thousand times smaller, having the ability to allow the visualization of its embedding spaces to the better understanding of the febrile conditions that it distinguishes, and having the flexibility to be able to incorporate other architectures in its classification stage that can improve its performance.  
 
 Appendix:  



Reviews: review: title: title: Review of AI-HEAT, an AI-based CDSS to distinguish challenging febrile illnesses
paper_summary: paper_summary: In this paper, authors present AI-HEAT, a clinical decision support system (CDSS), to facilitate distinguishing febrile illnesses with similar symptoms that makes their diagnosis challenging. AI-HEAT consists of two modules; an embedding module, where patient information is encoded via a Triplet Loss Siamese Network, and a classification module where this encoded information is fed to a fully connected neural network. Furthermore, a sample selection method is utilized to tackle issues regarding imbalance datasets, which its importance is demonstrated in results.
strength_weakness: strength_weakness: From the technical point of view, although AI-HEAT has comparable performance as state-of-the-art methods, its performance is very sensitive to the dataset, which can be unacceptable in medical scenario. Specifically, when using imbalance dataset, basically there is no advantage of using AI-HEAT. The proposed sampling method is not necessarily new (margin-based sampling, in Active Learning literature). In general, paper lacks originality and does not present any significant novel contributions. 

From the paper structure point of view, the paper is not properly organized, and the manuscript can be improved.  In different sections, like in IV.C, and at the end of IV.D, the text is poorly written and is ambiguous. Most figures have very low quality, specifically figures 3, 4, and 5. It is much more helpful to put all 4 figures regarding the visualizations in one figure with 4 subfigures. In section IV.A, in table I it is mentioned that there are 4 binary features, however, statistics of two of them is reported in table II. Also “Fever days before hospital” is denoted as binary in table I, while in table II it is considered as numerical. The abbreviations under table I are better to be mentioned in the manuscript or with a more distance from table. In the explanation of table II, authors mention "prevalence" however there is no such term in table II. Clearer labels are required as well (figure 6 does not have any label regarding the values).
questions: questions: NA
title: title: AI for febrile conditions in children
paper_summary: paper_summary: A solution based on embeddings of patient parameters and neural networks is described, parametrized, evaluated and compared with SotA systems specialising in febrile conditions in children
strength_weakness: strength_weakness: Algorithm 1 (later referred as Algorithm A) might just be a 4 set cross-validation segmentation of the dataset with matching conditions.
questions: questions: By using cross-validation, you may have more precise results by running the training (and validation) 4 times instead of 1, averaging out the results. This avoids accidental over-training and other dataset statistical anomalies which may affect the training process.

In algorithm 1, are age intervals and sex considered "condition"? this might be confusing.

Minor notes: all line numbers for Algorithm 1 are "0" which defeats the purpose of having line numbers, they are not referred later in the paper, so maybe it would be good to remove them. In  IV. B. a reference to Algorithm A is stated, probably this is Algorithm 1.