Knowledge-Empowered Dynamic Graph Network for Irregularly Sampled Medical Time Series

Authors: yicheng  luo
zhen  liu
linghao  wang
junhao  zheng
binquan  wu
qianli  ma
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 04:35:52

Abstract: Irregularly Sampled Medical Time Series (ISMTS) are commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features at the variable level in ISMTS. We propose Knowledge-Empowered Dynamic Graph Network (KEDGN), a graph neural network empowered by variables' textual medical knowledge, aiming to model variable-specific temporal dependencies and inter-variable dependencies in ISMTS. Specifically, we leverage a pre-trained language model to extract semantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from a medical perspective. Based on this, we allocate variable-specific parameter spaces to capture variable-specific temporal patterns and generate a complete variable graph to measure medical correlations among variables. Additionally, we employ a density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variable-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra-variable and inter-variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods.

Main paper: 
 introduction: 
 In the medical environment, the widely used Electronic Health Records (EHRs) have abundant typical Irregularly Sampled Medical Time Series (ISMTS) data [1] . Each ISMTS typically comprises multiple medical variables for a patient, each with distinct medical properties, resulting in significant differences in the sampling patterns of each variable series. Additionally, due to the dynamic changes in a patient's condition, the sampling rate of variables varies over different periods, resulting in uneven sampling intervals [2] .Many existing methods for ISMTS primarily focus on addressing uneven sampling intervals and have proposed approaches such as Ordinary Differential Equations (ODEs) [3, 4] and continuous-time embeddings [5, 6] , etc, which have already achieved significant success. Recent advancements in regularly sampled multivariate time series analysis [7, 8] underscore the importance of capturing variable-specific temporal patterns. However, many existing methods for ISMTS have not adequately considered this aspect, having limited ability to explicitly distinguish multiple variable series with different time patterns and thus lacking finer-grained capturing of variable-level features. Particularly in ISMTS, different variables have distinct medical properties, further intensifying the degree of differences among variables. In such cases, capturing differentiated variable patterns requires a deeper exploration of inherent differences among variables. Despite the differentiated temporal patterns occurring among variables within ISMTS, they are not entirely independent but exhibit medical correlations. Due to the dynamic changes in the patient's condition, this correlation varies along with the sampling density of variables at different periods, as illustrated in Figure 1 . Some existing work has introduced graph neural networks [9, 10] to model the time-varying correlations among variables in ISMTS. However, due to the lack of prior medical knowledge, these methods learn variable correlation graphs from misaligned and imbalanced observations in variables of ISMTS and rely solely on downstream tasks for graph optimization. Consequently, the variable graphs learned by these methods may face challenges in accurately reflecting the general medical correlations among variables, resulting in suboptimal performance.To this end, we aim to explicitly consider the differences and correlations among variables in ISMTS, empowering the model to capture fine-grained features at the variable level. However, the sampling rates, sampling times, observation spans, and observation lengths of different variables vary in ISMTS, as shown in three subgraphs in Figure 2 . This makes it tough and complex to infer these two aspects only from the time series modality of variables. We rethink the issue based on an intuitive observation as shown in Figure 2 : Variables that exhibit (dis)similar temporal patterns frequently have (dis)similar medical properties in reality, which motivates us to consider the differences and correlations of variables from the perspective of domain knowledge directly. Recent work [11, 12] has successfully represented domain knowledge through textual modal information, enhancing model performance in medical imaging, which provides us with insights. Specifically, the medical properties of each variable can be described in natural language. Leveraging the powerful semantic understanding capabilities of the Pre-trained Language model (PLM), we can obtain semantic representations from the textual knowledge of each variable. This set of textual representations forms an overall view of variables from the perspective of medical knowledge, clearly showing inherent differences and correlations among variables-exactly what we need.Based on the above analysis, we propose the Knowledge-Empowered Dynamic Graph Network (KEDGN), which utilizes textual semantic representations of variables obtained through PLM as guidance. On this basis, we 1) allocate unique parameter space for each variable to capture their specific temporal pattern, 2) generate a complete variable graph and introduce a density-aware mechanism to explicitly model time-varying correlations among variables in ISMTS. Finally, these The time patterns (sampling rates, sampling times, observation spans, observation lengths, trends, etc.) of variables among different subgraphs exhibit significant differences, as they have distinct medical properties. Meanwhile, variables within the same subgraph share similar time patterns, and their medical properties are closely related. (More variable groups can be found in Figure 7 ).two modules are integrated with a graph convolutional recurrent network to capture both temporal and inter-variable dependencies in ISMTS. Our contributions can be summarized in three aspects:• We leverage variable-specific textual medical knowledge to empower the model to capture variable-specific temporal patterns in ISMTS distinctively.• We introduce a density-aware mechanism based on the knowledge-empowered variable graph to model the time-varying inter-variable dependencies in ISMTS.• Empirical results on four real-world medical datasets demonstrate that KEDGN outperforms state-of-the-art methods. Visualization analysis further illustrates the strong interpretability of our approach. 
related work: 
 Irregularly Sampled Multivariate Time Series Modeling Existing methods can be roughly categorized into interpolation-based and raw data-based approaches. The former, employing methods such as kernel-based approaches [13, 14] , Gaussian process [15] or hourly aggregation [16] , aims to obtain a set of regularly spaced observations. However, interpolation may result in the loss of useful information about the original sequences, such as missing patterns. The latter, raw databased methods, directly learn from irregular time series. To adapt to uneven sampling intervals, [17] improves recurrent neural networks, [3, 4] introduce neural ordinary differential equations and [6, 5, 18, 19] adopt time embeddings. [20] converts ISMTS into line graph images and utilizes pretrained vision transformers for extracting features. These methods primarily focus on overall temporal dependencies, needing more consideration for fine-grained variable-level patterns and correlations. Despite recent research introducing attention [21, 22] or graph neural networks [9, 23] to account for variable correlations, these methods have limited performance due to the lack of prior knowledge and the use of shared parameter spaces among all variables.Graph Neural Networks for Multivariate Time Series In recent years, a series of studies have integrated GNN with various time series modeling frameworks to effectively capture both intervariable and inter-temporal dependencies in MTS [24] . These approaches have been widely applied in diverse domains, including transportation [25] , healthcare [26] , economics [27] , demonstrating promising results in mainstream tasks such as prediction [28] , classification [29] , and imputation [30] .Although recent work [31, 32, 33, 34] has proposed the idea of modeling variable relationships through learning dynamic graphs, most of these methods are primarily designed for regularly sampled MTS with synchronous observations, and further improvements are needed to adapt them for irregularly sampled time series.Several studies have utilized the rich domain knowledge in the medical field to enhance models. [12, 35] apply knowledge for computing additional features, while [36, 37] utilize knowledge to guide the final training loss, demonstrating the effectiveness of medical prior knowledge. However, existing methods commonly focus on visual language pretraining in medical scenes or medical report generation [11] . How to effectively integrate domain knowledge to guide medical time series modeling remains a challenge. 
problem definition: 
 Given a dataset D = {(s i , y i ) | i = 1, . . . , N } containing N patient samples, the i-th sample consists of an irregular multivariate time series s i and a label y i . For the dataset with a total variable count of V and a maximum sample observation length of T , s i can be denoted as a tuple:s i = (t i , x i , m i ), where t i ∈ R T represents the observation timestamps, x i ∈ R V ×T represents the multivariate time series observations, and any unobserved values or the missing parts of the time series shorter than the maximum sample observation length are filled with 0. The binary indicator m i has the same size as x i , indicating which elements in the x i are actually observed. We use 1 to represent observed values and 0 to represent missing values. In this paper, we focus on patient mortality and morbidity prediction, i.e., classification task, aiming to correctly predict class label ŷi given a sample s i .Heart rate can be used as an indicator of how well a.… Body temperature is typically measured using … MAP reflects the average pressure in a patient's … Figure 3 : The model framework of KEDGN. We (1) utilize a PLM to extract semantic representations for each variable from textual medical properties (Section 4.1). Based on this, we (2) allocate variable-specific parameter space to capture variable-specific temporal patterns (Section 4.2), (3) generate dynamic variable graphs by combining knowledge-empowered graph with a density-aware mechanism to model time-varying correlations among variables (Section 4.3). ( 4) The above two modules are injected into graph convolutional recurrent network to model intra-variable and intervariable dependencies in ISMTS simultaneously (Section 4.4).Pretrained Language Model 
variable semantic representations extraction: 
 First, we introduce how to extract semantic representation for each variable from medical knowledge. Let V = {v 1 , v 2 , . . . , v V } be the set of variables, and a descriptive sentence of medical properties associated with the j th variable can be denoted as:EQUATIONwhere l j is the length of the j th variable's sentence and w j,i denotes the i th word of the j th sentence. We leverage a PLM to represent each text description P j as a d-dimensional embedding. Considering the diverse types of PLMs with varying methods of utilization, we use the widely adopted encoderbased model BERT [38] as an illustration:EQUATIONwhere [CLS] and [SEP] are special tokens indicating a sequence's beginning and end, respectively. BERT generates an embedding for each token of the input sequence. Since the embedding at the [CLS] position captures the semantic information of the entire input sequence, we take the hidden state at the [CLS] position as the overall semantic representation of each variable:E j = e j ([CLS]).This yields a semantic representation matrixE = [E 1 , E 2 , ..., E V ] ∈ R V ×dand forms an overall view of variables from the perspective of medical knowledge.Since temporal patterns of ISMTS vary from variable to variable, simply using shared parameter space for all variables is insufficient to capture differentiated temporal dependencies. In this section, we adjust parameter space for different variables to adapt to differentiated temporal patterns based on the extracted variables' semantic representations. For any parameter matrix w ∈ R I×O with input dimension I and output dimension O, the total parameter space needed for V variables is Θ ∈ R V ×I×O . Inspired by [39] , we decompose Θ into two matrices: a variable representation matrix Q ∈ R V ×q and a weight pool matrix W ∈ R q×I×O , where q is a hyperparameter for the intermediate dimension. Here, Q consists of q-dimensional query vectors for V variables used to distinguish differences among variables and obtain variable-specific parameters from the weight pool.We use a projection f (•) : R d → R q to variable semantic representations E for obtaining query vectors rather than directly using E. On the one hand, the dimension of the query vector q directly determines the size of the weight pool, but the output dimension of PLM is often large (e.g., 768 in BERT), leading to a sharp increase in model complexity. On the other hand, there is a modality gap between textual embeddings and the temporal parameter space. The projection f (•) can achieve both feature reduction and modality transformation. In our implementation, we use a nonlinear projection with one additional hidden layer (and ReLU activation). Thus, the parameter space specific to variable i can be obtained using the following formula:EQUATIONThe approach we employ to generate variable-specific parameter space is general and not restricted to a specific model backbone because any model architecture is composed of multiple parameter matrices W .In this section, we introduce how to generate dynamic correlation graphs of variables for explicitly modeling the time-varying correlations among variables in ISMTS.The misaligned and imbalanced observations of the variables in ISMTS make it difficult to learn the variable correlations from the time series. Therefore, we extract a static complete variable correlation graph based on the textual semantic representations of the variables directly from the perspective of the actual medical properties of variables. We apply another non-linear projection g(•) : R d → R n to the textual representations of variables E to obtain n-dimensional node embeddings for each variable. Subsequently, we calculate the pairwise cosine similarity among the node embeddings of variables, resulting in a V × V matrix of variable similarity. Finally, we use the softmax function to normalize the edge weights corresponding to each node, producing a normalized graph of variable correlations. The correlation weight between the i th and j th variables can be calculated as:EQUATIONwhere • represents vector dot product and ∥ • ∥ represents the vector magnitude. The introduction of g(•) in this context not only performs feature reduction but also avoids using a completely fixed prior graph. It preserves the model's ability to adaptively optimize the graph structure based on different data distributions and downstream tasks. Thus, we obtain a knowledge-empowered complete graph with V nodes to measure the static correlation among variables in general medical cases, and its adjacency matrix is denoted as A.Due to the varying subsets of variables observed at each timestamp in ISMTS, we use different subgraphs of A to describe the variable correlations at different timestamps. Specifically, we use a mask matrix M (t) ∈ R V ×V to indicate the subgraph topology at timestamp t:M (t) ij =1, if both variables i and j are observed at t 0, otherwise ,Therefore, we can calculate the variable correlation subgraph A (t) at timestamp t throughA (t) = A ⊙ M (t), where ⊙ represents Hadamard product. Additionally, we introduce a density-aware mechanism to dynamically adjust edge weights of subgraphs in different timestamps to fit in the time-varying correlations among variables mentioned in Figure 1 . Specifically, we estimate the sampling density of any observation point by considering the average time interval between each observation point and its preceding and succeeding observations. If there is no preceding/succeeding observation, we take the time interval of the succeeding/preceding observation as the density. If neither a preceding nor a succeeding observation exists, it indicates that this observation is the only one for the variable, and we take half of the maximum observation time span as the density. The formula for calculating the sampling density of the i-th observation of variable v at timestamp t is:EQUATIONThen we calculate the density scores for various variables at timestamp t through:EQUATION)where σ is an activation function and α is a hyperparameter that controls the proportion. At timestamp t, the edge weight between the i th and j th variables is adjusted as:EQUATIONwhere W ∈ R V ×V is a learnable parameter matrix. Thereby, we achieve the dynamic adjustment of the variable graph weights in response to changes in variable sampling density.Under the empowerment of variable textual representation, we have obtained variable-specific parameters Θ ∈ R V ×I×O and dynamic variable graph G ∈ R T ×V ×V . In this section, we integrate these two modules into the graph convolutional neural network to handle ISMTS. GCRNN [40] is a backbone network that introduces graph convolutional operations on top of an RNN variant, Gated Recurrent Unit [41] . This structure is simple, effective, and easy-to-adapt for ISMTS, as it enables variable-level parallel computation of asynchronous observations without explicit interpolation. Specifically, we allocate a unique hidden state for each variable, updating the state only at the observed timestamps to avoid imputation and preserve the individual sampling patterns of each variable. The graph convolution operation over a graph signal S ∈ R V ×I containing V nodes at timestamp t is defined as follows:EQUATIONwhere I V ∈ R V ×V is identity matrix, × represents batch matrix multiplication. Here, we adopt 1 st -order Chebyshev polynomial expansion approximation [42] for graph convolution. The updated formulas for variable states at timestamp t are:EQUATIONEQUATIONEQUATIONEQUATIONwhere || denotes the concatenate operation, H (t-1) ∈ R V ×h is the variable states at the previous timestamp and X (t) ∈ R V ×k denotes the input representation at current timestamp. We follow the structured input encoding method of [22] , using multiple fully-connected mappings to encapsulate each observed value and its corresponding timestamp into a k-dimensional input representation (All 0 vectors for unobserved values) to indicate flexible observation time and adapt to the uneven intervals within variables. r (t) , u (t) ∈ R V ×h are reset gate and updated gate, respectively. Θ r , Θ u , Θ C ∈ R V ×(k+h)×h are variable-specific parameters obtained by respectively multiplying the query vectors matrix Q ∈ R V ×q with three weight matrices W r , W u , W C ∈ R q×(k+h)×h .We calculate the sum of h channels for each variable's hidden state H i at the last observed timestamp to get a V-dimensional vector C. Additionally, we follow the approach used in [9] to incorporate the static features. Specifically, static features of each sample are mapped into a static vector S through a linear layer. Finally, C and S are concatenated to predict the final classification probabilities:ŷ = Softmax(W y [C||S] + b y ).The training objective is minimizing the cross-entropy loss between ŷ and y. The pseudo-code for KEDGN is presented in Appendix A (Algorithm 1).We conduct experiments on four widely used irregular medical time series datasets, namely P19 [43] , Physionet [44] , MIMIC-III [45] and P12 [46] where Physionet is a reduced version of P12 considered by prior work [6] . We compare our method with the state-of-the-art methods for modeling irregular time series, including GRU-D [17] , ODE-RNN [4] , IP-Net [14] ,SeFT [5] , mTAND [6] , Raindrop [9] , StraTS [18] , DuETT [19] , ViTST [20] and Warpformer [22] .In addition, we also compare our method with two approaches initially designed for forecasting tasks, namely DGM 2 -O [13] and MTGNN [47] . The implementation and hyperparameter settings of these baselines are kept consistent with those used in [9] . More details of datasets and baselines can be found in Appendix B and C.Evaluation Setup For the data pre-processing of MIMIC-III, we follow the method described in [48] and divide the dataset into three parts for training, validation, and testing with the ratio of 70%,15%,15%. For the remaining three datasets, we adhered to [9] 's approaches, and the ratio of training, validation, and testing set is 8:1:1. We measure the classification performance with the Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC) since all the four datasets are binary classification datasets with highly imbalanced class distribution. AUPRC has better sensitivity to sample imbalance [49] ; thus, the optimal model parameters that achieve the best AUPRC on the validation set are used for the test set. More details of metrics can be found in Appendix D.Implementation Details We adopt the Adam [50] optimizer, and the number of training epochs is set as 10. Due to differences in dataset sizes, the learning rate is set as 0.001 for Physionet and P12 and 0.005 for MIMIC-III and P19. The textual sources for variable descriptions are flexible; we chose three sources, including the variable's full name, Wikipedia source, and ChatGPT source (Default), corresponding to model names KEDGN-Name, KEDGN-Wiki, and KEDGN-ChatGPT, respectively. All experiments are conducted with five random seeds, and the average and standard deviation are reported. More implementation details and hyperparameter settings can be found in Appendix E. Classic time series classification The evaluation results are summarized in Table 1 , in which we use BERT to extract variables' semantic representations. More experimental results and analyses using other PLMs can be found in Appendix F.1. Overall, our KEDGN achieves the best performance on all four datasets and outperforms the strongest baseline by an average of 0.9% on AUROC and 1.6% on AUPRC. In addition, the results of models using different text sources are similar, which demonstrates that our method is not limited to a specific text source and exhibits generalizability. We also provide an analysis of computational costs in Appendix F.2.To demonstrate the robustness of our method, we test whether KEDGN can achieve good performance when a subset of variables is completely missing. We uniformly discard 10%, 20%, 30%, 40%, and 50% of the variables, hiding all their observations in both validation and test sets. Table 2 reports the results on the MIMIC-III dataset, while the results for the remaining datasets are presented in Appendix F.3 (Table 8 ). Our method achieves the highest performance on 35 out of 40 metrics across 4 datasets as the missing rate increases from 10% to 50%. This may be attributed to KEDGN only handling actually observed points, avoiding the accumulation of imputation errors, particularly in cases of higher missing ratios, thus exhibiting a degree of robustness. We remove the density-aware adjustment for edge weights of the graph, using static adjacency matrix A (t) during different periods. ( 6) w/o TE: We remove the timestamp embedding part of the structured input encoding X (t) . The results on the P19 dataset are presented in Table 3 , while the results for the remaining datasets are presented in Appendix F.4 (Table 9 ). The results show that all model components are necessary and variable-specific parameter space makes the most significant contribution to the performance of KEDGN. 
visualization of variables textual representations: 
 In this section, we explore why textual information is effective for time series modeling through visualization analysis. We first group variables with similar time patterns on the P12 dataset, as illustrated in Figure 2 (More variables groups can be found in Appendix F.7 (Figure 7 )). Subsequently, we use T-SNE [51] to visualize the distribution of variable semantic representations. Figures 4a and 4b respectively display the distributions of ChatGPT and Wikipedia sources, while Figure 4c shows the final distribution learned by replacing the text representation with randomly initialized learnable embeddings. Variables with similar time patterns are labeled with the same color. It can be observed that the textual representation space of variables exhibits distinct clustering. Although there may be occasional outliers when using different text sources, such as BUN in Figure 4a and HCO3 in Figure 4b , the overall clusters are generally consistent with the variable groups we divide based on time series patterns. However, the learnable variable embeddings in Figure 4c , optimized by classification loss, tend to be distributed uniformly, which is difficult to effectively reflect the intrinsic differences among variables. More visualizations of other datasets and other PLMs can be found in Appendix F.8 and F.9, respectively. Based on this phenomenon, we infer that text descriptions and time series are both external manifestations of the inherent sense of variables; they just belong to different data modalities. These two forms of data for the same variable should exhibit relative consistency. Therefore, the relative distribution among variables extracted from text and time series should ideally be similar. Leveraging PLM allows for the straightforward and efficient extraction of this universal view from textual descriptions, which is equally applicable to describing the relative distribution of temporal patterns among variables. Thus, the effectiveness of PLM and the cross-modal relative consistency are the keys to guiding time series modeling based on textual information. We visualize the learned partial inter-variable correlation graphs on the MIMIC-III dataset in the form of heatmaps. In Figure 5a , we depict the graph learned based on node embeddings mapped from textual representations, while in Figure 5b , the graph is learned based on randomly initialized learnable node embeddings. We observe that the top-left and bottom-right corners of the heatmap in Figure 5a exhibit darker colors, indicating strong correlations consistent with medical domain knowledge. Specifically, the variables in the top-left corner, Heart Rate (HR) and Respiration Rate (RR), are commonly monitored together in clinical settings for assessing vital signs and respiratory system function [52] . The variables in the bottom-right corner, GCS-MR, GCS-T, and GCS-VR, collectively constitute the Glasgow Coma Scale (GCS), which is used to assess a patient's neurological status and level of consciousness [53] . These correlations are not evident in the graph without textual representations. This once again validates our perspective that relying solely on downstream task optimization for adaptive learning in graphs in ISMTS is insufficient to reflect the actual medical correlations among variables and lacks interpretability. In contrast, textual representations can guide the model to accurately extract variable correlations aligned with domain knowledge to provide high interpretability.In Figure 6 , we visualize the dynamic density-aware graph for the sample in Figure 1 . We present the time series of three variables and the corresponding correlation heatmaps learned by our model at  
limitations: 
 Although our proposed method effectively guides ISMTS modeling through the domain knowledge from text modality, it has some limitations. The backbone of our model is a recurrent-based architecture, which inherently has a sequential computation characteristic that can be a bottleneck in terms of runtime. Additionally, our method is specifically tailored for medical applications, and its performance may be limited in other irregular multivariate time series applications, such as human activity recognition, where variables lack domain knowledge and thus cannot generate high-quality text descriptions. 
conclusion: 
 In this paper, we propose KEDGN for modeling ISMTS. The proposed method leverages a PLM to flexibly extract semantic representation for each variable from the textual medical knowledge. Based on these representations, we allocate the variable-specific parameter space to capture variable-specific temporal patterns and extract a complete variable graph as a measure of the variables' static medical correlations. Considering the time-varying variables correlations in ISMTS, we introduce a densityaware mechanism to dynamically adjust the subgraph across different periods. Our experimental results demonstrate that KEDGN outperforms existing methods in ISMTS classification tasks and provides high interpretability. Our future work will focus on investigating the applicability of KEDGN in a range of related tasks, such as interpolation, extrapolation, and regression. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: Input: An ISMTS sample observation values matrix: x (1:T ) ∈ R V ×T , binary mask matrix m (1:T ) ∈ R V ×T , observation timestamps t (1:T ) ∈ R T , the sentences of V variables' medical properties P = {P 1 , P 2 , . . . , P V } Output: Predicted classification label ŷ 1: extract semantic representations E ∈ R V ×d for variables from P through PLM using Eq.( 2) 2: obtain variable-specific parameter space Θ (including reset gate Θ r , update gate Θ u and candidate gate Θ C ∈ R V ×(k+h)×h ) using Eq.(3) 3: obtain complete variable correlation graph A using Eq.( 4) 4: initialize the hidden state of V variables H (0) = 0 V ×h 5: for i = 1, 2, ..., T do 6:obtain dynamic variable subgraph G (i) ∈ R V ×V according to m (i) and Eq.( 8)7:encode observation values x (i) and timestamp t (i) into X (i) ∈ R V ×k by multiple fullyconnected mappings 8: update H (i) = GCRNN(Θ, G (i) , X (i) , H (i-1) ) using Eq.( 10), ( 11), ( 12) and (11) 9: end for 10: calculate the sum of h channels forH (i) to get C ∈ R V 11: calculate ŷ = Sof tmax(W y C + b y ) 12: return ŷWe use four irregularly sampled medical time series datasets to evaluate the classification performance of our model and baseline models. The dataset statistics are summarized in Table 4 . The P12 [46] dataset comprises data from 11,988 patients after 12 inappropriate samples identified by [5] were removed from the dataset. Each patient's record in the P12 dataset includes multivariate time series data collected during their initial 48-hour stay in the ICU. The time series data consists of measurements from 36 sensors (excluding weight). Additionally, each sample is associated with a static vector containing 9 elements, including age, gender, and other relevant attributes. Furthermore, each patient in the P12 dataset is assigned a binary label indicating the length of their stay in the ICU. A negative label signifies a hospitalization period of three days or shorter, while a positive label indicates a hospitalization period exceeding three days. It is available at https://physionet.org/content/challenge-2012/1.0.0/.The MIMIC-III [45] dataset is a widely used database that comprises de-identified Electronic Health Records of patients who were admitted to the ICU at Beth Israel Deaconess Medical Center from 2001 to 2012. It Originally encompassed around 57,000 records of ICU patients, containing diverse variables such as medications, in-hospital mortality, and vital signs. [48] established a variety of benchmark tasks using a subset of this database. In this paper, we focus on the binary in-hospital mortality prediction task to assess classification performance. Following preprocessing, our dataset consists of 16 features from the preceding 48 hours and 21,107 samples. It is available at https://physionet.org/content/mimiciii/1.4/.Physionet Physionet [44] contains the data from the first 48 hours of ICU patients, which is a reduced version of P12 considered by prior work. Therefore, we follow the same preprocessing methods used for the P12 dataset. The processed data set includes 3997 labeled instances. We focus on predicting in-hospital mortality. It is available at https://physionet.org/content/challenge-2012/.The implementation of baselines follows the corresponding papers or default implementation of their code repositories. Considering that different baselines have varying convergence speeds, we search for the number of epochs in the range of {10, 20, 50} to ensure all baselines can reach convergence. We search for the learning in the range of {0.001, 0.005} due to the differences in model complexity and dataset size. Since all four datasets are highly imbalanced, we upsample the minority class in each batch to make the batch balance. Here are the detailed hyperparameter settings of these baselines: StraTS [18] is a self-supervised transformer for sparse IMTS. We use the implementation at https://github.com/sindhura97/STraTS and the following setting in our experiment: hidden_dim = 64, num_layers = 2, num_heads = 16, dropout = 0.2.ODE-DuETT [19] is a dual event time transformer for Electronic Health Records (EHRs). We use the implementation at https://github.com/layer6ai-labs/DuETT and the default settings of the model declaration in this repository.ViTST [20] transforms IMTS into line graph images and adapts powerful vision transformers to perform time series classification in the same way as image classification. We use the implementation at https://github.com/Leezekun/ViTST.Warpformer [22] : A transformer-based network that captures features at different scales in IMTS using warping modules and dual attention mechanisms. We use three scales with normalized length L (0) = 0, L (1) = 0.2 and L (2) = 1. The dimension of representations D is set as 32. The attention heads and the layers of the warpformer are set as 1 and 2, respectively. We use the implementation at https://github.com/imJiawen/Warpformer.AUROC AUROC is commonly employed in binary classification tasks, where one class is designated as positive and the other as negative. It represents the area under the Receiver Operating Characteristic (ROC) curve, constructed by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR). AUROC ranges from 0 to 1, with a higher value indicating better model performance in accurately discriminating between positive and negative instances. An AUROC equal to 0.5 indicates a model's performance equivalent to random guessing, while an AUROC greater than 0.5 signifies superiority over random guessing.AUPRC The Area Under the Precision-Recall Curve is widely used as a performance metric for imbalanced binary classification tasks. It provides a comprehensive assessment of a model's precision-recall trade-off. The Precision-Recall curve is constructed by plotting recall on the x-axis and precision on the y-axis. AUPRC ranges from 0 to 1, and a higher value indicates better model performance in achieving high precision and recall simultaneously. It has been suggested as a good criterion for unevenly distributed classification problems [54] .Generation of Variables' Textual Descriptions We have chosen three textual sources: the full name of the variable, Wikipedia, and ChatGPT, for variable descriptions. The full name of the variable and Wikipedia sources can be directly extracted from dataset descriptions and web pages without requiring special processing, so we don't delve into them further here. Regarding the ChatGPT source, the medical properties description for each variable is generated by providing consistent question templates to ChatGPT, aiming to maintain a relatively uniform format across variables. We initiate queries to ChatGPT using the following template: What medical prior knowledge do you know about the medical variable 1 ⃝ in 2 ⃝? to obtain a textual description for each variable in a specific dataset. The position marked as ' 1 ⃝' corresponds to the variable name to be queried, and the position marked as ' 2 ⃝' corresponds to the dataset-specific task description. For instance, the task for the MIMIC-III dataset is ICU patients' in-hospital mortality prediction, so the query for obtaining textual information associated with the variable Heart Rate could be: What medical prior knowledge do you know about the medical variable Heart Rate in diagnosing whether ICU patients will die during hospitalization? The following is a part of ChatGPT's answer for this example:Heart rate is a fundamental physiological variable that plays a significant role in diagnosing and assessing the condition of ICU patients and their potential risk of mortality during hospitalization.Here's what you need to know about the medical variable "heart rate" in this context: Normal Heart Rate: A normal resting heart rate for adults typically ranges between 60 to 100 beats per minute (bpm). Deviations from this range can indicate potential health issues. Tachycardia: An elevated heart rate (tachycardia), often above 100 bpm, can be a sign of various medical conditions ... .The complete text description can be obtained through the code link.Hyperparameters We search all hyperparameters in the grid to find the best hyperparameters for our proposed model KEDGN. Specifically, our model has a total of 5 hyperparameters: dimension of query vectors q, dimension of variables' node embeddings n, proportion of density score α, dimension of variables' hidden state h, and dimension of structured encoding representations k. For all datasets, h and k are set to be equal, and we search them over the range {8, 12, 16}. Additionally, we search the dimension of query vectors q in {5, 7, 9}, the dimension of variables' node embedding n in {7, 9, 11} and the proportion of density score α in {1.0, 2.0, 3.0}. The best hyperparameters for each dataset are reported in the code. In KEDGN, we utilize a PLM to extract sentence embeddings of variables' textual medical information as variable semantic representations. The choice of PLM is diverse, and in our default implementation, we use BERT. Additionally, we have experimented with other PLMs, including T5 [55] , Bart [56] , GPT2 [57] , LED [58] , and Pegasus [59] . The detailed information of these PLMs is listed in Table 5 . We extract the hidden state at the [CLS] position for BERT. For other models, we uniformly take the last hidden state (average pooling) of the models as the representations of the variables. The classification results of using different PLMs on four datasets are listed in Table 6 .The results show that BERT (Encoder-Only model) achieved high performance on most datasets. Several Encoder-Decoder-based models also demonstrated competitive results, with Bart outperforming BERT on the P12 dataset, LED slightly surpassing BERT on the Physionet dataset and T5 showing slightly lower performance than BERT on the MIMIC-III dataset. However, the Decoder-Only GPT exhibited notably lower results on the P19 dataset and MIMIC-III compared to other models. This situation may be attributed to the fact that, in our task, PLMs are used to extract sentence-level semantic representations, which are utilized to differentiate variables and measure correlations among them, resembling a text classification process among variables. On the one hand, Encoder-Only models may excel at understanding tasks such as text classification, with BERT specifically adding a special [CLS] token dedicated to extracting sentence-level overall semantic information. On the other hand, models incorporating a decoder involve tasks related to predicting the next word and may be more focused on text generation tasks. They may exhibit a slight deficiency in extracting distinctive sentence-level representations, especially for Decoder-Only models. These models sometimes require carefully designed prompts to guide them in generating high-quality outputs.In summary, variable semantic representations extracted by different PLMs can impact downstream task performance. Choosing a PLM that is more suitable for the task and dataset can further enhance the performance of KEDGN. In the context of the application in this paper, a simple pure encoder model may be more suitable for achieving optimal results. We conduct a analysis of the time and space overhead on the Physionet dataset, with a batch size of 128, and utilizing Nvidia 1080Ti GPU infrastructure. The results are shown in Table 7 . Our method achieves a balanced time and space overhead. The introduction of textual information involves generating semantic embeddings using PLMs, and the adjustment of the variable graph based on observed local density, both of which can be predetermined and integrated into the preprocessing step without increasing model training overhead. Attention-based methods (SeFT, mTAND) achieve parallel computing in the time dimension, resulting in low time overhead, but this sacrifices fine-grained feature extraction at the variable level. Furthermore, compared to the same RNN-based method GRU-D, our method only deals with actual observation points at each timestamp, thus significantly reducing time overhead. While our method's runtime is 1.3 times that of the latest SOTA model, Warpformer, our space overhead is only 16% of its size. In situations where computational resources permit, our method can further reduce runtime by employing the space-for-time trade-off strategy (such as increasing batch size). We find that the model performance declines after introducing text information on P12 dataset. As indicated in Table 6 , when we replace PLM from BERT to Bart, there is a slight improvement of 0.5% in AUPRC on the P12 dataset, slightly outperforming the model without text. Therefore, the reason for the decline in model performance is likely attributed to the choice of default PLM (BERT), which may not be optimal for extracting textual embeddings on this dataset, rather than the text itself causing the decline. In other words, introducing text offers the potential for enhancing model performance, with the extent of improvement depending on the degree of effective utilization of text.F.5 Effects of different activation functions in Eq.( 7) On one hand, the activation function in Eq.( 7) reflects the dynamics of variables, such as time decay or exponential increase. On the other hand, this activation function serves a normalization purpose because the edge weights in the knowledge-empowered complete graph A are normalized values. If directly using the absolute value of the density to adjust the edge weights, the values might become excessively large or small, which would severely disrupt the basic graph structure learned from textual knowledge. Typical activation functions with normalization capabilities include Sigmoid and Tanh.As shown in Table 10 , Tanh is chosen since it performs better. As shown in Table 11 , the parameter count of our model is not particularly high. It is on the same order of magnitude as Warpformer and significantly lower than Raindrop by three orders of magnitude.Although we calculate an independent parameter space for each variable, the total W t ∈ R V ×I×O does not equate to the parameter count. W t is derived from the multiplication of two matrices: the variable embedding matrix Q ∈ R V ×q and the weight matrix W ∈ R q×I×O . The first matrix is computed from textual embeddings, and only the second matrix belongs to the model parameters.The sizes q, I, and O are hyperparameters, independent of the number of variables, and typically set to be less than 16. This ensures that the parameter complexity of our model remains within an acceptable range. We conduct T-SNE visualization analysis on variable representations obtained from different PLMs. Figures 12 and 13 correspond to the results of the P19 and MIMIC-III datasets, respectively. We observe that when the clusters of variable representations extracted by PLMs are consistent with the grouping of variable time series patterns and exhibit distinctiveness, the corresponding downstream classification task performance tends to be better. For example, on the P19 dataset, BERT and LED achieve higher classification performance, with their corresponding variable representations having good distinctiveness: the three groups of variables represented by blue, yellow, and green colors show high cohesion and low coupling, while the T5 model, which exhibits suboptimal classification performance, has a distribution where Resp, WBC, and Platelets are notably confused with the green variables. Additionally, in the MIMIC-III dataset, the variable representations obtained from BERT and T5, which achieve higher performance, with blue and green clusters having longer distances, indicating better distinctiveness. On the other hand, variable representations obtained from GPT2 show poor distinctiveness, with the variable RR being close to GCS-EO and GCS-MR. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper presents a novel approach to tackle a critical problem in healthcare data analysis: the irregular sampling of time series data, which is common in medical applications. The Knowledge-Empowered Dynamic Graph Network (KEDGN) leverages pre-trained language models (PLMs) to generate semantic embeddings for each variable based on their medical properties. This enables the dynamic modeling of inter-variable correlations that evolve over time. By incorporating variable-specific parameter spaces and dynamically adjusting the variable graph, the proposed model enhances the ability to capture fine-grained temporal dependencies and correlations in ISMTS, significantly outperforming existing methods. This innovative integration of textual knowledge with dynamic graph neural networks represents a meaningful contribution to the field, offering potential improvements in clinical decision-making and predictive modeling.

Furthermore, the proposed work is well-grounded in extensive experimentation on multiple healthcare datasets, showcasing the effectiveness of KEDGN against state-of-the-art methods. The detailed analyses, including an ablation study and visual exploration of the learned variable correlations, provide a comprehensive understanding of the model's capabilities and behavior. It effectively communicates the importance of modeling variable-specific and time-varying correlations in healthcare time series, a topic that is highly relevant to the machine learning and medical communities. The incorporation of medical knowledge through PLMs, combined with the novel graph-based approach, provides a robust framework for future research in irregularly sampled time series modeling.

Reviews: review: paper_summary: paper_summary: This paper investigates the problem of irregularly sampled medical timeseries classification. The core idea is to use PLM to obtain semantic embeddings for variables, which are used to form a variable correlation graph. Then, the variable correlation graph is dynamically adjusted with the observations, based on which a spatiotemporal graph neural network is used to learn timeseries representation for classification. Typical experiments on widely used benchmarks are conducted to evaluate model performance.
strength_weakness: strength_weakness: 1. The parameter settings for baselines are not fair. In lines 228-230, the authors simply follow parameters in reference. However, such setting is not a fair way comparison due to several reasons: Some datasets like MIMIC-III for experiments are not public, i.e., although we have the same raw data source, the experimented datasets are never exactly same, and may be very different from distribution and statistics. This can be validated from the comparison between the results of Raindrop on P19 dataset from this paper and Raindrop paper. The performance has a significant difference, which implies the parameters should searched completely. Thus, grid search for baseline parameters is necessary.

2. The choice of GCRNN here is not well clarified. There are many spatiotemporal graph neural networks like STGNN /ST-GCN which can be used here. Moreover, the ablation study of graph convolution and temporal modeling for GCRNN should be conducted to evaluate the performance gains.

3. Some claims or statements intend to emphasize the model performance, which however is obviously set by design itself instead of model learning. For example, in Line 337 the authors state ''no correlation is observed with DiasABPas it has not been observed yet''. Actually, this is not by learning but the initial masking operation of adjacency matrix.

4. An important baseline of irregularly sampled medical timeseries modeling is missed here, i.e., StraTS [1].

5. The ablation study doesn't identify very effective module of the proposed work. For example, Text, KEE and DAG's performance gains are marginal, so are these module necessary?
[1] Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series, TKDD.

Typos:
1. The symbols or bold texts in Equation (10-12) are wrong.
questions: questions: 1. What's the function of activation function $\sigma$ in equation (7). In my opinion, this activation function reflects the dynamics of variables, like time decay or exponential increase. So, how to determine the function and what are the effects of different activation functions.
2. How is the parameter complexity of the model? Since each variable will require an independent $W_i$, the computation complexity might be very large.
3. This paper emphasizes the importance of using text information of variables. However, the performance of different text sources seems close. Especially, only using Name can achieve similar even better performance. From ablation study in Table 3, we can observe even without textual representations, the model can achieve almost similar performance. Thus, this part looks useless, which is opposite to the commonsense. 
4. Is Figure 5(a) calculated from f(E) or E? If in the f(E), the authors should also provide the correlation graph from E, so that we can evaluate whether model learning is effective. To my experience, even bert embedding of these variables can achieve similar correlation graph. Also, why graph (a) is not symmetric? What's meaning of different correlations between (FiO2, HR) and (HR, FiO2)?
5. What's the basic time resolution or sampling rate for each dataset?
6. How is the static features combined with timeseries are not described in paper.
7. How are the input length and prediction length determined are not provided. Usually, in mortality prediction, we use the first 48 hours ICU data as input to predict whether the patient will die in the hospitalization. What's setting for this paper?
limitations: limitations: The authors didn't comprehensively discuss the limitations of this paper, which is not accord with the paper checklist.
paper_summary: paper_summary: Throughout this article, the authors have described an approach (KEDGN) to tackle Irregularly Sampled Medical Time Series. In this task the data is composed of multiple variables represented with time series. The sample rate of a time series can be irregular and two time series can have their samples taken at a different timestamp.
In the approach described the authors propose to first find common knowledge correlation between the variables in the dataset. To do so, they compute a non-linear cosine similarity between embeddings of textual descriptions, obtained using a Pre-trained Language Model.
They then precise this common knowledge correlation by adding a timestamp aware correlation. Finally using this correlation information and a Graph Convolutional Recurrent Network, their approach can predict the class expected of the time series.
They have then analysed the results of their approach first through a benchmark with multiple state of the art approaches. They then added an ablation study and different visual analysis to give a better overview of the behavior of the approach.
strength_weakness: strength_weakness: * Details of the Proposed Model could be better explained by giving the intuition behind the equations. (Especially for 4.3.2 and 4.4)
questions: questions: * Did the authors try to use definitions given by health care professionals to confirm the definition given by the GPT model?
* It is not clear how equation (7) uses equation (6). Indeed, $Z^{(i)}_{v}$ was defined with $i$ being the $i$-th observation point (I believe the $i$-th sample of this specific variable $v$).
But on the other hand the equation (7) uses $Z$ with $t$ being a timestamp. --

It is not clear if $Z^{(i)}_{v}$ is defined for a timestamp in which no observation was made for a variable.
* It is not clear why the equation (9) contains a transpose.
limitations: limitations: It appears the limitations have been discussed in sufficient detail.
paper_summary: paper_summary: This work considers the irregular timestamp contained in current medical data. They employ a density-aware mechanism to the time-varying correlations among variables.
strength_weakness: strength_weakness: 1. In formulation (6), the density of the observation point is calculated by the average time interval. What if the point is not observed in the next or previous time? e.g. t_v^(i+1) or t_v^(i) is None.
2. I am just a little bit confused by the node in the dynamic variable graph. As we know, the patient's EHR data will contain lots of clinical notes and some like treatment, medications, and so on. You have shown the description in your model's framework in Figure 3. So do you consider other variables in patient's EHR data? Besides, the node here denotes a text, not detailed medical terminology, right?
3. Could you please add a comparison graph between the original variable representation and the final representations?
questions: questions: see weakness
limitations: limitations: Yes
paper_summary: paper_summary: The paper aims to handle difference and correlation between multiple variables in the irregularly sampled medical time series data. By proposing a model named KEDGN with textual medical knowledge and dynamic variable graphs, it is able to outperforms baselines on a variety of datasets. Comprehensive experiments have also been conducted to demonstrate the learned correlations between variables, which aligns with the motivation of this paper.
strength_weakness: strength_weakness: 1.	The idea of learning dynamic graphs that model variable relationships is quite common in literatures. It would be better for the authors to cite related works, compare with existing works and make arguments about novelty and effectiveness of the proposed modules. Here are some references:
    -	Jiang, Renhe, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. "Spatio-temporal meta-graph learning for traffic forecasting." In Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 7, pp. 8078-8086. 2023.
    -	Huang, Qihe, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. "Crossgnn: Confronting noisy multivariate time series via cross interaction refinement." Advances in Neural Information Processing Systems 36 (2023): 46885-46902.
    -	Wang, Dingsu, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. "Networked time series imputation via position-aware graph enhanced variational autoencoders." In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2256-2268. 2023.
    -	Wang, Binwu, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. "Towards Dynamic Spatial-Temporal Graph Learning: A Decoupled Perspective." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, pp. 9089-9097. 2024.
2.	There exist some missing references and baselines for irregularly sampled time series classification problems. It would be better to cite related works and compare with them in the experiment sections. Here are some examples:
    -	Li, Zekun, Shiyang Li, and Xifeng Yan. "Time Series as Images: Vision Transformer for Irregularly Sampled Time Series." Advances in Neural Information Processing Systems 36 (2024).
    -	Labach, Alex, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. "Duett: Dual event time transformer for electronic health records." In Machine Learning for Healthcare Conference, pp. 403-422. PMLR, 2023.
questions: questions: 1.	It seems some of the baselines such as DGM^2-O can reach good performance with relatively less time and space requirements. Can there be any potential improvements to the efficiency of the proposed KEDGN model? It seems BERT is also optimized during training, it would be interesting to see some efficiency experiments and analysis regarding this aspect.
limitations: limitations: N/A

Comments: title: Response to rebuttal
comment: Dear Reviewer vHK1,

Hope this message finds you well.

We appreciate the diligent efforts of you in evaluating our paper. We have responded in detail to your questions. As the discussion period will end soon, we would like to kindly ask whether there are any additional concerns or questions we might be able to address.

Once more, we appreciate the time and effort you've dedicated to our paper.

Best regards,

Authors
The variables in patients' EHR data mean the diagnosis and procedure codes. For example, the diagnosis code reflects the disease the patient has. It also should be a node in your graph. It doesn't mean multiple modalities. I agree with you that you are focusing on the time series topic. But you discuss it in the field of clinical notes, I believe it would be better if you could make full use of the data. Thanks for your response, I will keep my score.
Thank you for the detailed response. All my concerns have been addressed properly and I'd like to keep my score.
# Responses to Reviewer vHK1(3/3)

**Q7. How are the input length and prediction length determined. **

A7. We apologize for the omission. As you mentioned, for the MIMIC-III, P12, and PhysioNet datasets, we use ICU data from the preceding 48 hours as input to predict the mortality during the hospitalization. For the P19 dataset, we use up to 60 hours of ICU data to predict whether sepsis will occur within the subsequent 6 hours. We will include these details in the "Appendix B: Datasets" section of the revised version.



**Q8. Limitations**

A8. Although our proposed method effectively guides ISMTS modeling through the domain knowledge from text modality, it has some limitations. The backbone of our model is based on an RNN architecture, which inherently has a sequential computation characteristic that can be a bottleneck in terms of runtime. Additionally, our method is specifically tailored for medical applications, and its performance may be limited in other ISMTS applications, such as human activity recognition, where variables may lack domain knowledge and thus cannot generate high-quality text descriptions. We will include a detailed discussion of these limitations in the revised version of the paper.