Title: Knowledge-Empowered Dynamic Graph Network for Irregularly Sampled Medical Time Series
Venue: NeurIPS 2024
Accepted: yes
Keywords: healthcare

irregularly sampled medical time series

graph neural network

recurrent neural network
Abstract: Irregularly Sampled Medical Time Series (ISMTS) are commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features at the variable level in ISMTS. We propose Knowledge-Empowered Dynamic Graph Network (KEDGN), a graph neural network empowered by variables' textual medical knowledge, aiming to model variable-specific temporal dependencies and inter-variable dependencies in ISMTS. Specifically, we leverage a pre-trained language model to extract semantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from a medical perspective. Based on this, we allocate variable-specific parameter spaces to capture variable-specific temporal patterns and generate a complete variable graph to measure medical correlations among variables. Additionally, we employ a density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variable-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra-variable and inter-variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods.

Knowledge-Empowered Dynamic Graph Network for Irregularly Sampled Medical Time Series

Abstract

Irregularly Sampled Medical Time Series (ISMTS) are commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features at the variable level in ISMTS. We propose Knowledge-Empowered Dynamic Graph Network (KEDGN), a graph neural network empowered by variables' textual medical knowledge, aiming to model variable-specific temporal dependencies and inter-variable dependencies in ISMTS. Specifically, we leverage a pre-trained language model to extract semantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from a medical perspective. Based on this, we allocate variable-specific parameter spaces to capture variable-specific temporal patterns and generate a complete variable graph to measure medical correlations among variables. Additionally, we employ a density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variable-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra-variable and inter-variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods. Our code is available at https://github.com/qianlima-lab/KEDGN .

1 Introduction

In the medical environment, the widely used Electronic Health Records (EHRs) have abundant typical Irregularly Sampled Medical Time Series (ISMTS) data [1]. Each ISMTS typically comprises multiple medical variables for a patient, each with distinct medical properties, resulting in significant differences in the sampling patterns of each variable series. Additionally, due to the dynamic changes in a patient's condition, the sampling rate of variables varies over different periods, resulting in uneven sampling intervals [2].

Many existing methods for ISMTS primarily focus on addressing uneven sampling intervals and have proposed approaches such as Ordinary Differential Equations (ODEs) [3, 4] and continuous-time embeddings [5, 6], etc, which have already achieved significant success. Recent advancements in regularly sampled multivariate time series analysis [7, 8] underscore the importance of capturing variable-specific temporal patterns. However, many existing methods for ISMTS have not adequately considered this aspect, having limited ability to explicitly distinguish multiple variable series with

different time patterns and thus lacking finer-grained capturing of variable-level features. Particularly in ISMTS, different variables have distinct medical properties, further intensifying the degree of differences among variables. In such cases, capturing differentiated variable patterns requires a deeper exploration of inherent differences among variables.

Despite the differentiated temporal patterns occurring among variables within ISMTS, they are not entirely independent but exhibit medical correlations. Due to the dynamic changes in the patient's condition, this correlation varies along with the sampling density of variables at different periods, as illustrated in Figure 1. Some existing work has introduced graph neural networks [9, 10] to model the time-varying correlations among variables in ISMTS. However, due to the lack of prior medical knowledge, these methods learn variable correlation graphs from misaligned and imbalanced observations in variables of ISMTS and rely solely on downstream tasks for graph optimization. Consequently, the variable graphs learned by these methods may face challenges in accurately reflecting the general medical correlations among variables, resulting in suboptimal performance.

To this end, we aim to explicitly consider the differences and correlations among variables in ISMTS, empowering the model to capture fine-grained features at the variable level. However, the sampling rates, sampling times, observation spans, and observation lengths of different variables vary in ISMTS, as shown in three subgraphs in Figure 2. This makes it tough and complex to infer these two aspects only from the time series modality of

1

2

variables. We rethink the issue based on an intuitive observation as shown in Figure 2: Variables that exhibit (dis)similar temporal patterns frequently have (dis)similar medical properties in reality, which motivates us to consider the differences and correlations of variables from the perspective of domain knowledge directly. Recent work [11, 12] has successfully represented domain knowledge through textual modal information, enhancing model performance in medical imaging, which provides us with insights. Specifically, the medical properties of each variable can be described in natural language. Leveraging the powerful semantic understanding capabilities of the Pre-trained Language model (PLM), we can obtain semantic representations from the textual knowledge of each variable. This set of textual representations forms an overall view of variables from the perspective of medical knowledge, clearly showing inherent differences and correlations among variables-exactly what we need.

Based on the above analysis, we propose the Knowledge-Empowered Dynamic Graph Network (KEDGN), which utilizes textual semantic representations of variables obtained through PLM as guidance. On this basis, we 1) allocate unique parameter space for each variable to capture their specific temporal pattern, 2) generate a complete variable graph and introduce a density-aware mechanism to explicitly model time-varying correlations among variables in ISMTS. Finally, these

(a) Invasive arterial blood pressure

(b) Renal function indicators

(c) Ion concentration indicators

two modules are integrated with a graph convolutional recurrent network to capture both temporal and inter-variable dependencies in ISMTS. Our contributions can be summarized in three aspects:

We leverage variable-specific textual medical knowledge to empower the model to capture variable-specific temporal patterns in ISMTS distinctively.

We introduce a density-aware mechanism based on the knowledge-empowered variable graph to model the time-varying inter-variable dependencies in ISMTS.

Empirical results on four real-world medical datasets demonstrate that KEDGN outperforms state-of-the-art methods. Visualization analysis further illustrates the strong interpretability of our approach.

2 Related Work

Irregularly Sampled Multivariate Time Series Modeling Existing methods can be roughly categorized into interpolation-based and raw data-based approaches. The former, employing methods such as kernel-based approaches [13, 14], Gaussian process [15] or hourly aggregation [16], aims to obtain a set of regularly spaced observations. However, interpolation may result in the loss of useful information about the original sequences, such as missing patterns. The latter, raw databased methods, directly learn from irregular time series. To adapt to uneven sampling intervals, [17] improves recurrent neural networks, [3, 4] introduce neural ordinary differential equations and [6, 5, 18, 19] adopt time embeddings. [20] converts ISMTS into line graph images and utilizes pretrained vision transformers for extracting features. These methods primarily focus on overall temporal dependencies, needing more consideration for fine-grained variable-level patterns and correlations. Despite recent research introducing attention [21, 22] or graph neural networks [9, 23] to account for variable correlations, these methods have limited performance due to the lack of prior knowledge and the use of shared parameter spaces among all variables.

Graph Neural Networks for Multivariate Time Series In recent years, a series of studies have integrated GNN with various time series modeling frameworks to effectively capture both intervariable and inter-temporal dependencies in MTS [24]. These approaches have been widely applied in diverse domains, including transportation [25], healthcare [26], economics [27], demonstrating promising results in mainstream tasks such as prediction [28], classification [29], and imputation [30]. Although recent work [31, 32, 33, 34] has proposed the idea of modeling variable relationships through learning dynamic graphs, most of these methods are primarily designed for regularly sampled MTS with synchronous observations, and further improvements are needed to adapt them for irregularly sampled time series.

Medical Knowledge Enhanced Models Several studies have utilized the rich domain knowledge in the medical field to enhance models. [12, 35] apply knowledge for computing additional features, while [36, 37] utilize knowledge to guide the final training loss, demonstrating the effectiveness of medical prior knowledge. However, existing methods commonly focus on visual language pretraining in medical scenes or medical report generation [11]. How to effectively integrate domain knowledge to guide medical time series modeling remains a challenge.

3 Problem Definition

Given a dataset D = { ( s i , y i ) | i = 1 , . . . , N } containing N patient samples, the i -th sample consists of an irregular multivariate time series s i and a label y i . For the dataset with a total variable count of V and a maximum sample observation length of T , s i can be denoted as a tuple: s i = ( t i , x i , m i ) , where t i ∈ R T represents the observation timestamps, x i ∈ R V × T represents the multivariate time series observations, and any unobserved values or the missing parts of the time series shorter than the maximum sample observation length are filled with 0. The binary indicator m i has the same size as x i , indicating which elements in the x i are actually observed. We use 1 to represent observed values and 0 to represent missing values. In this paper, we focus on patient mortality and morbidity prediction, i.e., classification task, aiming to correctly predict class label ˆ y i given a sample s i .

Query Vectors

Heart rate can be used as

an indicator of how well a.…

Body temperature is

typically measured using …

MAP reflects the average

pressure in a patient's …

Text description of variables'

medical properties

Pretrained

4

4

1

3

4

2

3

(2) Variable-specific Parameter Learning

Density-aware Dynamic Adjustment

Variable Correlation Graph

Input Representations

3

1

1

2

2

5

5

5

q

q

×

…

Weights Pool

…

𝑾𝟏 ∈ ℝ 𝑰

×

𝑶

𝑾𝟐 ∈ ℝ 𝑰

×

𝑶

𝑾𝟓 ∈ ℝ 𝑰

×

𝑶

𝑾 ∈ℝ 𝑰

×

𝑶

Pair-Wise

Cosine Similarity

Unobserved variables

×

Matrix multiplication

…

Node

Embeddings

…

n

(3) Dynamic Variable Graph Generation

Frozen

Projection heads

G (0)

G (1)

G (2)

(1) Variable Semantic

Representations Extraction

Observed variables

(4) Graph Convolutional Recurrent Network

𝒇

𝒈

𝒇

/

𝒈

t = t

0

5

2

1

t = t

1

3

5

1

t = t

2

5

2

4 The Proposed Model

4.1 Variable Semantic Representations Extraction

First, we introduce how to extract semantic representation for each variable from medical knowledge. Let V = { v 1 , v 2 , . . . , v V } be the set of variables, and a descriptive sentence of medical properties associated with the j th variable can be denoted as:

where l j is the length of the j th variable's sentence and w j,i denotes the i th word of the j th sentence. We leverage a PLM to represent each text description P j as a d -dimensional embedding. Considering the diverse types of PLMs with varying methods of utilization, we use the widely adopted encoderbased model BERT [38] as an illustration:

where [CLS] and [SEP] are special tokens indicating a sequence's beginning and end, respectively. BERT generates an embedding for each token of the input sequence. Since the embedding at the [CLS] position captures the semantic information of the entire input sequence, we take the hidden state at the [CLS] position as the overall semantic representation of each variable: E j = e j ([CLS]) . This yields a semantic representation matrix E = [ E 1 , E 2 , ..., E V ] ∈ R V × d and forms an overall view of variables from the perspective of medical knowledge.

4.2 Variable-specific Parameter Learning

Since temporal patterns of ISMTS vary from variable to variable, simply using shared parameter space for all variables is insufficient to capture differentiated temporal dependencies. In this section, we adjust parameter space for different variables to adapt to differentiated temporal patterns based on the extracted variables' semantic representations. For any parameter matrix w ∈ R I × O with input dimension I and output dimension O , the total parameter space needed for V variables is Θ ∈ R V × I × O . Inspired by [39], we decompose Θ into two matrices: a variable representation matrix Q ∈ R V × q and a weight pool matrix W ∈ R q × I × O , where q is a hyperparameter for the intermediate dimension. Here, Q consists of q -dimensional query vectors for V variables used to distinguish differences among variables and obtain variable-specific parameters from the weight pool.

We use a projection f ( · ) : R d → R q to variable semantic representations E for obtaining query vectors rather than directly using E . On the one hand, the dimension of the query vector q directly determines the size of the weight pool, but the output dimension of PLM is often large (e.g., 768 in BERT), leading to a sharp increase in model complexity. On the other hand, there is a modality gap between textual embeddings and the temporal parameter space. The projection f ( · ) can achieve both feature reduction and modality transformation. In our implementation, we use a nonlinear projection with one additional hidden layer (and ReLU activation). Thus, the parameter space specific to variable i can be obtained using the following formula:

The approach we employ to generate variable-specific parameter space is general and not restricted to a specific model backbone because any model architecture is composed of multiple parameter matrices W .

4.3 Dynamic Variable Graph Generation

In this section, we introduce how to generate dynamic correlation graphs of variables for explicitly modeling the time-varying correlations among variables in ISMTS.

4.3.1 Complete Variable Correlation Graph Learning

The misaligned and imbalanced observations of the variables in ISMTS make it difficult to learn the variable correlations from the time series. Therefore, we extract a static complete variable correlation graph based on the textual semantic representations of the variables directly from the perspective of the actual medical properties of variables. We apply another non-linear projection g ( · ) : R d → R n to the textual representations of variables E to obtain n -dimensional node embeddings for each variable. Subsequently, we calculate the pairwise cosine similarity among the node embeddings of variables, resulting in a V × V matrix of variable similarity. Finally, we use the softmax function to normalize the edge weights corresponding to each node, producing a normalized graph of variable correlations. The correlation weight between the i th and j th variables can be calculated as:

where · represents vector dot product and ∥ · ∥ represents the vector magnitude. The introduction of g ( · ) in this context not only performs feature reduction but also avoids using a completely fixed prior graph. It preserves the model's ability to adaptively optimize the graph structure based on different data distributions and downstream tasks. Thus, we obtain a knowledge-empowered complete graph with V nodes to measure the static correlation among variables in general medical cases, and its adjacency matrix is denoted as A .

4.3.2 Dynamic Density-aware Adjustment Mechanism

Due to the varying subsets of variables observed at each timestamp in ISMTS, we use different subgraphs of A to describe the variable correlations at different timestamps. Specifically, we use a mask matrix M ( t ) ∈ R V × V to indicate the subgraph topology at timestamp t :

Therefore, we can calculate the variable correlation subgraph A ( t ) at timestamp t through A ( t ) = A ⊙ M ( t ) , where ⊙ represents Hadamard product. Additionally, we introduce a density-aware mechanism to dynamically adjust edge weights of subgraphs in different timestamps to fit in the time-varying correlations among variables mentioned in Figure 1. Specifically, we estimate the sampling density of any observation point by considering the average time interval between each observation point and its preceding and succeeding observations. If there is no preceding/succeeding observation, we take the time interval of the succeeding/preceding observation as the density. If neither a preceding nor a succeeding observation exists, it indicates that this observation is the only

one for the variable, and we take half of the maximum observation time span as the density. The formula for calculating the sampling density of the i -th observation of variable v at timestamp t is:

Then we calculate the density scores for various variables at timestamp t through:

where σ is an activation function and α is a hyperparameter that controls the proportion. At timestamp t , the edge weight between the i th and j th variables is adjusted as:

where W ∈ R V × V is a learnable parameter matrix. Thereby, we achieve the dynamic adjustment of the variable graph weights in response to changes in variable sampling density.

4.4 Variable-specfic Dynamic Graph Convolutional Recurrent Network

Under the empowerment of variable textual representation, we have obtained variable-specific parameters Θ ∈ R V × I × O and dynamic variable graph G ∈ R T × V × V . In this section, we integrate these two modules into the graph convolutional neural network to handle ISMTS. GCRNN [40] is a backbone network that introduces graph convolutional operations on top of an RNN variant, Gated Recurrent Unit [41]. This structure is simple, effective, and easy-to-adapt for ISMTS, as it enables variable-level parallel computation of asynchronous observations without explicit interpolation. Specifically, we allocate a unique hidden state for each variable, updating the state only at the observed timestamps to avoid imputation and preserve the individual sampling patterns of each variable. The graph convolution operation over a graph signal S ∈ R V × I containing V nodes at timestamp t is defined as follows:

where I V ∈ R V × V is identity matrix, × represents batch matrix multiplication. Here, we adopt 1 st -order Chebyshev polynomial expansion approximation [42] for graph convolution. The updated formulas for variable states at timestamp t are:

where || denotes the concatenate operation, H ( t -1) ∈ R V × h is the variable states at the previous timestamp and X ( t ) ∈ R V × k denotes the input representation at current timestamp. We follow the structured input encoding method of [22], using multiple fully-connected mappings to encapsulate each observed value and its corresponding timestamp into a k -dimensional input representation (All 0 vectors for unobserved values) to indicate flexible observation time and adapt to the uneven intervals within variables. r ( t ) , u ( t ) ∈ R V × h are reset gate and updated gate, respectively. Θ r , Θ u , Θ C ∈ R V × ( k + h ) × h are variable-specific parameters obtained by respectively multiplying the query vectors matrix Q ∈ R V × q with three weight matrices W r , W u , W C ∈ R q × ( k + h ) × h .

We calculate the sum of h channels for each variable's hidden state H i at the last observed timestamp to get a V -dimensional vector C . Additionally, we follow the approach used in [9] to incorporate the static features. Specifically, static features of each sample are mapped into a static vector S through a linear layer. Finally, C and S are concatenated to predict the final classification probabilities: ˆ y = Softmax ( W y [ C || S ] + b y ) . The training objective is minimizing the cross-entropy loss between ˆ y and y . The pseudo-code for KEDGN is presented in Appendix A (Algorithm 1).

5 Experiment

5.1 Experimental Setting

Datasets and Baselines We conduct experiments on four widely used irregular medical time series datasets, namely P19 [43], Physionet [44], MIMIC-III [45] and P12 [46] where Physionet is a reduced version of P12 considered by prior work [6]. We compare our method with the state-of-the-art methods for modeling irregular time series, including GRU-D [17], ODE-RNN [4], IP-Net [14], SeFT [5], mTAND [6], Raindrop [9], StraTS [18], DuETT [19], ViTST [20] and Warpformer [22]. In addition, we also compare our method with two approaches initially designed for forecasting tasks, namely DGM 2 -O [13] and MTGNN [47]. The implementation and hyperparameter settings of these baselines are kept consistent with those used in [9]. More details of datasets and baselines can be found in Appendix B and C.

Evaluation Setup For the data pre-processing of MIMIC-III, we follow the method described in [48] and divide the dataset into three parts for training, validation, and testing with the ratio of 70%,15%,15%. For the remaining three datasets, we adhered to [9] 's approaches, and the ratio of training, validation, and testing set is 8:1:1. We measure the classification performance with the Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC) since all the four datasets are binary classification datasets with highly imbalanced class distribution. AUPRC has better sensitivity to sample imbalance [49]; thus, the optimal model parameters that achieve the best AUPRC on the validation set are used for the test set. More details of metrics can be found in Appendix D.

Implementation Details We adopt the Adam [50] optimizer, and the number of training epochs is set as 10. Due to differences in dataset sizes, the learning rate is set as 0 . 001 for Physionet and P12 and 0 . 005 for MIMIC-III and P19. The textual sources for variable descriptions are flexible; we chose three sources, including the variable's full name, Wikipedia source, and ChatGPT source (Default), corresponding to model names KEDGN-Name, KEDGN-Wiki, and KEDGN-ChatGPT, respectively. All experiments are conducted with five random seeds, and the average and standard deviation are reported. More implementation details and hyperparameter settings can be found in Appendix E.

5.2 Main Results

Classic time series classification The evaluation results are summarized in Table 1, in which we use BERT to extract variables' semantic representations. More experimental results and analyses using other PLMs can be found in Appendix F.1. Overall, our KEDGN achieves the best performance on all four datasets and outperforms the strongest baseline by an average of 0.9% on AUROC and

1.6% on AUPRC. In addition, the results of models using different text sources are similar, which demonstrates that our method is not limited to a specific text source and exhibits generalizability. We also provide an analysis of computational costs in Appendix F.2.

Leave-variables-out To demonstrate the robustness of our method, we test whether KEDGN can achieve good performance when a subset of variables is completely missing. We uniformly discard 10%, 20%, 30%, 40%, and 50% of the variables, hiding all their observations in both validation and test sets. Table 2 reports the results on the MIMIC-III dataset, while the results for the remaining datasets are presented in Appendix F.3 (Table 8). Our method achieves the highest performance on 35 out of 40 metrics across 4 datasets as the missing rate increases from 10% to 50%. This may be attributed to KEDGN only handling actually observed points, avoiding the accumulation of imputation errors, particularly in cases of higher missing ratios, thus exhibiting a degree of robustness.

5.3 Ablation Study

In this section, we investigate the performance benefits generated by each key component of the proposed method on all four datasets. We compare the full versioned method with its six variants: (1) w/o VSW : We apply shared RNN parameter weights for all variables; (2) w/o Text : We replace the variable-specific textual semantic representations E i with randomly initialized learnable embeddings; (3) w/o Graph : We set the graph G to be a fully zero matrix, disregarding dependencies among variables; (4) w/o KEE : We replace knowledge-empowered node embeddings of variables g ( E i ) with randomly initialized learnable embeddings, (5) w/o DAG : We remove the density-aware adjustment for edge weights of the graph, using static adjacency matrix A ( t ) during different periods. (6) w/o TE : We remove the timestamp embedding part of the structured input encoding X ( t ) . The results on the P19 dataset are presented in Table 3, while the results for the

remaining datasets are presented in Appendix F.4 (Table 9). The results show that all model components are necessary and variable-specific parameter space makes the most significant contribution to the performance of KEDGN.

5.4 Visualization Analysis

5.4.1 Visualization of Variables Textual Representations

In this section, we explore why textual information is effective for time series modeling through visualization analysis. We first group variables with similar time patterns on the P12 dataset, as illustrated in Figure 2 (More variables groups can be found in Appendix F.7 (Figure 7)). Subsequently, we use T-SNE [51] to visualize the distribution of variable semantic representations. Figures 4a and 4b respectively display the distributions of ChatGPT and Wikipedia sources, while Figure 4c shows the final distribution learned by replacing the text representation with randomly initialized learnable embeddings. Variables with similar time patterns are labeled with the same color. It can be observed

(a) ChatGPT

(b) Wikipedia

(c) Random Initialized

that the textual representation space of variables exhibits distinct clustering. Although there may be occasional outliers when using different text sources, such as BUN in Figure 4a and HCO3 in Figure 4b, the overall clusters are generally consistent with the variable groups we divide based on time series patterns. However, the learnable variable embeddings in Figure 4c, optimized by classification loss, tend to be distributed uniformly, which is difficult to effectively reflect the intrinsic differences among variables. More visualizations of other datasets and other PLMs can be found in Appendix F.8 and F.9, respectively. Based on this phenomenon, we infer that text descriptions and time series are both external manifestations of the inherent sense of variables; they just belong to different data modalities. These two forms of data for the same variable should exhibit relative consistency. Therefore, the relative distribution among variables extracted from text and time series should ideally be similar. Leveraging PLM allows for the straightforward and efficient extraction of this universal view from textual descriptions, which is equally applicable to describing the relative distribution of temporal patterns among variables. Thus, the effectiveness of PLM and the cross-modal relative consistency are the keys to guiding time series modeling based on textual information.

5.4.2 Visualization of Variable Correlation Graph

(a)

(b)

We visualize the learned partial inter-variable correlation graphs on the MIMIC-III dataset in the form of heatmaps. In Figure 5a, we depict the graph learned based on node embeddings mapped from textual representations, while in Figure 5b, the graph is learned based on randomly initialized learnable node embeddings. We observe that the top-left and bottom-right corners of the heatmap in Figure 5a exhibit darker colors, indicating strong correlations consistent with medical domain knowledge. Specifically, the variables in the top-left corner, Heart Rate (HR) and Respiration Rate (RR), are commonly monitored together in clinical settings for assessing vital signs and respiratory system function [52]. The variables in the bottom-right corner, GCS-MR, GCS-T, and GCS-VR, collectively constitute the Glasgow Coma Scale (GCS), which is used to assess a patient's neurological status and level of consciousness [53]. These correlations are not evident in the graph without textual representations. This once again validates our perspective that relying solely on downstream task optimization for adaptive learning in graphs in ISMTS is insufficient to reflect the actual medical correlations among variables and lacks interpretability. In contrast, textual representations can guide the model to accurately extract variable correlations aligned with domain knowledge to provide high interpretability.

5.4.3 Visualization of Dynamic Density-aware Graph

In Figure 6, we visualize the dynamic density-aware graph for the sample in Figure 1. We present the time series of three variables and the corresponding correlation heatmaps learned by our model at

timestamps 4 , 15 and 56 . From the time series, we observe that around t = 4 , HR shows a strong correlation with NIDiasABP, while the correlation with DiasABP is masked as 0 since DiasABP has not been observed yet. Around t = 15 , the correlation between HR and NIDiasABP decreases, while a relatively strong correlation with DiasABP emerges. By t = 56 , HR exhibits a strong correlation with both variables. This process is clearly reflected in the heatmaps: the color between HR and NIDiasABP transitions from dark to light from t = 4 to t = 15 and darkens again from t = 15 to t = 56 . The color between HR and DiasABP remains dark at t = 15 and t = 56 . This demonstrates that our dynamic density-aware mechanism exactly

reflects the time-varying correlations among variables in ISMTS.

6 Limitations

Although our proposed method effectively guides ISMTS modeling through the domain knowledge from text modality, it has some limitations. The backbone of our model is a recurrent-based architecture, which inherently has a sequential computation characteristic that can be a bottleneck in terms of runtime. Additionally, our method is specifically tailored for medical applications, and its performance may be limited in other irregular multivariate time series applications, such as human activity recognition, where variables lack domain knowledge and thus cannot generate high-quality text descriptions.

7 Conclusion

In this paper, we propose KEDGN for modeling ISMTS. The proposed method leverages a PLM to flexibly extract semantic representation for each variable from the textual medical knowledge. Based on these representations, we allocate the variable-specific parameter space to capture variable-specific temporal patterns and extract a complete variable graph as a measure of the variables' static medical correlations. Considering the time-varying variables correlations in ISMTS, we introduce a densityaware mechanism to dynamically adjust the subgraph across different periods. Our experimental results demonstrate that KEDGN outperforms existing methods in ISMTS classification tasks and provides high interpretability. Our future work will focus on investigating the applicability of KEDGN in a range of related tasks, such as interpolation, extrapolation, and regression.

References

Chenxi Sun, Shenda Hong, Moxian Song, and Hongyan Li. A review of deep learning methods for irregularly sampled medical time series data. arXiv preprint arXiv:2010.12493 , 2020.

Satya Narayan Shukla and Benjamin M Marlin. A survey on principles, models and methods for learning from irregularly sampled time series. arXiv preprint arXiv:2012.00168 , 2020.

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In Advances in Neural Information Processing Systems , volume 32, 2019.

Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning , pages 4353-4363. PMLR, 2020.

Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations , 2021.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations , 2023.

Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, ICLR , 2022.

Zhen Wang, Ting Jiang, Zenghui Xu, Jianliang Gao, and Ji Zhang. Irregularly sampled multivariate time series classification: A graph learning approach. IEEE Intelligent Systems , 2023.

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21372-21383, 2023.

Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications , 14(1):4542, 2023.

Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan B Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 651-659, 2021.

Satya Narayan Shukla and Benjamin Marlin. Interpolation-prediction networks for irregularly sampled time series. In International Conference on Learning Representations , 2019.

Qingxiong Tan, Mang Ye, Grace Lai-Hung Wong, and Pong Chi Yuen. Cooperative joint attentive network for patient outcome prediction on irregular multi-rate multivariate health data. In IJCAI , pages 1586-1592, 2021.

Liantao Ma, Junyi Gao, Yasha Wang, Chaohe Zhang, Jiangtao Wang, Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 825-832, 2020.

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports , 8(1):6085, 2018.

Sindhu Tipirneni and Chandan K Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD) , 16(6):1-17, 2022.

Alex Labach, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. Duett: Dual event time transformer for electronic health records, 2023.

Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Sheo Yon Jhin, Minju Jo, Taeyong Kong, Jinsung Jeon, and Noseong Park. Ace-node: Attentive co-evolving neural ordinary differential equations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 736-745, 2021.

Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. Warpformer: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 3273-3285, New York, NY, USA, 2023. Association for Computing Machinery.

Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 364-373, 2021.

Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. arXiv preprint arXiv:2307.03759 , 2023.

Saeed Rahmani, Asiye Baghbani, Nizar Bouguila, and Zachary Patterson. Graph neural networks for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems , 2023.

Lijing Wang, Aniruddha Adiga, Jiangzhuo Chen, Adam Sadilek, Srinivasan Venkatramanan, and Madhav Marathe. Causalgnn: Causal-based graph neural networks for spatio-temporal epidemic forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 36, pages 12191-12199, 2022.

Jianian Wang, Sheng Zhang, Yanghua Xiao, and Rui Song. A review on graph neural network methods in financial applications. arXiv preprint arXiv:2111.15367 , 2021.

Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang, Junbo Zhang, and Yu Zheng. Spatio-temporal graph neural networks for predictive learning in urban computing: A survey. IEEE Transactions on Knowledge and Data Engineering , 2023.

Huaiyuan Liu, Xianzhang Liu, Donghua Yang, Zhiyu Liang, Hongzhi Wang, Yong Cui, and Jun Gu. Todynet: Temporal dynamic graph neural network for multivariate time series classification. arXiv preprint arXiv:2304.05078 , 2023.

Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g_ap_s: Multivariate time series imputation by graph neural networks. In International Conference on Learning Representations , 2022.

Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. Spatio-temporal meta-graph learning for traffic forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 8078-8086, 2023.

Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refinement. Advances in Neural Information Processing Systems , 36:46885-46902, 2023.

Dingsu Wang, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Networked time series imputation via position-aware graph enhanced variational autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2256-2268, 2023.

Binwu Wang, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Towards dynamic spatial-temporal graph learning: A decoupled perspective. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 9089-9097, 2024.

Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1439-1449, Online, November 2020. Association for Computational Linguistics.

Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng. Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading. IEEE transactions on medical imaging , 39(5):1483-1493, 2019.

Qing Liao, Ye Ding, Zoe L Jiang, Xuan Wang, Chunkai Zhang, and Qian Zhang. Multi-task deep convolutional neural network for cancer diagnosis. Neurocomputing , 348:66-73, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems , 33:17804-17815, 2020.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR '18) , 2018.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 , 2014.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC) , pages Page-1. IEEE, 2019.

Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting inhospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology , pages 245-248. IEEE, 2012.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation , 101(23):e215-e220, 2000.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 753-763, 2020.

Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data , 6(1):96, 2019.

Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning , pages 233-240, 2006.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Craig Lockwood, Tiffany Conroy-Hiller, and Tamara Page. Vital signs. JBI Evidence Synthesis , 2(6):1-38, 2004.

George L Sternbach. The glasgow coma scale. The Journal of emergency medicine , 19(1):67-71, 2000.

Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. volume 06, 06 2006.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. CoRR , abs/1910.13461, 2019.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150 , 2020.

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.

A Algorithm of KEDGN

Algorithm 1 The pseudo-code of KEDGN

Input : An ISMTS sample observation values matrix: x (1: T ) ∈ R V × T , binary mask matrix m (1: T ) ∈ R V × T , observation timestamps t (1: T ) ∈ R T , the sentences of V variables' medical properties P = { P 1 , P 2 , . . . , P V }

Output : Predicted classification label ˆ y

1: extract semantic representations E ∈ R V × d for variables from P through PLM using Eq.(2)

2: obtain variable-specific parameter space Θ (including reset gate Θ r , update gate Θ u and candidate gate Θ C ∈ R V × ( k + h ) × h ) using Eq.(3)

3: obtain complete variable correlation graph A using Eq.(4)

4: initialize the hidden state of V variables H (0) = 0 V × h

5: for i = 1 , 2 , ..., T do

6: obtain dynamic variable subgraph G ( i ) ∈ R V × V according to m ( i ) and Eq.(8)

7: encode observation values x ( i ) and timestamp t ( i ) into X ( i ) ∈ R V × k by multiple fullyconnected mappings

8: update H ( i ) = GCRNN ( Θ , G ( i ) , X ( i ) , H ( i -1) ) using Eq.(10), (11), (12) and (11)

9: end for

10: calculate the sum of h channels for H ( i ) to get C ∈ R V

11: calculate ˆ y = Softmax ( W y C + b y )

12: return ˆ y

B Datasets

We use four irregularly sampled medical time series datasets to evaluate the classification performance of our model and baseline models. The dataset statistics are summarized in Table 4.

P19 The PhysioNet Sepsis Early Prediction Challenge 2019 [43] dataset consists of medical records from 38,803 patients. Each patient's record includes 34 variables of up to 60 hours and a static vector indicating attributes such as age, gender, the time interval between hospital admission and ICU admission, type of ICU, and length of stay in the ICU measured in days. Additionally, each patient is assigned a binary label indicating whether sepsis occurs within the subsequent 6 hours. We follow the procedures of [9] to ensure certain samples with excessively short or long time series are excluded. It is available at https://physionet.org/content/challenge-2019/1.0.0/.

P12 The P12 [46] dataset comprises data from 11,988 patients after 12 inappropriate samples identified by [5] were removed from the dataset. Each patient's record in the P12 dataset includes multivariate time series data collected during their initial 48-hour stay in the ICU. The time series data consists of measurements from 36 sensors (excluding weight). Additionally, each sample is associated with a static vector containing 9 elements, including age, gender, and other relevant attributes. Furthermore, each patient in the P12 dataset is assigned a binary label indicating the length of their stay in the ICU. A negative label signifies a hospitalization period of three days or shorter, while a positive label indicates a hospitalization period exceeding three days. It is available at https://physionet.org/content/challenge-2012/1.0.0/.

MIMIC-III The MIMIC-III [45] dataset is a widely used database that comprises de-identified Electronic Health Records of patients who were admitted to the ICU at Beth Israel Deaconess

Medical Center from 2001 to 2012. It Originally encompassed around 57,000 records of ICU patients, containing diverse variables such as medications, in-hospital mortality, and vital signs. [48] established a variety of benchmark tasks using a subset of this database. In this paper, we focus on the binary in-hospital mortality prediction task to assess classification performance. Following preprocessing, our dataset consists of 16 features from the preceding 48 hours and 21,107 samples. It is available at https://physionet.org/content/mimiciii/1.4/.

Physionet Physionet[44] contains the data from the first 48 hours of ICU patients, which is a reduced version of P12 considered by prior work. Therefore, we follow the same preprocessing methods used for the P12 dataset. The processed data set includes 3997 labeled instances. We focus on predicting in-hospital mortality. It is available at https://physionet.org/content/challenge-2012/.

C Baselines

The implementation of baselines follows the corresponding papers or default implementation of their code repositories. Considering that different baselines have varying convergence speeds, we search for the number of epochs in the range of { 10 , 20 , 50 } to ensure all baselines can reach convergence. We search for the learning in the range of { 0 . 001 , 0 . 005 } due to the differences in model complexity and dataset size. Since all four datasets are highly imbalanced, we upsample the minority class in each batch to make the batch balance. Here are the detailed hyperparameter settings of these baselines:

ODE-RNN [4]: ODE-RNN uses neural ODEs to model hidden state dynamics and an RNN to update the hidden state in the presence of a new observation. The latent dimension is set as 40 , and the ODE function has 3 layers with 50 units. The source code can be found at https://github.com/YuliaRubanova/latent_ode

GRU-D [17]: GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture. The number of hidden states of GRU-D is set as 49 . We use the open source code from https://github.com/Han-JD/GRU-D

SEFT [5]: A set function approach where all the observations are modeled individually before pooling them together using an attention-based approach. We use a constant architecture for the attention network f ′ with 2 layers, 4 heads and dimensionality of the dot product space d of 128 . In addition, the attention network f ′ was always set to use mean aggregation. We use the open source code from https://github.com/BorgwardtLab/SeFT.

mTAND [6]: A deep learning framework for ISMTS data that learns an embedding of continuous time values and uses an attention mechanism to produce a fixed-length representation. We use the encoder of the overall framework for classification tasks. We set the latent dimension and the hidden size of GRU to 32 . The number of reference points and the dimension of time embedding is 128 . The source code can be found at https://github.com/reml-lab/mTAN.

IP-Net [14]: A model architecture for ISMTS data based on several semi-parametric interpolation layers organized into an interpolation network followed by a prediction network GRU. The number of reference points is set as 192 . The hidden size of GRU is 100 . We take the source code at https://github.com/mlds-lab/interp-net.

DGM 2 -O [13]: A generative model, which tracks the transition of latent clusters instead of isolated feature representations, achieves robust sparse time series modeling. We use the DGM 2 -O and set both the hidden dimension and the cluster_num as 10 . We use the source code at https://github.com/thuwuyinjun/DGM2.

MTGNN [47]: A general graph neural network framework designed for MTS data. We use 5 graph convolution and 5 temporal convolution modules with the dilation exponential factor 2 . The graph convolution and temporal convolution modules have 16 output channels. The skip connection layers all have 32 output channels. The first layer of the output module has 64 output channels, and the second layer has 1 output channel. We use the open source code from: https://github.com/nnzhan/MTGNN.

Raindrop [9]: A graph neural network that embeds ISMTS while learning the dynamics of sensors purely from observation data. The dimension of observation embedding is 4 . The dimensions of time representation p t and r v are 16 . We set the number of Raindrop layers L as 2 . The d k is set to 20 . The d a is set equal to the number of sensors. The source code can be found at https://github.com/mimsharvard/Raindrop

StraTS [18] is a self-supervised transformer for sparse IMTS. We use the implementation at https://github.com/sindhura97/STraTS and the following setting in our experiment: hidden_dim = 64 , num_layers = 2 , num_heads = 16 , dropout = 0 . 2 .

DuETT [19] is a dual event time transformer for Electronic Health Records (EHRs). We use the implementation at https://github.com/layer6ai-labs/DuETT and the default settings of the model declaration in this repository.

ViTST [20] transforms IMTS into line graph images and adapts powerful vision transformers to perform time series classification in the same way as image classification. We use the implementation at https://github.com/Leezekun/ViTST.

Warpformer [22]: A transformer-based network that captures features at different scales in IMTS using warping modules and dual attention mechanisms. We use three scales with normalized length ˜ L (0) = 0 , ˜ L (1) = 0 . 2 and ˜ L (2) = 1 . The dimension of representations D is set as 32 . The attention heads and the layers of the warpformer are set as 1 and 2 , respectively. We use the implementation at https://github.com/imJiawen/Warpformer.

D Performance Metrics

AUROC AUROC is commonly employed in binary classification tasks, where one class is designated as positive and the other as negative. It represents the area under the Receiver Operating Characteristic (ROC) curve, constructed by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR). AUROC ranges from 0 to 1, with a higher value indicating better model performance in accurately discriminating between positive and negative instances. An AUROC equal to 0.5 indicates a model's performance equivalent to random guessing, while an AUROC greater than 0.5 signifies superiority over random guessing.

AUPRC The Area Under the Precision-Recall Curve is widely used as a performance metric for imbalanced binary classification tasks. It provides a comprehensive assessment of a model's precision-recall trade-off. The Precision-Recall curve is constructed by plotting recall on the x-axis and precision on the y-axis. AUPRC ranges from 0 to 1, and a higher value indicates better model performance in achieving high precision and recall simultaneously. It has been suggested as a good criterion for unevenly distributed classification problems [54].

E More Implementation Details

Generation of Variables' Textual Descriptions We have chosen three textual sources: the full name of the variable, Wikipedia, and ChatGPT, for variable descriptions. The full name of the variable and Wikipedia sources can be directly extracted from dataset descriptions and web pages without requiring special processing, so we don't delve into them further here. Regarding the ChatGPT source, the medical properties description for each variable is generated by providing consistent question templates to ChatGPT, aiming to maintain a relatively uniform format across variables. We initiate queries to ChatGPT using the following template: What medical prior knowledge do you know about the medical variable 1 ⃝ in 2 ⃝ ? to obtain a textual description for each variable in a specific dataset. The position marked as ' 1 ⃝ ' corresponds to the variable name to be queried, and the position marked as ' 2 ⃝ ' corresponds to the dataset-specific task description. For instance, the task for the MIMIC-III dataset is ICU patients' in-hospital mortality prediction , so the query for obtaining textual information associated with the variable Heart Rate could be: What medical prior knowledge do you know about the medical variable Heart Rate in diagnosing whether ICU patients will die during hospitalization? The following is a part of ChatGPT's answer for this example:

Heart rate is a fundamental physiological variable that plays a significant role in diagnosing and assessing the condition of ICU patients and their potential risk of mortality during hospitalization. Here's what you need to know about the medical variable "heart rate" in this context: Normal Heart Rate: A normal resting heart rate for adults typically ranges between 60 to 100 beats per minute (bpm). Deviations from this range can indicate potential health issues. Tachycardia: An elevated heart rate (tachycardia), often above 100 bpm, can be a sign of various medical conditions ... .

The complete text description can be obtained through the code link.

Hyperparameters We search all hyperparameters in the grid to find the best hyperparameters for our proposed model KEDGN. Specifically, our model has a total of 5 hyperparameters: dimension of query vectors q , dimension of variables' node embeddings n , proportion of density score α , dimension of variables' hidden state h , and dimension of structured encoding representations k . For all datasets, h and k are set to be equal, and we search them over the range { 8 , 12 , 16 } . Additionally, we search the dimension of query vectors q in { 5 , 7 , 9 } , the dimension of variables' node embedding n in { 7 , 9 , 11 } and the proportion of density score α in { 1 . 0 , 2 . 0 , 3 . 0 } . The best hyperparameters for each dataset are reported in the code.

F Additional Experiment

F.1 Classification Results Using Different PLMs

In KEDGN, we utilize a PLM to extract sentence embeddings of variables' textual medical information as variable semantic representations. The choice of PLM is diverse, and in our default implementation, we use BERT. Additionally, we have experimented with other PLMs, including T5 [55], Bart [56], GPT2 [57], LED [58], and Pegasus [59]. The detailed information of these PLMs is listed in Table 5. We extract the hidden state at

the [CLS] position for BERT. For other models, we uniformly take the last hidden state (average pooling) of the models as the representations of the variables. The classification results of using different PLMs on four datasets are listed in Table 6.

The results show that BERT (Encoder-Only model) achieved high performance on most datasets. Several Encoder-Decoder-based models also demonstrated competitive results, with Bart outperforming BERT on the P12 dataset, LED slightly surpassing BERT on the Physionet dataset and T5 showing slightly lower performance than BERT on the MIMIC-III dataset. However, the Decoder-Only GPT exhibited notably lower results on the P19 dataset and MIMIC-III compared to other models. This situation may be attributed to the fact that, in our task, PLMs are used to extract sentence-level semantic representations, which are utilized to differentiate variables and measure correlations among them, resembling a text classification process among variables. On the one hand, Encoder-Only models may excel at understanding tasks such as text classification, with BERT specifically adding a special [CLS] token dedicated to extracting sentence-level overall semantic information. On the other hand, models incorporating a decoder involve tasks related to predicting the next word and may be more focused on text generation tasks. They may exhibit a slight deficiency in extracting distinctive sentence-level representations, especially for Decoder-Only models. These models sometimes require carefully designed prompts to guide them in generating high-quality outputs.

In summary, variable semantic representations extracted by different PLMs can impact downstream task performance. Choosing a PLM that is more suitable for the task and dataset can further enhance the performance of KEDGN. In the context of the application in this paper, a simple pure encoder model may be more suitable for achieving optimal results.

F.2 Computational Cost Analysis

We conduct a analysis of the time and space overhead on the Physionet dataset, with a batch size of 128, and utilizing Nvidia 1080Ti GPU infrastructure. The results are shown in Table 7. Our method achieves a balanced time and space overhead. The introduction of textual information involves generating semantic embeddings using PLMs, and the adjustment of the variable graph based on observed local density, both of which can be predetermined and integrated into the preprocessing step without increasing model training overhead. Attention-based methods (SeFT, mTAND) achieve parallel computing in the time dimension, resulting in low time overhead, but this sacrifices fine-grained feature

extraction at the variable level. Furthermore, compared to the same RNN-based method GRU-D, our method only deals with actual observation points at each timestamp, thus significantly reducing time overhead. While our method's runtime is 1.3 times that of the latest SOTA model, Warpformer, our space overhead is only 16% of its size. In situations where computational resources permit, our method can further reduce runtime by employing the space-for-time trade-off strategy (such as increasing batch size).

F.3 More Results for Leave-variables-out

F.4 More Results for Ablation Study

We find that the model performance declines after introducing text information on P12 dataset. As indicated in Table 6, when we replace PLM from BERT to Bart, there is a slight improvement of 0.5% in AUPRC on the P12 dataset, slightly outperforming the model without text. Therefore, the reason for the decline in model performance is likely attributed to the choice of default PLM (BERT), which may not be optimal for extracting textual embeddings on this dataset, rather than the text itself causing the decline. In other words, introducing text offers the potential for enhancing model performance, with the extent of improvement depending on the degree of effective utilization of text.

F.5 Effects of different activation functions in Eq.(7)

On one hand, the activation function in Eq.(7) reflects the dynamics of variables, such as time decay or exponential increase. On the other hand, this activation function serves a normalization purpose because the edge weights in the knowledge-empowered complete graph A are normalized values. If directly using the absolute value of the density to adjust the edge weights, the values might become excessively large or small, which would severely disrupt the basic graph structure learned from textual knowledge. Typical activation functions with normalization capabilities include Sigmoid and Tanh. As shown in Table 10, Tanh is chosen since it performs better.

F.6 Parameter Complexity Analysis

As shown in Table 11, the parameter count of our model is not particularly high. It is on the same order of magnitude as Warpformer and significantly lower than Raindrop by three orders of magnitude. Although we calculate an independent parameter space for each variable, the total W t ∈ R V × I × O does not equate to the parameter count. W t is derived from the multiplication of two matrices: the variable embedding matrix Q ∈ R V × q and the weight matrix W ∈ R q × I × O . The first matrix is computed from textual embeddings, and only the second matrix belongs to the model parameters. The sizes q , I , and O are hyperparameters, independent of the number of variables, and typically set to be less than 16 . This ensures that the parameter complexity of our model remains within an acceptable range.

F.7 More variables groups on the P12 dataset

(a) Invasive arterial blood pressure

(b) Renal function indicators

(c) Ion concentration indicators

(d) Non-invasive blood pressure

(e) Blood cell indicators

(f) Oxygenation status indicators.

F.8 More Visualizations of Variables Textual Representations

(a) Vital signs indicators

(b) Ion concentration indicators

(c) Blood cell indicators

(a) ChatGPT

(b) Wikipedia

(c) Random Initialized

(a) Vital signs indicators

(b) Neurological scoring indicators

(a) ChatGPT

(b) Wikipedia

(c) Random Initialized

F.9 Visualization of Variable Semantic Representations Using Different PLMs

(a) BERT(AUPRC=62.0)

(b) LED(AUPRC=60.3)

(c) GPT2(AUPRC=54.8)

(a) BERT(AUPRC=48.3)

(b) T5(AUPRC=48.0)

(c) GPT2(AUPRC=43.1)

We conduct T-SNE visualization analysis on variable representations obtained from different PLMs. Figures 12 and 13 correspond to the results of the P19 and MIMIC-III datasets, respectively. We observe that when the clusters of variable representations extracted by PLMs are consistent with the grouping of variable time series patterns and exhibit distinctiveness, the corresponding downstream classification task performance tends to be better. For example, on the P19 dataset, BERT and LED achieve higher classification performance, with their corresponding variable representations having good distinctiveness: the three groups of variables represented by blue, yellow, and green colors show high cohesion and low coupling, while the T5 model, which exhibits suboptimal classification performance, has a distribution where Resp, WBC, and Platelets are notably confused with the green variables. Additionally, in the MIMIC-III dataset, the variable representations obtained from BERT and T5, which achieve higher performance, with blue and green clusters having longer distances, indicating better distinctiveness. On the other hand, variable representations obtained from GPT2 show poor distinctiveness, with the variable RR being close to GCS-EO and GCS-MR.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction have clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limiations of the work in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code and data are provided by a Github repository link.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data are provided by a Github repository link.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in Section 5.1 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide the standard deviation of the mean in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on the computer resources in Appendix F.2

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We release our model by a Github URL.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to handle difference and correlation between multiple variables in the irregularly sampled medical time series data. By proposing a model named KEDGN with textual medical knowledge and dynamic variable graphs, it is able to outperforms baselines on a variety of datasets. Comprehensive experiments have also been conducted to demonstrate the learned correlations between variables, which aligns with the motivation of this paper.\n\nSTRENGTHS:\n1.\tThe motivation of the paper is solid. The authors provide enough evidence and arguments to demonstrate the dynamic correlation patterns between time series data.\n2.\tThe paper is generally well written with clear figures and equations.\n3.\tThe proposed model demonstrates strong performance in various experiment settings. Comprehensive analysis is also being conducted to prove the effectiveness of each module design.\n4.\tIt is good to see some visualizations about the variable correlations learned by the model, which aligns with the motivation of this paper.\n\nWEAKNESSES:\n1.\tThe idea of learning dynamic graphs that model variable relationships is quite common in literatures. It would be better for the authors to cite related works, compare with existing works and make arguments about novelty and effectiveness of the proposed modules. Here are some references:\n    -\tJiang, Renhe, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. \"Spatio-temporal meta-graph learning for traffic forecasting.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 7, pp. 8078-8086. 2023.\n    -\tHuang, Qihe, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. \"Crossgnn: Confronting noisy multivariate time series via cross interaction refinement.\" Advances in Neural Information Processing Systems 36 (2023): 46885-46902.\n    -\tWang, Dingsu, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. \"Networked time series imputation via position-aware graph enhanced variational autoencoders.\" In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2256-2268. 2023.\n    -\tWang, Binwu, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. \"Towards Dynamic Spatial-Temporal Graph Learning: A Decoupled Perspective.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, pp. 9089-9097. 2024.\n2.\tThere exist some missing references and baselines for irregularly sampled time series classification problems. It would be better to cite related works and compare with them in the experiment sections. Here are some examples:\n    -\tLi, Zekun, Shiyang Li, and Xifeng Yan. \"Time Series as Images: Vision Transformer for Irregularly Sampled Time Series.\" Advances in Neural Information Processing Systems 36 (2024).\n    -\tLabach, Alex, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. \"Duett: Dual event time transformer for electronic health records.\" In Machine Learning for Healthcare Conference, pp. 403-422. PMLR, 2023.\n\nQUESTIONS:\n1.\tIt seems some of the baselines such as DGM^2-O can reach good performance with relatively less time and space requirements. Can there be any potential improvements to the efficiency of the proposed KEDGN model? It seems BERT is also optimized during training, it would be interesting to see some efficiency experiments and analysis regarding this aspect.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This work considers the irregular timestamp contained in current medical data. They employ a density-aware mechanism to the time-varying correlations among variables.

STRENGTHS:
This work addresses an important topic in this field and is well-organized and written. The experiments are sufficient to prove their conclusions.

WEAKNESSES:
1. In formulation (6), the density of the observation point is calculated by the average time interval. What if the point is not observed in the next or previous time? e.g. t_v^(i+1) or t_v^(i) is None.
2. I am just a little bit confused by the node in the dynamic variable graph. As we know, the patient's EHR data will contain lots of clinical notes and some like treatment, medications, and so on. You have shown the description in your model's framework in Figure 3. So do you consider other variables in patient's EHR data? Besides, the node here denotes a text, not detailed medical terminology, right?
3. Could you please add a comparison graph between the original variable representation and the final representations?

QUESTIONS:
see weakness

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThroughout this article, the authors have described an approach (KEDGN) to tackle Irregularly Sampled Medical Time Series. In this task the data is composed of multiple variables represented with time series. The sample rate of a time series can be irregular and two time series can have their samples taken at a different timestamp.\nIn the approach described the authors propose to first find common knowledge correlation between the variables in the dataset. To do so, they compute a non-linear cosine similarity between embeddings of textual descriptions, obtained using a Pre-trained Language Model.\nThey then precise this common knowledge correlation by adding a timestamp aware correlation. Finally using this correlation information and a Graph Convolutional Recurrent Network, their approach can predict the class expected of the time series.\nThey have then analysed the results of their approach first through a benchmark with multiple state of the art approaches. They then added an ablation study and different visual analysis to give a better overview of the behavior of the approach.\n\nSTRENGTHS:\n* The proposal to use Common Knowledge through PLM to find correlation\n* Detailed Experiment with a detailed comparison with state of the art approaches\n* Detailed exploration of the behavior of the model through an ablation study and visual analysis of the Embedding of the explanations.\n\nWEAKNESSES:\n* Details of the Proposed Model could be better explained by giving the intuition behind the equations. (Especially for 4.3.2 and 4.4)\n\nQUESTIONS:\n* Did the authors try to use definitions given by health care professionals to confirm the definition given by the GPT model?\n* It is not clear how equation (7) uses equation (6). Indeed, $Z^{(i)}_{v}$ was defined with $i$ being the $i$-th observation point (I believe the $i$-th sample of this specific variable $v$).\nBut on the other hand the equation (7) uses $Z$ with $t$ being a timestamp. --\n\nIt is not clear if $Z^{(i)}_{v}$ is defined for a timestamp in which no observation was made for a variable.\n* It is not clear why the equation (9) contains a transpose.\n\nLIMITATIONS:\nIt appears the limitations have been discussed in sufficient detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of irregularly sampled medical timeseries classification. The core idea is to use PLM to obtain semantic embeddings for variables, which are used to form a variable correlation graph. Then, the variable correlation graph is dynamically adjusted with the observations, based on which a spatiotemporal graph neural network is used to learn timeseries representation for classification. Typical experiments on widely used benchmarks are conducted to evaluate model performance.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand. The authors clearly tell the story of using variable correlation graph to solve irregularly sampled medical timeseries classification problem.\n2. The investigation of different PLMs on variable correlation graph discovery is important and useful to the community, although the conclusion is that most PLMs achieve similar results, and even name can achieve comparable performance with wikipedia.\n3. The experiment section is well defined with various results illustrated.\n\nWEAKNESSES:\n1. The parameter settings for baselines are not fair. In lines 228-230, the authors simply follow parameters in reference. However, such setting is not a fair way comparison due to several reasons: Some datasets like MIMIC-III for experiments are not public, i.e., although we have the same raw data source, the experimented datasets are never exactly same, and may be very different from distribution and statistics. This can be validated from the comparison between the results of Raindrop on P19 dataset from this paper and Raindrop paper. The performance has a significant difference, which implies the parameters should searched completely. Thus, grid search for baseline parameters is necessary.\n\n2. The choice of GCRNN here is not well clarified. There are many spatiotemporal graph neural networks like STGNN /ST-GCN which can be used here. Moreover, the ablation study of graph convolution and temporal modeling for GCRNN should be conducted to evaluate the performance gains.\n\n3. Some claims or statements intend to emphasize the model performance, which however is obviously set by design itself instead of model learning. For example, in Line 337 the authors state ''no correlation is observed with DiasABPas it has not been observed yet''. Actually, this is not by learning but the initial masking operation of adjacency matrix.\n\n4. An important baseline of irregularly sampled medical timeseries modeling is missed here, i.e., StraTS [1].\n\n5. The ablation study doesn't identify very effective module of the proposed work. For example, Text, KEE and DAG's performance gains are marginal, so are these module necessary?\n[1] Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series, TKDD.\n\nTypos:\n1. The symbols or bold texts in Equation (10-12) are wrong.\n\nQUESTIONS:\n1. What's the function of activation function $\\sigma$ in equation (7). In my opinion, this activation function reflects the dynamics of variables, like time decay or exponential increase. So, how to determine the function and what are the effects of different activation functions.\n2. How is the parameter complexity of the model? Since each variable will require an independent $W_i$, the computation complexity might be very large.\n3. This paper emphasizes the importance of using text information of variables. However, the performance of different text sources seems close. Especially, only using Name can achieve similar even better performance. From ablation study in Table 3, we can observe even without textual representations, the model can achieve almost similar performance. Thus, this part looks useless, which is opposite to the commonsense. \n4. Is Figure 5(a) calculated from f(E) or E? If in the f(E), the authors should also provide the correlation graph from E, so that we can evaluate whether model learning is effective. To my experience, even bert embedding of these variables can achieve similar correlation graph. Also, why graph (a) is not symmetric? What's meaning of different correlations between (FiO2, HR) and (HR, FiO2)?\n5. What's the basic time resolution or sampling rate for each dataset?\n6. How is the static features combined with timeseries are not described in paper.\n7. How are the input length and prediction length determined are not provided. Usually, in mortality prediction, we use the first 48 hours ICU data as input to predict whether the patient will die in the hospitalization. What's setting for this paper?\n\nLIMITATIONS:\nThe authors didn't comprehensively discuss the limitations of this paper, which is not accord with the paper checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}