Knowledge-Empowered Dynamic Graph Network for Irregularly Sampled Medical Time Series
Authors: Yicheng Luo
Zhen Liu
Linghao Wang
Binquan Wu
Junhao Zheng
Qianli Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: healthcare
irregularly sampled medical time series
graph neural network
recurrent neural network
Primary area: machine_learning_for_healthcare

Abstract: Irregularly Sampled Medical Time Series (ISMTS) are commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features at the variable level in ISMTS. We propose Knowledge-Empowered Dynamic Graph Network (KEDGN), a graph neural network empowered by variables' textual medical knowledge, aiming to model variable-specific temporal dependencies and inter-variable dependencies in ISMTS. Specifically, we leverage a pre-trained language model to extract semantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from a medical perspective. Based on this, we allocate variable-specific parameter spaces to capture variable-specific temporal patterns and generate a complete variable graph to measure medical correlations among variables. Additionally, we employ a density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variable-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra-variable and inter-variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods.

Knowledge-Empowered Dynamic Graph Network for Irregularly Sampled Medical Time Series
Yicheng Luo, Zhen Liu", Linghao Wang, Junhao Zheng, Binquan Wu, Qianli Ma'
School of Computer Science and Engineering.
South China University of Technology, Guangzhou, China
(csluoyicheng200) cszhenliu, cskyun_ } @ mail.scut.edue edu.en. (linghaowang6, junhaozhenng (oreeeennnn@@ouooooooooooooo @outlook.com, qianlima scut.edu.cn
Abstract
Irregularly Sampled Medical Time Series (ISMTS) commonly found in the healthcare domain, where different variables exhibit unique temporal patterns while interrelated. However, many existing methods fail to efficiently consider the differences and correlations among medical variables together, leading to inadequate capture of fine-grained features the variable level in ISMTS. We propose Showledge-Eimpowere Dynamic Graph Nerwork (KEDGN), graph neural network empowered by variables' textual medical knowledge. aiming model variable-specific temporal dependencies and inter-variabl dependencies ISMTS. Specifically, we leverage pre-trained language model extract se- mantic representations for each variable from their textual descriptions of medical properties, forming an overall semantic view among variables from medical perspective. Based on this, we allocate variable-specific parameter spaces to cap- ture variable-specifie temporal patterns and generate a complete variable graph measure medical correlations among variables. Additionally, we employ density-aware mechanism to dynamically adjust the variable graph at different timestamps, adapting to the time-varying correlations among variables in ISMTS. The variablc-specific parameter spaces and dynamic graphs are injected into the graph convolutional recurrent network to capture intra- variable and variable dependencies in ISMTS together. Experiment results on four healthcare datasets demonstrate that KEDGN significantly outperforms existing methods. Our code is available at htttt::///// thub. com/ qianl ima Lab/KEDOn com/qianl
Introduction
In the medical environment, the widely used Flectronic Health Records (EHRs) have abundant typical Irregularly Sampled Medical Time Series (ISMTS) data [I. Fach ISMTS typically comprises multiple medical variables for a patient, each with distinct medical properties, resulting in significant differences in the sampling patterns of each variable series, Additionally, due to the dynamic changes a patient's condition, the sampling rate variables varies over different periods, resulting in uneven sampling intervals [2].
Many existing methods for ISMTS primarily focus on addressing uneven sampling intervals and have proposed approaches such as Ordinary Differential Equations (ODEs) [3] 4 and continuous- time embeddings [5] [6) etc, which have already achieved significant success. Recent advancements in regularly sampled multivariate time series analysis 7 g underscore the importance of capturing variable-specific temporal patterns. However, many existing methods ISMTS have not adequately considered this aspect, having limited ability to explicitly distinguish multiple variable series with
"Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

different time patterns and thus lacking finer- aner-grained capturing of variable-level features. Particularly in ISMTS, different variables have distinct medical properties, further intensifying the degree of differences among variables. In such cases, capturing differentiated variable patterns requires deeper exploration of inherent differences among variables.
Despite the differentiated temporal patterns occurring among variables within ISMTS, they are not entirely independent but exhibit medical correlations. Due to the dynamic changes in the patient's 's condition, this correlation varies along with the sampling density of variables at different periods, as illustrated in Figure I Some existing work has introduced graph neural networks [9] [O] to model the time-varying correlations among variables in ISMTS. However, due the lack of prior medical knowledge, these methods learn variable correlation graphs from misaligned and imbalanced observations variables of ISMTS and rely solely downstream tasks for graph optimization. Consequently, the variable graphs learned by these methods may face challenges in accurately reflecting the general medical cor- relations among variables, resulting in suboptimal performance.
NiDiesAB
DiasABP
10
20
30
40
50
60
Lim
Figure Illustration of three vari- ables in an ISMTS sample. In the first 7 seconds (Box 1), a strong correlation between HR and NI- DiasABP observed. As NIDi- as becomes more sparse, the correlation between HR and NIDi- asABP weakens between 7 and 50 seconds (Box 2), while the corre- lation between HR and DiasABP increases.
To this end, we aim to explicitly consider the differences and correlations among variables in ISMTS, empowering the model to capture fine-grained features at the variable level. However, the sampling rates, sampling times, observation spans, and ob- servation lengths of different variables vary in ISMTS, as shown in three subgraphs Figure 2 This makes it tough and complex to infer these two aspects only from the time series modality of variables. We rethink the issue based on an intuitive observation as shown in Figure 2 Variables that exhibit (dis)similar temporal patterns frequently have (dis similar medical properties in reality, which motivates us to consider the differences and correlations of variables from the perspective of domain knowledge directly. Recent work [1I] [22 has successfully represented domain knowledge through textual modal information, enhancing model performance in medical imaging, which provides us with insights. Specifically, the medical properties of each variable can be described in natural language. Leveraging the powerful semantic understanding capabilities (PLM), obtain semantic representations from the textual knowledge of each variable. This set of textual representations forms an overall view of variables from the perspective of medical knowledge, clearly showing inherent differences and correlations among variables-exactu what need.
the Pre-trained Language model
Based the above analysis, propose the Bhooledge-Empoweeee Dynamic Graph Network (KEDGN), which utilizes textual semantic representations of variables obtained through PLM as guidance. On this basis, 1) allocate unique parameter space for each variable capture their specific temporal pattern, 2) generate a complete variable graph and introduce a density-aware mechanism explicitly model tire-iarying correlations among variables in ISMTS. Finally, these
M3
No
HC03
(a) Invasive arterial blood pressure
Renal function indicators
(c) Ion concentration indicators
The time patterns (sampling rates, sampling times, observation spans, observation lengths, trends, etc.) of variables among different subgraphs exhibit significant differences, as they have distinct medical properties. Meanwhile, variables within the same subgraph share similar time patterns, and their medical properties are closely related. (More variable groups can be found in Figure 7).
Figure 2:

---

two modules are integrated with a graph convolutional recurrent network to capture both temporal and inter-variable dependencies in ISMTS. Our contributions can be summarized in three aspects:
We leverage variable-specific textual medical knowledge empower the model to capture variable-specific temporal patterns in ISMTS distinctively.
We introduce density-award mechanism based the knowledge-cmpoweed variable graph to model the time-varying vater-variable dependencies in ISMTS.
Empirical results on four real-world medical datasets demonstrate that KEDGN outperforms state-of-the-art methods. Visualization analysis further illustrates the strong interpretability of our approach.
2 Related Work
Irregularly Sampled Multivariate Time Series Modeling Existing methods can be roughly categorized into interpolation-based and raw data-based approaches. The former, employing methods such as kernel-based approaches [3] 14), Gaussian process [5] or hourly aggregation [16), aims to obtain set of regularly spaced observations. However, interpolation may result in the loss of useful information about the original sequences, such as missing patterns. The latter, raw data- based methods, directly learn from irregular time series. To adapt to uneven sampling intervals, 17 improves recurrent neural networks, 3 introduce neural ordinary differential equations and
6 5 18 [9] adopt time embeddings. (0 converts ISMTS into line graph images and utilizes pretrained vision transformers for extracting features. These methods primarily focus on overall temporal dependencies, needing more consideration for greigga variable-level patterns and correlations. Despite recent research introducing attention (21 [2) or graph neural networks [9] 23 account for variable correlations, these methods have limited performance due to the lack of prior knowledge and the use of shared parameter spaces among all variables.
Graph Neural Networks for Multivariate Time Series In recent years, a series of studies have integrated GNN with various time series modeling frameworks to effectively capture both inter- variable and inter-temporal dependencies in MTS [24]. These approaches have been widely applied
in diverse domains, including transportation 25 healthcare (26), economics
(27), demonstrating
promising results in mainstream tasks such as prediction (28), classification (29] and imputation (30]
Although recent work [3T 33 32 has proposed the idea of modeling variable relationships through
learning dynamic graphs, most of these methods primarily designed for regularly sampled MTS
with synchronous observations, and further improvements are needed to adapt them for irregularly
sampled time series.
Medical Knowledge Enhanced Models Several studies have utilized the rich domain knowledge in the medical field to enhance models. (2] apply knowledge for computing additional features, utilize knowledge guide the final training loss, demonstrating the effectiveness of medical prior knowledge. However, existing methods commonly focus on visual language pretraining in medical scenes or medical report generation [II. How to effectively integrate domain knowledge to guide medical time series modeling remains a challenge.
while [36] 37
3 Problem Definition
Given a dataset D {(s), consists of an irregular multivariate time series and a label yr. For the dataset with a total variable count of V and a maximum sample observation length of s; can be denoted as a tuple: s; = (f,, I2. m,), where & E R represents the observation timestamps, 2; E R xT represents the multivariate time series observations, and any unobserved values or the missing parts of the time series shorter than the maximum sample observation length are filled with 0. The binary indicator m; has the same size as Vú, indicating which elements in the x; are actually observed. We use to represent observed values and 0 to represent missing values. In this paper, we focus on patient mortality and morbidity prediction, i.e., classification task, aiming to correctly predict class label ý given a sample st.
N} containing N patient samples, the i-th sample
3

---

Input Representations
Frozen
x Matris multiplication
f/x
Projection heads
Observed variables
Unobserved variables
Variable Semantic Representations Extraction
(2) Veriafie-specific Parameter Learning
on indicator of how well a...
Langusge Model
Bady temperature is typically measured using...
Pretrancd
Queey Vectors
Weights
(4) Graph Convolutional Recurrent Network
Go
MAP neflects the everage pressure a patient'
Cusine Similarity
Deasity awaze Dynamic Adjustment
Test description of variables! medical properties
Node
Embeeddings
Variable Correiation Graph
(3) Dynamic Variable Graph Generation
Figure 3: The model framework of KEDGN. We (1) utilize a PL.M to extract semantic representations
for each variable from textual medical properties (Section 4.1p Based this, (2) allocate variable-specific parameter space to capture variable-specifie temporal patterns (Section (42), (3) generate dynamic variable graphs by combining Knowledge-empoweet
graph with a density-aware
mechanism to model time-varying correlations among variables (Section 4.3). (4) The above two
modules injected into graph convolutional recurrent network model intra-variable and inter- variable dependencies in ISMTS simultaneously (Section 4.4)
4
The Proposed Model
4.1 Variable Semantic Representations Extraction
First, we introduce how extract semantic representation for each variable from medical knowledge. Let V = {V1, ....... } be the set of variables, and descriptive sentence of medical properties associated with the jth variable can be denoted as:
P3 U1,1, W5,2, -. Wy.,,y 1,2, V}.
where Iy is the length of the jth variable's sentence and w/, denotes the 7t word of the jth sentence. We leverage PLM to represent each text description P; as d-dimensional embedding. Considering the diverse types of PLMs with varying methods of utilization, we use the widely adopted encoder- based model BERT 38
as an illustration:
ej BERT (EERT([SS), Wy,1. W5.2..... WJl,, [SEP]) E (R)++2)xd
(2)
where [CLS] and [SEP] are special tokens indicating a sequence's beginning and end, respectively. BERT generates embedding for each token of the input sequence. Since the embedding at the [CLS) position captures the semantic information of the entire input sequence, we take the hidden state at the [CLS] position the overall semantic representation of each variable: E, ((CLS)) This yields semantic representation matrix E [E, Es.... Ev] RVxd and forms an overall view of variables from the perspective of medical knowledge.
4.2 Variable-speciffi Parameter Learning
Since temporal patterns of ISMTS vary from variable to variable, simply using shared parameter space for all variables is insufficient capture differentiated temporal dependencies. In this section, we adjust parameter space for different variables to adapt to differentiated temporal patterns based on the extracted variables' semantic representations. For any parameter matrix w R with input dimension I and output dimension O, the total parameter space needed for V variables is V Inspired by (39], we decompose θ into two matrices: variable representation matrix Q E Vx4 and weight pool matrix W E R where y is hyperparameter for the intermediate dimension. Here, Q consists of q-dimensional query vectors for V variables used to distinguish differences among variables and obtain variable-specific parameters from the weight pool.

---

We use projection f(-) R + Rº to variable semantic representations E for obtaining query vectors rather than directly using E. On the one hand, the dimension of the query vector q directly determines the size of the weight pool, but the output dimension of PLM is often large (e.g.. 768 in BERT), leading to sharp increase model complexity. On the other hand, there a modality gap between textual embeddings and the temporal parameter space. The projection f()) can achieve both feature reduction and modality transformation. In our implementation, we use a nonlinear projection with one additional hidden layer (and ReLU activation). Thus, the parameter space specific to variable i can be obtained using the following formula:
θ ( R
The approach we employ to generate variable-speciff parameter space is general and not restricted to specific model backbone because any model architecture is composed of multiple parameter matrices W.
4.3 Dynamic Variable Graph Generation
In this section, we introduce how to generate dynamic correlation graphs of variables for explicitly modeling the time-varying correlations among variables in ISMTS.
4.3.1 Complete Variable Correlation Graph Learning
The misaligned and imbalanced observations of the variables ISMTS make difficult to learn the variable correlations from the time series. Therefore, we extract static complete variable correlation graph based on the textual semantic representations of the variables directly from the perspective of the actual medical properties of variables. We apply another non-linear projection g(-) R - IR" to the textual representations of variables E to obtain nz-dimensional node embeddings for each variable. Subsequently, calculate the pairwise cosine similarity among the node embeddings of variables, resulting in V x V matrix of variable similarity. Finally, we use the softmax function to normalize the edge weights corresponding to each node, producing a normalized graph of variable correlations. The correlation weight between the ith and j variables can be calculated as:
g(Ea) g(E,) (g{())] I1,
Ag Softmax
where represents vector dot product and || represents the vector magnitude. The introduction of g(-) in this context not only performs feature reduction but also avoids using a completely fixed prior graph. It preserves the model's ability to adaptively optimize the graph structure based on different data distributions and downstream tasks. Thus, we obtain a snowledge-empowerre complete graph with V nodes to measure the static correlation among variables in general medical cases, and its adjacency matrix is denoted A.
4.3.2 Dynamic Density-aware Adjustment Mechanism
Due to the varying subsets of variables observed at each timestamp in ISMTS, we use different subgraphs of A to describe the variable correlations at different timestamps. Specifically, we use mask matrix (t) RVV to indicate the subgraph topology at timestamp t:
(8)
[1.
if both variables i and are observed at
(5)
0, otherwise
Therefore, we can calculate the variable correlation subgraph A (t) at timestamp through (4) A @ (( where @ represents Hadamard product. Additionally, we introduce density-aware mechanism dynamically adjust edge weights of subgraphs in different timestamps to fit in the time-varying correlations among variables mentioned in Figure [ Specifically, we estimate the sampling density of any observation point by considering the average time interval between each observation point and its preceding and succeeding observations. If there is precedng/suucceedng observation, we take the time interval of the pocceeding/preceding observation as the density. If neither a preceding nor succeeding observation exists, it indicates that this observation is the only
5

---

one for the variable, and we take half of the maximum observation time span as the density. The formula for calculating the sampling density of the i-th observation of variable at timestamp is:
((i, 14-1,)) + (f2+1,0 f,,,))/ if both 11+1,1 and 1,-1,0 exist
7 7,,00 =
1i, - 1,-1,
if 1,0 does not exist
1++1,v - ti
if 1: does not exist
2 if neither 4441,v nor 1,1, exists.
Then calculate the density scores for various variables timestamp through:
D ao az E RY,
(7)
where is an activation function and is hyperparameter that controls the proportion. At timestamp t, the edge weight between the jth and jth variables is adjusted as:
C (0)
(t) ! (1-
W. - D.
(8)
where W RVXV is learnable parameter matrix. Thereby, we achieve the dynamic adjustment of the variable graph weights in response to changes in variable sampling density,
4.4 Variable-spectie Dynamic Graph Convolutional Recurrent Network
Under the empowerment of variable textual representation, we have obtained variable-specifer parameters O E RV and dynamic variable graph E TXVVVVV In this section, we integrate these two modules into the graph convolutional neural network to handle ISMTS. GCRNN [40] is backbone network that introduces graph convolutional operations on top of an RNN variant, Gated Recurrent Unit [41]. This structure is simple, effective, and easy-to-adapt for ISMTS, as it enables variable-level parallel computation of asynchronous observations without explicit interpolation. Specifically, we allocate unique hidden state for each variable, updating the state only at the observed timestamps to avoid imputation and preserve the individual sampling patterns of each variable. The graph convolution operation over graph signal S E R Vx! containing V nodes at timestamp t is defined as follows:
(9) where Jv E RVVV is identity matrix, x represents batch matrix multiplication. Here, we adopt 1st -oddrr Chebyshev polynomial expansion approximation [2] for graph convolution. The updated formulas for variable states at timestamp / are:
② +oo S = (Iy + G 0
(t) = o(,, *oo [X (1(111111- br),
(10)
(t) = = t |
H (t- bu);
(11)
c((() = tanh(Oc * (x(0 @ H(-1)) + bc).
(12)
H ((1-1) if variable is unobserved at time
(13)
H (1-1 + us
( o c!!), otherwise
where || denotes the concatenate operation, (t E Rv is the variable states at the previous timestamp and ( Vv denotes the input representation at current timestamp. We follow the
structured input encoding method of [22] using multiple fully-connected mappings to encapsulate each observed value and its corresponding timestamp into a k-dimensional input representation (All 0 vectors for unobserved values) indicate flexible observation time and adapt to the uneven intervals within variables. y((t, (t) Rvx are reset gate and updated gate, respectively. θ,,, O., Oc R RVV VV((++))xX are variable-specifie parameters obtained by respectively multiplying the query vectors matrix Q E RVxx with three weight matrices W,, W., We E (k+)((+)) (i
We calculate the sum of channels for each variable's hidden state H, at the last observed timestamp to get V-dimensional vector C. Additionally, we follow the approach used in 99 to incorporate the statie features. Specifically, static features of each sample are mapped into static vector through a linear layer. Finally, C. and S are concatenated to predict the final classification probabilities: ÿ = Softmax (W*[C] + b.). The training objective is minimizing the cross- cross-entropy loss between ÿ and y. The pseudo-code for KEDGN is presented in Appendix A (Algorithm I).

---

Experiment
5.1 Experimental Setting
Datasets and Baselines We conduct experiments on four widely used irregular medical time
series datasets, namely P19 [43, Physionet 444, MIMIC-III 45 and P12 [46] where Physionet is
reduced version of P12 considered by prior work [6] We compare our method with the state-of-the-art
methods for modeling irregular time series, including GRU-D [17] ODE-RNN [4], IP-Net [14),
SeFT [5]. mTANN [6], Raindrop [9]. StraTS [88. DuETT [1) VITST (20] and Warpformer [22] In addition, also compare our method with two approaches initially designed for forecasting tasks,
namely DGM -0 [3] and MTGNN [77]. The implementation and hyperparameter settings of these
baselines are kept consistent with those used in found in Appendix C
9 More details of datasets and baselines can be
Evaluation Setup For the data pre-processing of MIMIC-III we follow the method described in [88] and divide the dataset into three parts for training, validation, and testing with the ratio of 70%,15%,,1o 15%. For the remaining three datasets, we adhered [9] approaches, and the ratio of training, validation, and testing set is 8:1:1. We measure the classification performance with the Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision Recall Curve (AUPRC) since all the four datasets are binary classification datasets with highly imbalanced class distribution. AUPRC has better sensitivity to sample imbalance [49]; thus, the optimal model parameters that achieve the best AUPRC the validation set used for the test set. More details of metrics can be found in Appendix D
Implementation Details We adopt the Adam [50] optimizer, and the number of training epochs set as 10. Due to differences in dataset sizes, the learning rate is set as 0.001 for Physionet and P12 and 0.005 for MIMIC-III and P19. The textual sources for variable descriptions are flexible; we chose three sources, including the variable's full name, Wikipedia source, and ChatGPT source (Default), corresponding model names KEDGN-Name, KEDGN- Wiki, and KEDGN-ChatGPP respectively. All experiments are conducted with five random seeds, and the average and standard deviation reported. More implementation details and hyperparameter settings can be found in Appendix
5.2
Main Results
Table 1: Method benchmarking on irregularly sampled medical time series classification. The best results highlighted in bold, and the second-best results are underlined. The results in the table are presented in the form of (Mean + Std %).
P19
Physionet
MIMIC-III
P12
Methods
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
GRU-D
88.7
2.3
79.1
42.7
82.2
43.3 2.1
79.6
41.7
ODE-RNN
87.1 + 1.0
3.2
75.5 2.8
33.7 4.11
8.6
42.3
78.8 1.6
37.4
NetNet
ScFT
84.0 0.3
0.5
75.5
2.94
6..9
2..2
78.1 0.5
3..9
MTGNN
$2.1
m TAND
82.9 + 0.9
12.2
86.9
1..
83.8
0.5
85.3 0.3
49.3 1.0
DGM2.0
9..6
60.0
85.8
50.4
81.0
37.6=111
85.8 1 0.1
0.7
Raindrop
0.6
6.12
82.7
1.38
2..
$2.1 2.1
StaTS
91.2
58.41
81.9
47.3
844404
4..4
16.7
DuETT
VITST
9..7
0.7
81.3 1.9
3..4
818
3..6213
8..
50.8
Warpformer
KEDGN-Name
92.3
62.5 0.7
87.9
56.0.13.2
84.8
48.4 1.5
87.1 07.1&.0.8
$4.1
KEDGN-Wiki
2.5
87.0
0.51
KEDGN-ChaGGP
92.2+
62.0 1.3
0..9
17.1
85.1 05.1
48.3 1.6
87.8 + 0.5
54.5 1.5
Classic time series classification The evaluation results are summarized in Table which we use BERT to extract variables' semantic representations. More experimental results and analyses using other PL can found in Appendix [..]] Overall, our KEDGN achieves the best performance on all four datasets and outperforms the strongest bascline by an average of 0.9% on AUROC and

---

1.6% on AUPRC. In addition, the results of models using different text sources are similar, which demonstrates that our method is not limited specific text source and exhibits generalizability. We also provide an analysis of computational costs in Appendix F.2
Leave-variables-out To demonstrate the robustness of our method, we test whether KEDGN can achieve good performance when a subset of variables is completely missing, We uniformly discard 10%, 20%, 30%, 40% and 50% of the variables, hiding all their observations both validation and test sets. Table 2 reports the results on the MIMIC-III dataset, while the results for the remaining datasets are presented in Appendix F3 (Table g). Our method achieves the highest performance on 35 out of 40 metrics across 4 datasets as the missing rate increases from 10% to 50%. This may be attributed to KEDGN only handling actually observed points, avoiding the accumulation of imputation errors, particularly in cases of higher missing ratios, thus exhibiting a degree of robustness.
Table 2: Classification performance samples with fixed set of left-out variables the MIMIC-III dataset. The best results are highlighted in bold and the second best results are underlined.
Discard
Methods
10%
20%
30%
40%
50%
AUROC
AUPPCC
AUROC
AUPRC
AUROC
AUPRC
AUPRCC
AUROC
AUPRO
GRU-D
01
0.8
80.3 0.9
4..7
79.2 + 1.8
41.0 1.4
2.1
40.4+1 1.6
77.9 2.2
39.9
iTTAND
01
80.4 1.1
1.2
4..0
19.3
3.3
DGM
0.
0.9
0.8
1.19
77.6 1.3
33.4 1.2
1.3
13.1
1.58
3266 3.
MTGNN
78.8 1.1
1.4
78.0 1.6
77.1 77.112
33.5 1.5
2..3
3288
3.2
Raindrup
78.2
2.9
77.5
03
76.4
1.
76.0220
32.5
75.7
2.0
3..
DuETT
0.
0.9
77.2+ 1.0
31.7
76.6
33.3+
76.4+1.2
13.00100
76.1
1.3
326 1.3
Warpfomer
4.2
1.2
41.3
KEDGN
83.0 0.7
44.8 x 2.0
8..3
4..4
81.3
43.6
8..6
43.0
80.0
2.5
5.3
Ablation Study
In this section, we investigate the performance benefits gen- erated by each key component of the proposed method on all four datasets. We compare the full versioned method with its six variants: (I) w/o VSW: We apply shared RNN pa- rameter weights for all variables; (2) w/o Text: We replace the variable-specific textual semantic representations E; with randomly initialized learnable embeddings; (3) w/o Graph: We set the graph G to be fully zero matrix, disregarding dependencies among variables; (4) w/o KEE: We replace www.ledg-eepoweree node embeddings of variables g( E.) with randomly initialized learnable embeddings, (5) w/o DAG: We remove the density-aware adjustment for edge weights of
Table 3: The ablation study of our proposed method KEDGN P19, The results in the table are presented in the form of (Mean + Std %).
Metrics
AUROC
AUPRC
w/o VSW
$6.9 0.3
w/o Text
9.6
3.0
wfo Graph
1.6 0.8
w/o KEE
60.4 9.4
91.5
w/o DAG
60.0 1.3
the graph, using static adjacency matrix A (4) during different periods. (6) w/o TE: We remove the timestamp embedding part of the structured input encoding ( The results on the P19 dataset are presented in Table 3. while the results for the remaining datasets are presented in Appendix (F.4]) [..4] (TTable 9. The results show that all model compo-
w/o TE
9.5
2.6
Full
92.2 0.5
62.0 + 1.3
nents are necessary and variable-speciife parameter space makes the most significant contribution to the performance of KEDGN.
5.4 Visualization Analysis
5.4.1 Visualization of Variables Textual Representations
In this section, we explore why textual information is effective for time series modeling through visualization analysis. We first group variables with similar time patterns on the P12 dataset, as illustrated in Figure (More variables groups can be found in Appendix F.7 (Figure Subsequently. we use T-SNE [11 to visualize the distribution of variable semantic representations. Figures 4a and 4b respecivell display the distributions of ChatGPT and Wikipedia sources, while Figure 4c shows the final distribution learned by replacing the text representation with randomly initialized learnable embeddings. Variables with similar time patterns are labeled with the same color. It can be observed

---

200
WBC
Plotelets
200 NIDESABP
NISySABP
Plateleto
WBC
PaCO2
NISySABI
SYABB BUN
Platelets
100
HC03
NE
Mg
PaO2
Creatinine
SaOZ
100-
DiasABP
SysABP
k
SaO2
DiasABP HCOP PaOz
Nu
NIDESSABP Ma
WBC
BUN
200
MAP
MAP
-100
NIDIasABP
PaCO2
$a02
BUN,
Creatinine
-200
NISyAAB
NIMAP
SYLABP
MAP
DixxABP
-100
PaO2 PaC02
Creatining HCO3
-600
NIMAF
-200
200
4000
-200
200
400
-200
200
(a) ChatGPT
(b) Wikipedia
(c) Random Initialized
Figure 4: T-SNE visualization partial variable representations on the P12 dataset.
that the textual representation space of variables exhibits distinct clustering. Although there may be occasional outliers when using different text sources, such BUN Figure 4a and HCO3 in Figure 4b. the overall clusters are generally consistent with the variable groups we divide based on time series patterns. However, the learnable variable embeddings Figure 4c| optimized by classification loss, tend to be distributed uniformly, which is difficult to effectively reflect the intrinsic differences among variables. More visualizations of other datasets and other PLMs can be found in Appendix infer that text descriptions and time series are both external manifestations of the inherent sense of variables; they just belong to different data modalities. These two forms of data for the same variable should exhibit relative consistency. Therefore, the relative distribution among variables extracted from text and time series should ideally be similar. Leveraging PLM allows for the straightforward and efficient extraction of this universal view from textual descriptions, which equally applicable to describing the relative distribution of temporal patterns among variables. Thus, the effectiveness of PLM and the cross-modal relative consistency are the keys to guiding time series modeling based on textual information.
F.8 and F.9 respectively. Based on this phenomenon,
5.4.2 Visualization of Variable Correlation Graph
We visualize the learned partial variablariable correlation graphs on the MIMIC-III dataset in the form of heatmaps. In Figure 5a we depict the graph learned based on node embeddings mapped from textual representations, while in Figure [5b] the graph is learned based on ran- domly initialized Icarnable node embeddings. We observe that the top-left and bottom-right corners of the heatmap in Figure [a exhibit darker colors, indicating strong correlations con- sistent with medical domain knowledge. Specif- ically, the variables in the top-left corner, Heart Rate (HR) and Respiration Rate (RR), are com- monly monitored together in clinical settings for assessing vital signs and respiratory system function [52]. The variables in the bottom-right corner, GCS-MR, GCS-T, and GCS-VR, collectively constitute the Glasgow Coma Scale (GCS), which is used to assess a patient's 's neurological status and level of consciousness [53]. These correlations are evident the graph without textual representations. This once again validates our perspective that relying solely on downstream task optimization for adaptive learning in graphs ISMTS is insufficient to reflect the actual medical correlations among variables and lacks interpretability. In contrast, textual representations can guide the model to accurately extract variable correlations aligned with domain knowledge to provide high interpretability.
6110
MAP
1R
0
0,393
DRP
SRP
8.5
Sy01
Sp02
Ho1
GCS
OCS-
GCS-1
GCS-VR VR
OCS-VX
MK
GCS-1
RR
orsam
crss
GCS VR
Figure 5: Visualization of the learned correlation graph of variables on the MIMIC-III dataset.
5.4.3 Visualization of Dynamic Density-aware Graph
In Figure we visualize the dynamic density- graph for the sample in Figure T We present the time series of three variables and the corresponding correlation heatmaps learned by our model at

---

timestamps 4, 15 and 56. From the time se- ries, we observe that around & 4, HR shows strong correlation with NIDiasABP, while the correlation with DiasABP is masked as 0 since DiasABP has not been observed yet. Around t = 15, the correlation between HR and NIDi- asABP decreases, while a relatively strong cor- relation with ABP emerges. By t = 56, HR exhibits strong correlation with both variables. This process is clearly reflected in the heatmaps: the color between HR and NIDiasABP transi- tions from dark to light from and darkens again from f 15 to 1 56. The color between HR and DBP remains dark at 15 and = 56. This demonstrates that our dynamic density-aware mechanism exactly
MiDesAOP
30
time
1-15
5:55
NIDIaARB
0.86
DisAHR
0084
AM
Figure 6: Visualization of the dynamic graph of three variables over time for the sample in Figure[] [ To enhance the contrast ratio, we set the diagonal elements to O.
to t 15
reflects the virying correlations among variables in ISMTS.
6 Limitations
Although our proposed method effectively guides ISMTS modeling through the domain knowledge from text modality, it has some limitations. The backbone of our model is a recurrent-based architecture, which inherently has a sequential computation characteristic that can be a bottleneck in terms of runtime. Additionally, our method is specifically tailored for medical applications, and its performance may be limited in other irregular multivariate time series applications, such as human activity recognition, where variables lack domain knowledge and thus cannot generate high-quality text descriptions.
7 Conclusion
In this paper, we propose KEDGN for modeling ISMTS. The proposed method leverages a PLM to flexibly extract semantic representation for each variable from the textual medical knowledge. Based on these representations, we allocate the variable-specific parameter space capture variable-specific temporal patterns and extract a complete variable graph as measure of the variables' static medical correlations. Considering the time-varying variables correlations ISMTS, we introduce a density- aware mechanism to dynamically adjust the subgraph across different periods. Our experimental results demonstrate that KEDGN outperforms existing methods in ISMTS classification tasks and provides high interpretabilityy Our future work will focus on investigating the applicability of KEDGN range of related tasks, such as interpolation, extrapolation, and regression.
Acknowledgements
The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 62272173), the Natural Science Foundation of Guangdong Province (Grant the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106 and the National Key R&D Program of China (Grant No. 2023YFA101160) 1601). Yicheng Luo and Zhen Liu equally contributed to this work.
Nos.
1022A15150100
References
[1] Chenxi Sun, Shenda Hong, Moxian Song, and Hongyan Li. A review of deep learning methods for irregularly sampled medical time series data. arXiv preprint arXiv:2010.122 /2493, 2020.
[2] Satya Narayan Shukla and Benjamin M Marlin. A survey principles, models and methods for learning from irregularly sampled time series. arXiv preprint arXiv:2012.00168, 2020.
[3] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
10

---

[4] Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampe time series. In Advances in Neural Information Processing Systems, volume 32, 2019.
[5] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning, pages 4353-4363. PMLR, 2020.
[6] Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2021.
[7] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting, arXiv preprint 100.:22310.06625, 2023.
[8] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong. and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Learning Representations, 2023.
(9] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guider network for irregularly sampled multivariate time series. In International Conference on Learning Representations, ICLR, 2022.
[10] Zhen Wang, Ting Jiang, Zenghui Xu, Jianliang Gao, and Ji Zhang. Irregularly sampled multivariate time series classification: A graph learning approach. IEEE Intelligent Systems, 2023.
[11] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21372-21383, 2023.
[12] Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanee visual-language pre-training chest radiology images. Nature Communications, 14(1):4542, 2023.
[13] Yinjun Wu, Jingchao Ni, Wei Cheng. Bo Zong. Dongjin Song. Zhengzhang Chen, Yanchi Liu, Xuchao Zhang. Haifeng Chen, and Susan Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 651 1599 2021.
[14] Satya Narayan Shukla and Benjamin Marlin. ceapollation-rrediiiti networks for irregularly sampled time series. In International Conference on Learning Representationss 2019.
[15] Qingxiong Tan, Mang Ye, Grace Lai-Hung Wong, and Pong Chi Yuen. Cooperative joint attentive network for patient outcome prediction on irregular multi-rate multivariate health data. In IJCAI, pages 1592, 2021.
[16] Liantao Ma, Junyi Gao, Yasha Wang. Chaohe Zhang, Jiangtao Wang. Wenjie Ruan, Wen Tang, Xin Gao, and Xinyu Ma. Adacare: Explainable clinical health status representation learning via scale-adaptive feature extraction and recalibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 825-832, 2020.
(17) Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
(18) Sindhu Tipirenni and Chandan K Reddy. Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1-17, 2022.
[19] Alex Labach, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. Duett: Dual event time transformer for electronic health records, 2023.
11

---

[20] Zekun Li. Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. In Thirty-see seventh Conference on Neural Information Processing Systems, 2023.
[21] Sheo Yon Jhin, Minju Jo. Taeyong Kong, Jinsung Jeon, and Noseong Park. Ace-node: Attentive co-evolving neural ordinary differential equations. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 736-745, 2021.
[22] Jiawen Zhang, Shun Zheng, Wei Cao, Jiang Bian, and Jia Li. Warpformer: A multi-scale modeling approach for irregular clinical time series. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD 23, page 3273- 3285, New York, NY, USA, 2023, Association for Computing Machinery.
[23] Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 364-373, 2021.
[24] Ming Jin, Huan Yee Koh, Qingsong Wen, Daniele Zambon, Cesare Alippi, Geoffrey I Webb, Irwin King, and Shirui Pan. A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection. arXiv preprint arXiv:2307.03759 2023.
[25] Saeed Rahmani, Asiye Baghbani, Nizar Bouguila, and Zachary Patterson. Graph neural networks for intelligent transportation systems: A survey. IEEE Transactions Intelligent Trunsportation Systems, 2023.
[26] Lijing Wang, Aniruddha Adiga, Jiangzhuo Chen, Adam Sadilek, Srinivasan Venkatramanan, and Madhav Marathe. Causalgnn: Causal-based graph neural networks for spatio-temporal epidemic forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 12191-12199, 2022.
[27] Jianian Wang, Sheng Zhang, Yanghua Xiao, and Rui Song. review on graph neural network 2021.
methods in financial applications. arXiv preprint
[28] Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang, Junbo Zhang, and Yu Zheng. Spatio-temporal graph neural networks for predictive learning in urban computing:
A survey. IEEE Transactions Knowledge and Data Engineering, 2023.
[29] Huaiyuan Liu, Xianzhang Liu, Donghua Yang. Zhiyu Liang, Hongzhi Wang, Yong Cui, and Jun Gu. Todynet: Temporal dynamic graph neural network for multivariate time series classification. arXiv preprint arXiv:2300.00000 304.00078. 2023.
[30] Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g s_ap_s: Multivariate time series imputation by graph neural networks. In International Conference on Learning Representations, 2022.
[31] Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. Spatio-temporal meta-graph leam- ing for traffic forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 8078-8086, 2023.
[32] Qihe Huang, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. Crossgnn: Confronting noisy multivariate time series via cross interaction refine- ment. Advances in Neural Information Processing Systems, 66.46885-46900 2023.
[33] Dingsu Wang, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. Networked time series imputation via position-aware graph enhanced vari- ational autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2266-2268. 2023.
[34] Binwu Wang, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. Towards dynamic spatial-temporal graph learning: A decoupled perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 9089-9097, 2024.
12

---

[35] Zhihong Chen, Yan Song, Tsung-Hou Hui Chang, and Xiang Wan. Generating radiology reports via memory driven transfomnr. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1439-1449, Online, November 2020. Association for Computational Linguistics.
[36] Xiaomeng Li, Xiaowei Hu, Lequan Yu, Lei Zhu, Chi-Wing Fu, and Pheng-Ann Heng. Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading. IEEE transactions on medical imaging, 39(5):1483-1499 2019.
[37] Qing Liao, Ye Ding, Zoe L Jiang, Xuan Wang, Chunkai Zhang, and Qian Zhang. Multi-task deep convolutional neural network for cancer diagnosis. Neurocomputing, 348:66-73, 2019.
[38] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
[39] Lei Bai, Lina Yao, Can Li, Xianzhi Wang. and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural information processing systems, 13:17804-77815 2020.
[40] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference Learning Representations (ICLR 18), 2018.
[41] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv:1412.338 2014.
[42] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations, 2017.
[43] Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Bran- don Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from clinical data: the physionet/compuuing cardiology challenge 2019. In 2019 Computing in Cardiology (CinC), pages Page-1. IEEE, 2019.
[44] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in- hospital mortality of icu patients: The physionet/comppuing in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245-248. IEEE, 2012.
[45] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody. Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii. a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[46] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus. George B Moody, Chung- Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101((3):e215-e22) 2000.
[47] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 753-763, 2020.
[48] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):66, 2019.
[49] Jesse Davis and Mark Goadrich. The relationship between mecision-reeal and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233-240, 2006.
[50] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv.141..6980, 2014.
13

---

[51] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[52] Craig Lockwood, Tiffany Conroy-Hiller, and Tamara Page. Vital signs. JBI Evidence Synthesis, 2(6):1-38, 2004.
[53] George L Sterbaah. The glasgow coma scale. The Journal emergency medicine, 19(1):67-71, 2000.
[54] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. volume 06, 06 2006.
[55] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(14))1167, 2020.
[56] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-tor -e-euuencc pre- training for natural language generation, translation, and comprehension. CoRR, abs/1910.13440 2019.
[57] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[58] Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv:2004.05150. 2020.
[59] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
14

---

A Algorithm of KEDGN
Algorithm The pseudo-code of KEDGN
Input: An ISMTS sample observation values matrix: x E RV binary mask matrix m(:7)
R V observation timestamps {(::T) R², the sentences of V variables' medical properties
P = P .v)
Output: Predicted classification label yj
extract semantic representations E R Vxd for variables from through PLM using Eq.
2: obtain variable-speciff parameter space 2 (including reset gate θ,, update gate O₂ and candi- date gate C E I ( using (
3: obtain complete variable correlation graph A using Eq.4
4: initialize the hidden state of V variables ( Oy xh
5: for = 1, 1,2,... do
6: obtain dynamic variable subgraph G E RVV according to m and Eq.
7: encode observation values (:()) and timestamp z(0 into X RV by multiple fully-
connected mappings
8: update (+) GCRNN(O, G ((),, (i)))) using Eq. 10
and
9: end for
10: calculate the sum channels for ( to get C ER
11: calculate أ = SW + bb)
12: return y
B Datasets
We use four irregularly sampled medical time series datasets to evaluate the classification performance of our model and baseline models. The dataset statistics are summarized in Table
Table 4: Dataset statistics.
Datasets
P19
P12
MIMIC-III
Physionet
Samples
38803
11988
21107
3997
# Variables
34
36
16
36
# Max Observation Length
60
215
292
215
Missing Ratio
94.9%
88.4%
65.5%
84.9%
P19 The PhysioNet Sepsis Early Prediction Challenge 2019 43 dataset consists medical records from 38,803 patients. Each patient's record includes 34 variables of up to 60 hours and static vector indicating attributes such as age, gender, the time interval between hospital admission and ICU admission, type of ICU, and length of stay in the ICU measured in days. Additionally, each patient is assigned a binary label indicating whether sepsis occurs within the subsequent 6 hours. We follow the procedures of [9] ensure certain samples with excessively short long time series excluded. It is available at httpss//pbhyiinee..oo//ommmmmmm/ http://hhaleleeeeie//..... 2019/1.0.0/
P12 The P12 |6 dataset comprises data from 11,988 patients after 12 inappropriate samples identified by [ were removed from the dataset. Each patient's record the P12 dataset includes multivariate time series data collected during their initial 48- 48-hour stay in the ICU. The time series data consists of measurements from 36 sensors (excluding weight). Additionally, each sample is associated with static vector containing 9 elements, including age, gender, and other relevant attributes. Furthermore, each patient in the P12 dataset assigned a binary label indicating the length of their stay in the ICU. A negative label signifies hospitalization period of three days shorter, while a positive label indicates a hospitalization period exceeding three days. It is available http:://hhysoone...oooooonnnnnnc
MIMIC-III The MIMIC-III 45 dataset widely used database that comprises de-identified Electronic Health Records of patients who were admitted the ICU at Beth Israel Deaconess
15

---

Medical Center from 2001 to 2012. It Originally encompassed around 57,000 records of ICU patients, containing diverse variables such as medications, in-hospital mortality, and vital signs. [48] established a variety of benchmark tasks using a subset of this database. In this paper, we focus on the binary in-hospital mortality prediction task to assess classification performance. Following preprocessing, our dataset consists of 16 features from the preceding 48 hours and 21,107 samples. It is available at
Physionet Physionet 44 contains the data from the first 48 hours of ICU patients, which is a reduced version of P12 considered by prior work. Therefore, we follow the same preprocessing methods used for the P12 dataset. The processed data set includes 3997 labeled instances. We focus on predicting in-hospital mortality. is available https//ahyyinneeenooooomnnnnnend
C Baselines
The implementation of baselines follows the corresponding papers or default implementation of their code repositories. Considering that different baselines have varying convergence speeds, we search for the number of epochs in the range of {10, 20, 50} to ensure all baselines can reach convergence. We search for the learning in the range of (0.001, 0.005] due the differences in model complexity and dataset size. Since all four datasets are highly imbalanced, we upsample the minority class in each batch make the batch balance. Here are the detailed hyperparameter settings of these baselines:
ODE-RNN 4: ODE-RNN uses neural ODEs to model hidden state dynamics and an RNN to update the hidden state in the presence of a new observation. The latent dimension is set as 40, and the ODE function has 3 layers with 50 units. The source code can be found at https://github.comm/ https////hhbbccomm//aaaaaabaaa/a YuliaRubunnvv//atenn, ode
GRU-D 17
GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval, and effectively incorporates them into a deep model architecture. The number of hidden states of GRU-D set as 49. We the open source code from https///jttub.comm/Haaa///CC
SEFT []] A set function approach where all the observations are modeled individually before pooling them together using attention-basee approach. We use constant architecture for the attention network f' with 2 layers, heads and dimensionality of the dot product space d of 128. In addition, the attention network f' was always set to use mean aggregation. We use the open source code from https//gitubb.cmmBooraaddd Bowoo////Bogaaaaaaaa/sseeee
A deep learning framework for ISMTS data that learns embedding of continuous time values and uses an attention mechanism to produce fixed-length representation. We use the encoder of the overall framework for classification tasks. We set the latent dimension and the hidden size GRU to 32. The number reference points and the dimension of time embedding is 128. The source code can be found at http://gttuu.ccom//mmmllaaee
mTAND
6
IP-Net 14:
A model architecture for ISMTS data based on several camimearaettric interpolation layers organized into an interpolation network followed by prediction network GRU. The number of reference points is set as 192. The hidden size of GRU is 100. We take the source code at http:///thbb.coommmmmmmala//e///
D B A generative model, which tracks the transition of latent clusters instead of iso- lated feature representations, achieves robust sparse time series modeling. We use the DGM2-0 and set both the hidden dimension and the cluster num as 10. We use the source code at
A general graph neural network framework designed for MTS data. We use 5 graph convolution and 5 temporal convolution modules with the dilation exponential factor 2. The graph con- volution and temporal convolution modules have 16 output channels. The skip connection layers all have 32 output channels. The first layer of the output module has 64 output channels, and the second layer has output channel. We use the open source code from: http:///hbbbboo/////////////AAA/
MTGNN 47
16

---

A graph neural network that embeds ISMTS while learning the dynamics of sensors purely from observation data. The dimension of observation embedding is 4. The dimensions of time representation p' and r. are 16. We set the number of Raindrop layers T. as 2. The de is set to 20. The da set equal to the number of sensors. The source code can be found at https///ithub..om/iingg harvard/Raindrop
and the default settings of the model Raindrop 19:
StraTS 18] is a self-supervised transformer for sparse IMTS. We use the implementation at and the following setting in our experiment: hidden dim 64, num layers- = 2, num heads = 16, dropout = 0.2.
DuETT
is dual event time transformer for Electronic Health Records (EHRs). We the implementation at
19
declaration in this repository.
20 transforms IMTS into line graph images and adapts powerful vision transformers to perform time series classification in the same way as image classification. We use the implementation at htttps//gthub..om/eeeerezuu wwt..////V//......ii.//i..
VITST
Warpformer 22 A transformer-basee network that captures features different scales in IMTS using warping modules and dual attention mechanisms. We use three scales with normalized length [(0) 0, (() 0.2 and ( 1. The dimension of representations D is set as 32. The attention heads and the layers of the warpformer are set as 1 and 2, respectively. We use the implementation at http:///tuuu..ooommmmmmaaasa Waann//ooeeeeee/trrrrrrrer Warrffomer
D Performance Metrics
AUROC AUROC is commonly employed in binary classification tasks, where one class is des- ignated as positive and the other as negative. It represents the area under the Receiver Operating Characteristic (ROC) curve, constructed by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR). AUROC ranges from 0 to 1, with higher value indicating better model perfor- mance in accurately discriminating between positive and negative instances. An AUROC equal 0.5 indicates a model's performance equivalent to random guessing, while an AUROC greater than 0.5 signifies superiority random guessing.
AUPRC The Area Under the Precision-Recall Curve is widely used as a performance metric for imbalanced binary classification tasks. It provides a comprehensive assessment of model's precision-recet trade-off. The Precision-Recal curve is constructed by plotting recall on the x-axis and precision on the y-axis. AUPRC ranges from 0 and a higher value indicates better model performance in achieving high precision and recall simultaneously. It has been suggested as good criterion for unevenly distributed classification problems [54]
E
More Implementation Details
Generation of Variables' Textual Descriptions We have chosen three textual sources: the full name of the variable, Wikipedia, and ChatGPT, for variable descriptions. The full name of the variable and Wikipedia sources be directly extracted from dataset deseriptions and web pages without requiring special processing, so we don't 't delve into them further here. Regarding the CharGPT source, the medical properties description for each variable generated by providing consistent question templates to ChatGPT, aiming to maintain relatively uniform format across variables. We initiate queries to ChatGPT using the following template: What medical prior knowledge do you know about the medical variable ! in ? to obtain a textual description for each variable in specific dataset. The position marked as 'O' corresponds the variable name to be queried, and the position marked as ' corresponds to the dataset-specific task description. For instance, the task for the MIMIC-III dataset ICU patients' in-hossittal mortality prediction, so the query for obtaining textual information associated with the variable Heart Rate could be: What medical prior knowledge do you know about the medical variable Heart Rate diagnosing whether ICU patients will die during hospitalization? The following is part of ChatGPT's s answer for this example:
17

---

Heart rate is fundamental physiological variable that plays significant role in diagnosing and assessing the condition of ICU patients and their potential risk of mortality during hospitalization. Here's what you need to know about the medical variable "heart rate" in this context: Normal Heart Rate: normal resting heart rate for adults rypically ranges between 60 to 100 beats per minute (bpm). Deviations from this range can indicate potential health issues. Tachycardia: An elevated heart rate (tachycardia), often above 100 bpm, can be a sign of various medical conditions... ....
The complete text description can be obtained through the code link.
Hyperparameters We search all hyperparameters in the grid to find the best hyperparameters for our proposed model KEDGN. Specifically, our model has a total of 5 hyperparameters: dimension of query vectors q. dimension of variables' node embeddings n, proportion of density score a, dimension of variables' hidden state h, and dimension of structured encoding representations k. For all datasets, ht and k are set to be equal, and we search them over the range (8. 12, 163. Additionally, search the dimension of query vectors q in (5, 7, 9}, the dimension of variables' node embedding n in {7.9. 9, 11} and the proportion of density score a in (1.0, 2.0, 3.0}. The best hyperparameters for each dataset are reported in the code.
F Additional Experiment
F1
Classification Results Using Different PLMs
In KEDGN, we utilize a PLM to extract sentence embeddings of variables' tex-
Table 5: Details of Pre-trained Language Models.
tual medical information as variable se- mantic representations. The choice of PLM is diverse, and in our default imple- mentation, we use BERT. Additionally, we have experimented with other PLMs, including T5 [55), Bart (56), GPT2 (57), LED (58), and Pegasus [59] The de- tailed information of these PLMs is listed
Model
Size
HuggingFacce Key
Architecture
T5
223M
15-base
Macodcr-Deccodee
Bart
139M
facebook/bart-ban
Encoder- Deccderr
GPT2
137M
gpt2
Decoder-Onle
LED
139M
allenai/led-basee 16384
Encoder- Decoder
Pegasus BERT
568M
googletpegasuu--sum
Encoder-Decoder
110M
bert-base-une -uttbbs--uncased
Encoder-Only
in Table We extract the hidden state at
the [CLS] position for BERT. For other models, we uniformly take the last hidden state (average pooling) of the models as the representations of the variables. The classification results of using different PLMS on four datasets are listed in 6
The results show that BERT (Encoder- Only model) achieved high performance on most datasets. Sev- models also demonstrated competitive results, with Bart outperforming
eral Encoder-Decoder BERT on the P12 dataset, LED slightly surpassing BERT on the Physionet dataset and T5 showing slightly lower performance than BERT on the MIMIC-II dataset. However, the Decoder-Only GPT exhibited notably lower results on the P19 dataset and MIMIC-III compared to other models. This situation may be attributed to the fact that, in our task, PLMS are used to extract sentence-level semantic representations, which are utilized to differentiate variables and measure correlations among them, resembling a text classification process among variables. On the one hand, Encoder-Oni Only models may excel at understanding tasks such as text classification, with BERT specifically adding special [CLS] token dedicated to extracting sentence-level overall semantic information. On the other hand, models incorporating decoder involve tasks related to predicting the next word and may be more focused on text generation tasks. They may exhibit a slight deficiency in extracting distinctive sentence-level representations, especially for Decoder- Only models. These models sometimes require carefully designed prompts to guide them in generating high-quality outputs.
In summary, variable semantic representations extracted by different PLMs can impact downstream task performance. Choosing a PL. that more suitable for the task and dataset can further enhance the performance of KEDGN. In the context of the application in this paper, simple pure encoder model may be more suitable for achieving optimal results.
F22 Computational Cost Analysis
18

---

Table 6: Classification results using different PLMs. The best results highlighted in bold, and the second best results are in underlined. The results in the table are presented in the form of (Mean Std %).
P19
Physionet
MIMIC-III
P12
AUROC
AUPRCC
AUPRO
AUROC
AUROC
I5
0.8
59.7
87.3 &
56.9
8.3
48.
8
53.
Bart
0.6
59.8 1.6
85.7 +
4..
82.5 =
44
87.7
2.0
GPT2
1.6
0.2
53.2
LED
0.5
6..
87.5 a 0.6
$7.3
84.5 = 0.4
4..9
86.8 0.5
51.1 1.6
Pegasus (ERT(D)aaul)
0.5
61.1
87.1 +
52.8
84.4 =
4.90
8.5
53.7
92.2 0.5
1.3
87.9 0.2
85.1
48.3
8.5
54.5 1.5
We conduct a analysis of the time and space overhead on the Physionet dataset, with batch size of 128, and utilizing Nvidia 1080Ti GPU infrastructure. The results shown in T Our method achieves balanced time and space overhead. The introduction of textual information involves generating semantic embeddings using PLMM, and the adjustment of the variable graph based on observed local density, both of which can be predetermined and integrated into the preprocess- ing step without increasing model training overhead. Attention-bel based methods (SeFT, mTAND) achieve par- allel computing in the time dimension, resulting low time overhead, but this sacrifices fine-grained feature
Table 7: Comparison of computational costs on Physionet dataset.
Model
tine(min/epuch)
space(MiBB
ODE-RNN
5.06
2582
GRU-D
1.32
796
SeFT
0.07
684
mTTANDD
4658
DGM
0.06
688
Raindrop
0.17
4864
Warpformer
0.33
11084
KEDGN
0.44
1798
extraction the variable level. Furthermore, compared to the same RNN-based method GRU-D, our method only deals with actual observation points at each timestamp. thus significantly reducing time overhead. While our method's runtime 1.3 times that of the latest SOTA model, Warpformer, our space overhead is only 16% of its size. In situations where computational resources permit, our method can further reduce runtime by employing the space-for-time trade-off strategy (such as increasing batch size).
F.3 More Results for Leave-variables-out
Table Classification performance on samples with a fixed set of left-out variables. The best results are highlighted in bold and the second best results are in underlined.
Discard miio
Dataset
Methods
10%
20%
30%
40%
50%
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
mTAND
7..9
24.0
36.5 1.5
138
34.1 3.7
20.6
33.2
2..1
32.5 3.5
DGM
15
16111
382
24
26
72.0..33
34.5+3.00
70.4
32.7
P12
MTGGNN
712
1.5
3.3
295+
68.94..
3.55
68.1
36
67.6+55
3..2
73.0 02
31.7
DoETT
23
2..1
3.7
3.86
Warpfommer
2.
2.3
33.0
77.1
32.2
Ours
79.7 0.4
43.6 1.2
79.2 0.8
42.5 1.6
77.7 m 2.2
40.0 4.0
77.2 2.2
3.7
76.9
2.2
GRU-D
2.3
3.66
2.1
34
88.0 2.5
7.5
87.5 2.8
49.6
86.4 3.5
8.6
1.8
28.
28.1
26.3
24.3
DGM -O
87.
13
873
1372
86616
4.9
1.9
4.97
4.2
45.7
P19
MIGNN
$4.514
48.9 23
84
493
$4.0199
4..2
83.3
45.5
82.5 2.9
42.7
9.2
Randdrup
1.5
15
13
19
$7.8
$9.1 1.7
87.6 1.1
5
87.1
$7.7 2.3
DuETT
1.0
1.0
818
03
84.110
13
84.3
52.7
1.3
2.0
Wappfomer
913
5..
913208
55.1
91440.8
4..0
9.
26.4
91.2
3..9
9122
09.6
GRU-D
3.0
4.11
69.5 2.6
31.1
69.2 3.0
$1.00
68.3 3.6
50.1
68.1 1..1
29.3
mEAND
$0.5221
428
3.4
405
76.3 4.0
75.6 3.9
36.
3.31
36.1
DGM
M12009
286=28
M04009
38
37.1
77.5 3.7
35.4+444
34.0
Physkonet
MIGNN
258
26
Reindrup
76.5
23
76.5
32.5
30.
29.7
73.6
3.9
DoFTT
78
3..9
78.3 3.0
30.003.
76.7 3.7
37.9
75.9 3.8
37.0+44
74.9+43
35.+
Warpformer
78.2 1.0
2..
77.7
1.6
33.6
1..
31.8
73.8 4.5
3..2
72.7 1.9
29.2
Ours
13
29.4
2.
5.3
81.7
464455
81.4
45.
81.1
45.2
19

---

F.4 More Results for Ablation Study
Table 9: The ablation study of our proposed method KEDGN. The results in the table are presented in the form of (Mean + Std %).
Physionet
MIMIC-III
P12
AUROC
AUPRRC
AUROC
AUPRC
AUROC
AUPRC
weo VSW
86.6
51.1
8..6
4.
0.9
50.8 1.2
w/o Text
87.7 2 0.4
$..3
0.7
4.5
87.7
$,99
w/o Graph
86.8 1
1.6
84
4.50
8.6
53.2 1.3
KEE
86.6
53.3 1.8
0.7
4
87
0..
wio DAG
87.2 0.9
2.4
00.
47.7 2.3
8.1
52.1+ 1.1
3.6
8..2
8..
Full
87.9
$ 1.8
85.1
48.3
87.8 0.5
41.5 1.5
We find that the model performance declines after introducing text information on P12 dataset. As indicated in Table when we replace PLM from BERT to Bart, there is slight improvement of 0.5% in AUPRC on the P12 dataset, slightly outperforming the model without text. Therefore, the reason for the decline in model performance is likely attributed to the choice of default PL.M (BERT), which may not be optimal for extracting textual embeddings on this dataset, rather than the text itself causing the decline. In other words, introducing text offers the potential for enhancing model performance, with the extent of improvement depending on the degree effective utilization of text.
F5 Effects of different activation functions in Eq.
Table 10: Comparison of the results of different activation functions in 7..
P19
Physionet
MIMIC-III
P12
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
w/o /7
90.3 = 0.9
58.1 1.8
86.2 + 0.8
52.8 4.5
84.3 0.6
4 1.5
86.8 = 0.5
$1.0 1.2
Sigmoid
91.6
2.6
a Tanh (Default)
9.5
62.0 1.3
87.9 0.2
$7.1 1.8
85.1 +
48.3 1.6
8.5
$4.5 1.5
On one hand, the activation function in Eq. reflects the dynamics of variables, such as time decay or exponential increase. On the other hand, this activation function serves normalization purpose because the edge weights in the thowledge-empowerd complete graph A are normalized values. If directly using the absolute value of the density adjust the edge weights, the values might become excessively large or small, which would severely disrupt the basic graph structure learned from textual knowledge. Typical activation functions with normalization capabilities include Sigmoid and Tanh. As shown in Table TO Tanh is chosen since it performs better.
F.6 Parameter Complexity Analysis
Table 11: Comparison the number of model parameters for three models.
Datasets
P19
Physionet
MIMIC-III
P12
Raindrop
1947668
19789024
35647028
19789024
Warpformer
43780
43974
42034
43974
Ours
80262
46545
51129
63087
As shown in Table [I] the parameter count of our model is not particularly high. It is on the same order of magnitude as Warpformer and significantly lower than Raindrop by three orders of magnitude. Although we calculate an independent parameter space for each variable, the total W, E R xxxxxo does not equate to the parameter count. W, is derived from the multiplication of two matrices: the variable embedding matrix Q E Vx4 and the weight matrix W E Rxx The first matrix is computed from textual embeddings, and only the second matrix belongs to the model parameters. The sizes g, I, and O are hyperparameters, independent of the number of variables, and typically set to be less than 16. This ensures that the parameter complexity of our model remains within an acceptable range.
20

---

F.7
More variables groups on the P12 dataset
Mg
No.
HCC3
Invasive arterial blood pressure
Renal function indicators
Ion concentration indicators
NDissaBP
WBC
PaD2
Platelets
5002
20
15
(d) Non-invasive blood pressure
(e) Blood cell indicators
(f) Oxygenation status indicators.
Figure 7: Variable groups (Partial) divided by temporal patterns on the P12 dataset.
F88
More Visualizations of Variables Textual Representations
Heart
sop Map
Magnssium
Phosphate
WBC
Poturolum
20
30
20
Lime
time
(a) Vital signs indicators
(b) Ion concentration indicators
(c) Blood cell indicators
Figure 8: Variable groups (Partial) divided by temporal patterns on the P19 dataset.
200
Heart rute Reep
DBP MAP SEP
200 Reep
SBI
DBP
MAP
WBC Platelets'
200
Platelets
Magnesium
DHP
WBC
100
100
Reep
Heart
Platelels WBC
Phosphate
Phosphate
-100
Mugnestum
#otassium
Magnesium
Chioride
-100
Potassium
Chloride
MAP
Phoophate
2001
280
ChatGPT
(b) Wikipedia
(c) Random Initialized
Figure 9: T-SNE visualization of partial variable semantic representations on the P19 dataset.
HR
MAP SUP RR
GCS-MR
GCS-VR
10
20
time
Uime
(a) Vital signs indicators
(b) Neurological scoring indicators
Figure 10: Variable groups (Partial) divided by temporal patterns on the MIMIC-II dataset.
21

---

200
GÓS-MR GCS-T
GUS-EO
100
RR
250-
JIR GCS-MR
RR
GCS-VR
SBP
MAP
100
DBF
MAP
pBP MAR SBP
-100
SBP GCS-T DRP
-200
GCS-VR GCS-EO
GCS-T,
-500-
GCSSYR
-100
Gts-MI&
-400
-200
200
50
100
150
-400
-200
200
400
(a) ChatGPT
(b) Wikipedia
(c) Random Initialized
Figure 11: T-SNE visualization of partial variable semantic representations on the MIMIC-III dataset.
E.9
Visualization of Variable Semantic Representations Using Different PLMs
200
Heart rate
Resp
DBP MAP
400
WRC
Platelets
100
DBP
Heart
100-
Plntelets WBC
Heart
-50
200
Reep
-100
Magnesium
Phosphate
100-
DBP MAP
SBP
Chloride
Potassium
Magnesiumg Phosphate?
-100
-150
Magneslum
Potassiurs
Phosphate Chloride
Resp
Chloride
WBO Platekets
200
-50
50
(a) BERT(AUPRC=62.0)
(b) LED(AUPRC=00.3)
GPT2(AUPRC=54.))
Figure 12: T-SNE visualization of variable semantic representations generated by different PLMs on
the P19 dataset.
GCS-MR GGCS-T
CUS-EO
100
MAAP
'R
PBP
SBP
SBP
HRR
DBP
*GCS-VR
MMAP
JRR
-100
GCS-EQ
DBP
0+
GCS-MR
JR HR
SBP
-300
GCS-EO GCS-VR
-50
GCS-T
GCS-T
GCS-VB.
-400
-300
-200
-200
200
(a) BERT(AUPRC==4..)
(b) T5AAUPCC-4800)
GPT2(AUPRC=43.I)
Figure 13: T-SNE visualization of variable semantic representations generated by different PLMs on the MIMIC-III dataset.
We conduct T-SNE visualization analysis on variable representations obtained from different PLMs. Figures I2 and 3 correspond to the results of the P19 and MIMIC-III datasets, respectively. We observe that when the clusters of variable representations extracted by PLMs are consistent with the grouping of variable time series patterns and exhibit distinctiveness, the corresponding downstream classification task performance tends to be better. For example, on the P19 dataset, BERT and LED achieve higher classification performance, with their corresponding variable representations having good distinctiveness: the three groups of variables represented by blue, yellow, and green colors show high cohesion and low coupling, while the T5 model, which exhibits suboptimal classification performance, has a distribution where Resp. WBC, and Platelets are notably confused with the green variables. Additionally, the MIMIC-II dataset, the variable representations obtained from BERT and T5, which achieve higher performance, with blue and green clusters having longer distances, indicating better distinctiveness. On the other hand, variable representations obtained from GPT2 show poor distinctiveness, with the variable RR being close to CCS-EE and GCS-MR.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limiations of the work in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The code and data are provided by Github repository link.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification:
The code and data are provided by Github repository link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
ceciionooolccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pullic//uuii wwwtii/gguie///eeeeceoocc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Section 5.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide the standard deviation of the mean in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information the computer resources in Appendix F22
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte::://euuis..eeeee wwweerrps.cccppppeeeeeee https//ccpp...itttcccccciddeeded
Answer:
[Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
26

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
27

---

Answer:
[NA]
Justification:
We release our model by a Github URL.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "LhVDMLbFgd", "reviewer": "Reviewer_vHK1", "summary": "This paper investigates the problem of irregularly sampled medical timeseries classification. The core idea is to use PLM to obtain semantic embeddings for variables, which are used to form a variable correlation graph. Then, the variable correlation graph is dynamically adjusted with the observations, based on which a spatiotemporal graph neural network is used to learn timeseries representation for classification. Typical experiments on widely used benchmarks are conducted to evaluate model performance.", "questions": "1. What's the function of activation function $\\sigma$ in equation (7). In my opinion, this activation function reflects the dynamics of variables, like time decay or exponential increase. So, how to determine the function and what are the effects of different activation functions.\n2. How is the parameter complexity of the model? Since each variable will require an independent $W_i$, the computation complexity might be very large.\n3. This paper emphasizes the importance of using text information of variables. However, the performance of different text sources seems close. Especially, only using Name can achieve similar even better performance. From ablation study in Table 3, we can observe even without textual representations, the model can achieve almost similar performance. Thus, this part looks useless, which is opposite to the commonsense. \n4. Is Figure 5(a) calculated from f(E) or E? If in the f(E), the authors should also provide the correlation graph from E, so that we can evaluate whether model learning is effective. To my experience, even bert embedding of these variables can achieve similar correlation graph. Also, why graph (a) is not symmetric? What's meaning of different correlations between (FiO2, HR) and (HR, FiO2)?\n5. What's the basic time resolution or sampling rate for each dataset?\n6. How is the static features combined with timeseries are not described in paper.\n7. How are the input length and prediction length determined are not provided. Usually, in mortality prediction, we use the first 48 hours ICU data as input to predict whether the patient will die in the hospitalization. What's setting for this paper?", "limitations": "The authors didn't comprehensively discuss the limitations of this paper, which is not accord with the paper checklist.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand. The authors clearly tell the story of using variable correlation graph to solve irregularly sampled medical timeseries classification problem.\n2. The investigation of different PLMs on variable correlation graph discovery is important and useful to the community, although the conclusion is that most PLMs achieve similar results, and even name can achieve comparable performance with wikipedia.\n3. The experiment section is well defined with various results illustrated.", "weaknesses": "1. The parameter settings for baselines are not fair. In lines 228-230, the authors simply follow parameters in reference. However, such setting is not a fair way comparison due to several reasons: Some datasets like MIMIC-III for experiments are not public, i.e., although we have the same raw data source, the experimented datasets are never exactly same, and may be very different from distribution and statistics. This can be validated from the comparison between the results of Raindrop on P19 dataset from this paper and Raindrop paper. The performance has a significant difference, which implies the parameters should searched completely. Thus, grid search for baseline parameters is necessary.\n\n2. The choice of GCRNN here is not well clarified. There are many spatiotemporal graph neural networks like STGNN /ST-GCN which can be used here. Moreover, the ablation study of graph convolution and temporal modeling for GCRNN should be conducted to evaluate the performance gains.\n\n3. Some claims or statements intend to emphasize the model performance, which however is obviously set by design itself instead of model learning. For example, in Line 337 the authors state ''no correlation is observed with DiasABPas it has not been observed yet''. Actually, this is not by learning but the initial masking operation of adjacency matrix.\n\n4. An important baseline of irregularly sampled medical timeseries modeling is missed here, i.e., StraTS [1].\n\n5. The ablation study doesn't identify very effective module of the proposed work. For example, Text, KEE and DAG's performance gains are marginal, so are these module necessary?\n[1] Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series, TKDD.\n\nTypos:\n1. The symbols or bold texts in Equation (10-12) are wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer vHK1(2/3)\n\n**Q1. The function of activation function in equation (7). **\n\nA1. On one hand, as you mentioned, this activation function reflects the dynamics of variables, such as time decay or exponential increase. On the other hand, this activation function serves a normalization purpose because the edge weights in the knowledge-empowered complete graph $\\boldsymbol{A}$ are normalized values. If we directly use the absolute value of the density to adjust the edge weights, the values might become excessively large or small, which would severely disrupt the basic graph structure learned from textual knowledge. Typical activation functions with normalization capabilities include Sigmoid and Tanh. Since Tanh performed better in our experiments, we chose Tanh.\n\nThe experimental results for different activation functions are shown in rows W/O $\\sigma$ and $\\sigma$ Sigmoid in Table 2 of the newly submitted PDF.\n\n\n\n**Q2. The parameter complexity of the model.**\n\nA2. We have calculated the number of model parameters for each dataset, as shown in Table 5 of the newly submitted PDF.\n\nAs shown, the parameter count of our model is not particularly high. It is on the same order of magnitude as Warpformer and significantly lower than Raindrop by three orders of magnitude. Although we calculate an independent $W$ for each variable, the total $ W \\in \\mathbb{R}^{V \\times I \\times O} $ does not equate to the parameter count. $ W $ is derived from the multiplication of two matrices: the variable embedding matrix $ Q \\in \\mathbb{R}^{V \\times q} $ and the weight matrix $ W \\in \\mathbb{R}^{q \\times I \\times O} $. The first matrix is computed from textual embeddings, and only the second matrix belongs to the model parameters. The sizes $ q $, $ I $, and $ O $ are hyperparameters, independent of the number of variables, and typically set to be less than 16. Hence, the parameter complexity of our model is not high. We will include this parameter complexity analysis in Appendix F.2 on computational costs.\n\n\n\n**Q3. The importance of using text information of variables.**\n\nA3. First, as analyzed in Section 5.4.1, the relative distribution of variable embeddings obtained from different text sources via PLM is consistent with the relative distribution of the variables' time series patterns. Both of them reflects the relative distribution of the variables' inherent sense. The embedding space of variables obtained from different text sources may vary in absolute distribution, but the relative distribution should be similar. So the performance from different text sources should also be similar, which aligns with our expectations and proves incorporating text information has a universally valid.\n\nMoreover, as shown in table 4 of the newly submitted PDF, the overall results of the ablation experiments indicate that the introduction of text contributes to performance improvement and is not useless.  More importantly, as demonstrated in the visualizations in Section 5.4.1, if the variable text embeddings are replaced with learnable embeddings, the resulting learned variable embedding space tends to be nearly uniformly distributed, which lacks interpretability. In contrast, using text embeddings ensures that the variable embedding distribution is consistent with domain knowledge, thereby providing the model with strong interpretability.\n\n\n\n**Q4. Calculation and symmetry issues of correlation graph**\n\nA4. Your observation is correct. Figure 5(a) is computed from $g(E)$. It is true that using the BERT embeddings of these variables can achieve a broadly similar correlation figure, but there are subtle differences induced by $ g(\\cdot) $. The introduction of $ g(\\cdot) $ is necessary as it not only performs feature reduction but also avoids using a completely fixed prior graph. This preserves the model’s ability to adaptively optimize the graph structure based on different data distributions and downstream tasks.\n\nTo validate the effectiveness of $ g(\\cdot)$, we provide the ablation experiment results on row W/O $ g(\\cdot)$ in table 2 of the newly submitted PDF.\n\nRegarding the asymmetry of Figure 5(a): As indicated in Eq.(4), after computing the cosine similarity between variable node embeddings, we apply the Softmax function to normalize each column to obtain the graph. This normalization ensures that the diagonal elements correspond to the maximum values in their respective columns, but not necessarily the maximum values in each row.\n\nWe intentionally set the graph to be asymmetric because some variable correlations are directional. For example, a decrease in insulin level may lead to an increase in blood glucose, but an increase in blood glucose does not necessarily lead to a decrease in insulin level. This is why the edge weight from insulin to blood glucose is not equal to the weight from blood glucose to insulin. The same reasoning applies to HR and FiO2.\n\n\n\n**Q5. The basic time resolution or sampling rate for each dataset.**\n\nA5. In ISMTS, there are significant differences in sampling rates among different samples, variables, and periods. As a result, calculating a basic sampling rate for an entire dataset is challenging and sometimes not very informative. Instead, more relevant statistics for ISMTS datasets include metrics such as missing rate and maximum length.\n\n\n\n**Q6. How are the static features combined with time series.**\n\nA6. We apologize for the omission. We follow the approach used in Raindrop to incorporate the static features. Specifically, static features are first mapped to static vectors through a linear layer and then concatenated with the feature vectors of the time series before being input into the classifier.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "to7wSaCku9", "reviewer": "Reviewer_yFgU", "summary": "Throughout this article, the authors have described an approach (KEDGN) to tackle Irregularly Sampled Medical Time Series. In this task the data is composed of multiple variables represented with time series. The sample rate of a time series can be irregular and two time series can have their samples taken at a different timestamp.\nIn the approach described the authors propose to first find common knowledge correlation between the variables in the dataset. To do so, they compute a non-linear cosine similarity between embeddings of textual descriptions, obtained using a Pre-trained Language Model.\nThey then precise this common knowledge correlation by adding a timestamp aware correlation. Finally using this correlation information and a Graph Convolutional Recurrent Network, their approach can predict the class expected of the time series.\nThey have then analysed the results of their approach first through a benchmark with multiple state of the art approaches. They then added an ablation study and different visual analysis to give a better overview of the behavior of the approach.", "questions": "* Did the authors try to use definitions given by health care professionals to confirm the definition given by the GPT model?\n* It is not clear how equation (7) uses equation (6). Indeed, $Z^{(i)}_{v}$ was defined with $i$ being the $i$-th observation point (I believe the $i$-th sample of this specific variable $v$).\nBut on the other hand the equation (7) uses $Z$ with $t$ being a timestamp. --\n\nIt is not clear if $Z^{(i)}_{v}$ is defined for a timestamp in which no observation was made for a variable.\n* It is not clear why the equation (9) contains a transpose.", "limitations": "It appears the limitations have been discussed in sufficient detail.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposal to use Common Knowledge through PLM to find correlation\n* Detailed Experiment with a detailed comparison with state of the art approaches\n* Detailed exploration of the behavior of the model through an ablation study and visual analysis of the Embedding of the explanations.", "weaknesses": "* Details of the Proposed Model could be better explained by giving the intuition behind the equations. (Especially for 4.3.2 and 4.4)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer yFgU (1/1)\n\nDear reviewer yFgU, \n\nThank you for your valuable feedback on the formula details of the article. Here is the response:\n\n**W1. Details of the Proposed Model could be better explained by giving the intuition behind the equations. (Especially for 4.3.2 and 4.4)**\n\nA1. Firstly, we need to correct equation(8) in Section 4.3.2 to:\n$ G^{(t)}_{ij} = A^{(t)}\\_{ij} \\times (1 - W\\_{ij} |D^{(t)}\\_{i} - D^{(t)}\\_{j}|).$\n\nThe intuition behind this equation is as follows: If two variables have similar densities at a given time, their correlation should be higher. Conversely, if the density of one variable increases or decreases significantly at subsequent times, the correlation between these variables will decrease. For example, Figure 6 in Section 5.4.3 shows that NIDiasABP and HR have similar densities at $t=4$, but by $t=15$, the density of NIDiasABP decreases, which reduces its correlation with HR. Therefore, we use the absolute difference in densities between the two variables and add a negative sign so that the density difference is negatively correlated with the correlation.\n\nRegarding the formulas in Section 4.4, they integrate variable-specific parameter spaces and dynamic graph networks into the GCRNN backbone network. Equation (9) represents the standard definition of  $1st$-order Chebyshev polynomial expansion approximation for graph convolution. Equations (10-12) define the operations in GCRNN. Equation (13) indicates that only the hidden states of observed variables are updated at each time step, ensuring that the model fully adheres to the original ISMTS pattern.\n\n\n\n**Q1. Did the authors try to use definitions given by health care professionals to confirm the definition given by the GPT model?**\n\nA2. In this paper, what the model required as input are general medical domain knowledge text descriptions of each variable. These domain knowledge is relatively well-established and fixed, and can be confirmed by consulting medical literature. Therefore, we did not seek help from health care professionals.\n\n\n\n**Q2. $Z^{t}$ 's definition is not clear.**\n\nA3. We apologize for the confusion regarding the notation. We have revised the definition of $Z^{(t)}$ in Equation (6) as follows:\n$$\nZ^{(t)}=Z_{i, v}=\n\\begin{cases}\n((t_{i,v}-t_{i-1,v}) + (t_{i+1,v}- t_{i, v})) / 2, \\quad  \\text{if }t_{i+1,v} \\text{ and } t_{i-1,v} \\text{ are both not None.}\\\\\\\\\nt_{i,v}-t_{i-1,v}, \\quad  \\text{if }t_{i+1,v}  \\text{ is None.} \\\\\\\\\nt_{i+1,v}- t_{i, v}, \\quad  \\text{if } t_{i-1,v} \\text{ is None.}\\\\\\\\\nt_{max} / 2, \\quad  \\text{if }t_{i+1,v} \\text{ and } t_{i-1,v} \\text{ are both None.}\n\\end{cases}\n$$\nHere, $Z^{(t)}=Z_{i, v}$ represents the average density at time stamp $t$ for the $v$-th variable's $i$-th observation point. It is important to note that density is used for dynamically adjusting edge weights, and edges only exist between actual observations. Therefore, when a variable has no observations, there are no edges to adjust, so the density calculation is not needed in such cases. Hence, we did not consider this situation in defining the average density.\n\n\n\n**Q3. It is not clear why the equation (9) contains a transpose.**\n\nA4. As indicated in Equation (4), after computing the cosine similarity between variable node embeddings, we apply the Softmax function to normalize each column to obtain the graph. Since matrix multiplication involves the left matrix’s specific rows and the right matrix’s specific columns, and each column of the right matrix $S \\in \\mathbb{R}^{V \\times I}$ represents the values of all nodes for a specific input channel, we transpose $I_V + G(t) $ in Equation (9). This transposition converts each column into a row, representing the correlations of one variable with all other variables, ensuring that the sum of the output edge weights for each variable node is 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AodU1KY8Bj", "reviewer": "Reviewer_68iN", "summary": "This work considers the irregular timestamp contained in current medical data. They employ a density-aware mechanism to the time-varying correlations among variables.", "questions": "see weakness", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work addresses an important topic in this field and is well-organized and written. The experiments are sufficient to prove their conclusions.", "weaknesses": "1. In formulation (6), the density of the observation point is calculated by the average time interval. What if the point is not observed in the next or previous time? e.g. t_v^(i+1) or t_v^(i) is None.\n2. I am just a little bit confused by the node in the dynamic variable graph. As we know, the patient's EHR data will contain lots of clinical notes and some like treatment, medications, and so on. You have shown the description in your model's framework in Figure 3. So do you consider other variables in patient's EHR data? Besides, the node here denotes a text, not detailed medical terminology, right?\n3. Could you please add a comparison graph between the original variable representation and the final representations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer 68iN (1/1)\n\nDear reviewer 68iN, \n\nThank you for your valuable feedback on the formula details and completeness of the paper. Here is the response:\n\n**W1. $Z^{t}$ 's definition is not clear**\n\nA1. We apologize for the oversight. If there are no observations at the preceding/following time points, we take the time interval of the following/preceding observation as the density. If neither a preceding nor a following observation exists, it indicates that this observation is the only one for the variable, so we take half of the maximum observation time span across all variables as the density.\n\nThe corrected definition is as follows:\n$$\nZ^{(t)}=Z_{i, v}=\\begin{cases}((t_{i,v}-t_{i-1,v}) + (t_{i+1,v}- t_{i, v})) / 2, \\quad  \\text{if }t_{i+1,v} \\text{ and } t_{i-1,v} \\text{ are both not None.}\\\\\\\\t_{i,v}-t_{i-1,v}, \\quad  \\text{if }t_{i+1,v}  \\text{ is None.} \\\\\\\\t_{i+1,v}- t_{i, v}, \\quad  \\text{if } t_{i-1,v} \\text{ is None.}\\\\\\\\t_{max} / 2, \\quad  \\text{if }t_{i+1,v} \\text{ and } t_{i-1,v} \\text{ are both None.}\\end{cases}\n$$\nHere, $Z^{(t)} = Z_{i, v} $ represents the average density at time $ t $ for the $ v $-th variable's $ i $-th observation.\n\n\n\n**W2. I am just a little bit confused by the node in the dynamic variable graph. As we know, the patient's EHR data will contain lots of clinical notes and some like treatment, medications, and so on. You have shown the description in your model's framework in Figure 3. So do you consider other variables in patient's EHR data? Besides, the node here denotes a text, not detailed medical terminology, right?**\n\nA2. Here, we use general medical semantic text to differentiate between variables and guide the modeling of correlations between these variables in irregular time series. If I understand correctly, the \"other variables in patient's EHR data\" you mentioned likely refer to patient-specific clinical records, treatment history, medication history, etc. These are personalized medical data and not general medical domain knowledge. They often include multiple modalities, such as text and images, which extend beyond the scope of time series analysis. Our datasets only contain time series data and do not include clinical notes. Thank you for the constructive suggestion; we may consider integrating multimodal medical data in future work.\n\n\n\n**W3. Could you please add a comparison graph between the original variable representation and the final representations?**\n\nA3. Certainly, but due to the limited information that can be uploaded during discussion period, we will include this in a later version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "419EYzBxGF", "reviewer": "Reviewer_n8KK", "summary": "The paper aims to handle difference and correlation between multiple variables in the irregularly sampled medical time series data. By proposing a model named KEDGN with textual medical knowledge and dynamic variable graphs, it is able to outperforms baselines on a variety of datasets. Comprehensive experiments have also been conducted to demonstrate the learned correlations between variables, which aligns with the motivation of this paper.", "questions": "1.\tIt seems some of the baselines such as DGM^2-O can reach good performance with relatively less time and space requirements. Can there be any potential improvements to the efficiency of the proposed KEDGN model? It seems BERT is also optimized during training, it would be interesting to see some efficiency experiments and analysis regarding this aspect.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe motivation of the paper is solid. The authors provide enough evidence and arguments to demonstrate the dynamic correlation patterns between time series data.\n2.\tThe paper is generally well written with clear figures and equations.\n3.\tThe proposed model demonstrates strong performance in various experiment settings. Comprehensive analysis is also being conducted to prove the effectiveness of each module design.\n4.\tIt is good to see some visualizations about the variable correlations learned by the model, which aligns with the motivation of this paper.", "weaknesses": "1.\tThe idea of learning dynamic graphs that model variable relationships is quite common in literatures. It would be better for the authors to cite related works, compare with existing works and make arguments about novelty and effectiveness of the proposed modules. Here are some references:\n    -\tJiang, Renhe, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. \"Spatio-temporal meta-graph learning for traffic forecasting.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 7, pp. 8078-8086. 2023.\n    -\tHuang, Qihe, Lei Shen, Ruixin Zhang, Shouhong Ding, Binwu Wang, Zhengyang Zhou, and Yang Wang. \"Crossgnn: Confronting noisy multivariate time series via cross interaction refinement.\" Advances in Neural Information Processing Systems 36 (2023): 46885-46902.\n    -\tWang, Dingsu, Yuchen Yan, Ruizhong Qiu, Yada Zhu, Kaiyu Guan, Andrew Margenot, and Hanghang Tong. \"Networked time series imputation via position-aware graph enhanced variational autoencoders.\" In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2256-2268. 2023.\n    -\tWang, Binwu, Pengkun Wang, Yudong Zhang, Xu Wang, Zhengyang Zhou, Lei Bai, and Yang Wang. \"Towards Dynamic Spatial-Temporal Graph Learning: A Decoupled Perspective.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, pp. 9089-9097. 2024.\n2.\tThere exist some missing references and baselines for irregularly sampled time series classification problems. It would be better to cite related works and compare with them in the experiment sections. Here are some examples:\n    -\tLi, Zekun, Shiyang Li, and Xifeng Yan. \"Time Series as Images: Vision Transformer for Irregularly Sampled Time Series.\" Advances in Neural Information Processing Systems 36 (2024).\n    -\tLabach, Alex, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. \"Duett: Dual event time transformer for electronic health records.\" In Machine Learning for Healthcare Conference, pp. 403-422. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer n8KK (1/1)\n\nDear reviewer n8KK, \n\nThank you for your valuable feedback on the supplement of relevant literature of the article. Here is the response:\n\n**W1. The idea of learning dynamic graphs that model variable relationships is quite common in literatures. It would be better for the authors to cite related works, compare with existing works and make arguments about novelty and effectiveness of the proposed modules.** \n\n- \"Spatio-temporal meta-graph learning for traffic forecasting.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 7, pp. 8078-8086. 2023.\n- \"Crossgnn: Confronting noisy multivariate time series via cross interaction refinement.\" Advances in Neural Information Processing Systems 36 (2023): 46885-46902.\n- \"Networked time series imputation via position-aware graph enhanced variational autoencoders.\" In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2256-2268. 2023.\n- \"Towards Dynamic Spatial-Temporal Graph Learning: A Decoupled Perspective.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 8, pp. 9089-9097. 2024.\n\nA1. Thank you for the reminder. In a later version, we will cite these four papers in the section on “Graph Neural Networks for Multivariate Time Series” in the related work. Although modeling dynamic graphs to capture relationships between variables is not new, many dynamic spatiotemporal graph methods are designed for traffic flow prediction tasks, which involve regularly sampled multivariate time series. Applying these methods directly to irregularly sampled medical time series classification tasks may lead to suboptimal performance. Our approach specifically addresses the challenges of ISMTS by using domain knowledge as guidance and adjusting edge weights in the graph based on observation masks and dynamic densities, which is a key distinction from these methods.\n\n\n\n**W2. There exist some missing references and baselines for irregularly sampled time series classification problems. It would be better to cite related works and compare with them in the experiment sections.** \n\n- Li, Zekun, Shiyang Li, and Xifeng Yan. \"Time Series as Images: Vision Transformer for Irregularly Sampled Time Series.\" Advances in Neural Information Processing Systems 36 (2023).\n- Labach, Alex, Aslesha Pokhrel, Xiao Shi Huang, Saba Zuberi, Seung Eun Yi, Maksims Volkovs, Tomi Poutanen, and Rahul G. Krishnan. \"Duett: Dual event time transformer for electronic health records.\" In Machine Learning for Healthcare Conference, pp. 403-422. PMLR, 2023.\n\nA2. Thank you for the reminder. In the revised version, we will cite these two papers in the section on “Irregularly Sampled Multivariate Time Series Modeling” in the related work. Additionally, we have conducted comparative experiments, and the results are as follows:\n\n|            | P19      |          | Physionet |          | MIMIC-III |          | P12      |          |\n| ---------- | -------- | -------- | --------- | -------- | --------- | -------- | -------- | -------- |\n|            | AUROC    | AUPRC    | AUROC     | AUPRC    | AUROC     | AUPRC    | AUROC    | AUPRC    |\n| ViTST      | 91.7±0.1 | 57.5±0.7 | 81.3±1.9  | 37.4±2.9 | 81.8±0.3  | 39.6±1.3 | 86.3±0.1 | 50.8±1.5 |\n| Duett      | 88.2±0.5 | 56.0±3.9 | 81.3±1.4  | 44.9±1.4 | 78.8±0.8  | 34.3±1.0 | 83.4±1.2 | 45.4±1.5 |\n| Ours(Best) | 92.3±1.0 | 62.5±0.7 | 88.2±1.1  | 57.5±2.5 | 85.1±0.3  | 48.4±1.5 | 87.8±0.5 | 54.5±1.5 |\n\n\n\n\n\n**Q1. It seems some of the baselines such as DGM^2-O can reach good performance with relatively less time and space requirements. Can there be any potential improvements to the efficiency of the proposed KEDGN model? It seems BERT is also optimized during training, it would be interesting to see some efficiency experiments and analysis regarding this aspect.**\n\nA3. Firstly, it should be clarified that BERT is frozen and does not participate in the optimization of training. We only take the sentence representation output by BERT as the semantic representation of variables, so the training cost is consistent with the variable embedding of random initialization. In addition, due to the sequential nature of RNN computation, there is an inherent bottleneck in the running time of our model. We have optimized the code implementation as much as possible to keep the overall running cost of the model within an acceptable range. We acknowledge this as a limitation of our model and will add it to the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We sincerely thank all the reviewers for their consistent positive feedback regarding the significance of our work, the novelty of our approach, the thoroughness of our experiments and analyses, and the quality of our presentation. Additionally, we greatly value the reviewers' insightful and constructive comments, which have significantly contributed to enhancing and refining this work. Here, we will response each of the weaknesses and questions raised by the reviewers in order to eliminate reviewers' concerns to the greatest extent possible. 

Due to the limited number of pages and characters allowed for rebuttal, we will begin our responses to Reviewer **vHK1** on this global rebuttal page. The responses to reviewers **yFgU, 68iN, and n8KK** can be found below the respective official review pages.



# Responses to Reviewer vHK1(1/3)

Dear reviewer vHK1, 

We thank for your carefully reading on our paper and valuable comments on the details of our work, the following is our responses, please download the 1-page PDF document we have newly uploaded at the left lower corner of this rebuttal page for easier reference and reading.



**W1.Grid search for baseline parameters is necessary.**

A1. We performed a grid search across four datasets around the optimal parameter range for the three most recent SOTA baselines due to time constraints. The results are shown in Table 1 of the newly submitted PDF.

As you mentioned, some baselines did indeed show some improvement in performance on certain datasets after the grid search. However, our method still consistently maintained leading performance across all datasets. In comparison, although some baselines demonstrated competitive performance, they still exhibited significant disadvantages on a few datasets. For instance, Warpformer on the Physionet dataset, $DGM^{2}-O$ on the MIMIC-III dataset, and Raindrop on the Physionet dataset. Therefore, the effectiveness of our method remains evident.



**W2. The choice of GCRNN here is not well clarified./The ablation study of graph convolution and temporal modeling.**

A2. Firstly, our main contribution is not the improvement of the GNN backbone network but the learning of variable-specific parameter spaces and dynamic graphs. Therefore, what we need is a simple, effective, and easy-to-adapt backbone GNN for ISMTS. Existing spatio-temporal graph neural networks can be roughly classified based on their temporal module into three main categories: Convolution-Based, Attention-Based, and Recurrence-Based.

- Convolution-Based methods capture temporal dependencies using TCNs. However, the fixed-step kernel size cannot perceive different time spans effectively, making them unsuitable for ISMTS.
- Attention-Based methods face challenges due to the misalignment of sampling times across different variables in ISMTS. Computing variable-specific attention is difficult to parallelize in the variable dimension, and their computational complexity scales quadratically with the sequence length, limiting scalability. Furthermore, attention outputs have the same dimensions as inputs, requiring additional design for extracting final sample features from observation-level representations.
- Recurrence-Based methods update variable latent states based on whether they are observed, which allows parallel computation in the variable dimension and suits ISMTS. The final sample feature representation can be directly obtained from the latent states at the last observation time, making this approach simple, effective, and well-suited for ISMTS.

Thus, we chose GCRNN as the backbone GNN for this paper.

Regarding the ablation study of graph convolution and temporal modeling for GCRNN, we set the graph into an identity matrix and remove the time embedding in the structured encoding respectively. The results are shown in rows W/O GCN and W/O TE in Table 2 of the newly submitted PDF.

The results show that both the introduction of graph convolution and time embedding are necessary.





**W3. Some claims or statements intend to emphasize the model performance, such as Line 337.**

A3. We apologize for the misunderstanding caused by our statement. We have revised the phrasing in Line 337 to: "Around t = 4, HR shows a strong correlation with NIDiasABP, while the correlation with DiasABP is masked as 0 since DiasABP has not been observed yet."



**W4. An important baseline of ISMTS modeling (StraTS) is missed here.**

A4. Thank you for your reminder. We will include StraTS in the "Irregularly Sampled Multivariate Time Series Modeling" section of the related work in future versions. Additionally, we have added comparative experiments with StraTS. The experimental results are shown in Table 3 of the newly submitted PDF.



**W5: The ablation study doesn't identify very effective module of the proposed work.**

A5. We have calculated the average performance improvement brought by the three modules across the four datasets, as shown in Table 4 of the newly submitted PDF.

Overall, all three modules contribute to performance improvement. Given that the baseline values of AUROC are relatively high (typically above 85%), the absolute value of the AUROC improvement may appear lower. In this context, a single module that can achieve an AUROC improvement greater than 0.5% is significant. Additionally, all three modules bring more than 1.5% ↑ in AUPRC, which has been shown to be more sensitive to imbalanced samples. Therefore, we consider these modules essential and indispensable components of our model.



**W6. The symbols or bold texts in Equation (10-12) are wrong.**

A6.  We have removed the bold text styling from the symbols in equations (10-12).
$$
r^{(t)}=\sigma(\Theta_{r} \star_{G^{(t)}} [X^{(t)}|| H^{(t-1)}] + b_{r}),
$$

$$
u^{(t)}=\sigma(\Theta_{u} \star_{G^{(t)}} [X^{(t)}|| H^{(t-1)}] + b_{u}),
$$

$$
C^{(t)}=tanh(\Theta_{C} \star_{G^{(t)}} [X^{(t)}|| (r^{(t)} \odot H^{(t-1)})] + b_{C}),
$$

Meta-review: The paper presents a novel approach to tackle a critical problem in healthcare data analysis: the irregular sampling of time series data, which is common in medical applications. The Knowledge-Empowered Dynamic Graph Network (KEDGN) leverages pre-trained language models (PLMs) to generate semantic embeddings for each variable based on their medical properties. This enables the dynamic modeling of inter-variable correlations that evolve over time. By incorporating variable-specific parameter spaces and dynamically adjusting the variable graph, the proposed model enhances the ability to capture fine-grained temporal dependencies and correlations in ISMTS, significantly outperforming existing methods. This innovative integration of textual knowledge with dynamic graph neural networks represents a meaningful contribution to the field, offering potential improvements in clinical decision-making and predictive modeling.

Furthermore, the proposed work is well-grounded in extensive experimentation on multiple healthcare datasets, showcasing the effectiveness of KEDGN against state-of-the-art methods. The detailed analyses, including an ablation study and visual exploration of the learned variable correlations, provide a comprehensive understanding of the model's capabilities and behavior. It effectively communicates the importance of modeling variable-specific and time-varying correlations in healthcare time series, a topic that is highly relevant to the machine learning and medical communities. The incorporation of medical knowledge through PLMs, combined with the novel graph-based approach, provides a robust framework for future research in irregularly sampled time series modeling.
Decision: Accept (poster)