An Iterative Self-Learning Framework for Medical Domain Generalization
Authors: Zhenbang Wu
Huaxiu Yao
David Liebovitz
Jimeng Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: healthcare
clinical predictive model
domain generalization
TL;DR: A new clinical predictive model for medical domain generalization

Abstract: Deep learning models have been widely used to assist doctors with clinical decision-making. However, these models often encounter a significant performance drop when applied to data that differs from the distribution they were trained on. This challenge is known as the domain shift problem. Existing domain generalization algorithms attempt to address this problem by assuming the availability of domain IDs and training a single model to handle all domains. However, in healthcare settings, patients can be classified into numerous latent domains, where the actual domain categorizations are unknown. Furthermore, each patient domain exhibits distinct clinical characteristics, making it sub-optimal to train a single model for all domains. To overcome these limitations, we propose SLGD, a self-learning framework that iteratively discovers decoupled domains and trains personalized classifiers for each decoupled domain. We evaluate the generalizability of SLGD across spatial and temporal data distribution shifts on two real-world public EHR datasets: eICU and MIMIC-IV. Our results show that SLGD achieves up to 11% improvement in the AUPRC score over the best baseline.

An Iterative Self-Learning Framework for Medical Domain Generalization
Zhenbang Wu
Huaxiu Yao? David M Liebovitz 3 Jimeng Sun
University of Illinois Urbana-Chann Chbama-Champaign {zw12, jimeng) iiliinosi...ii.....eiieiiiiiiiim edu 2 University of North Carolina at Chapel Hill, huaxiulcs. unc. Northwestern University, david. Liwbovitztnmnmmmmmmmmm.. org
Abstract
Deep learning models have been widely used to assist doctors with clinical decision- making. However, these models often encounter significant performance drop when applied to data that differs from the distribution they were trained on. This challenge known as the domain shift problem. Existing domain generalization algorithms attempt to address this problem by assuming the availability of domain IDs and training single model handle all domains. However, in healthcare settings, patients can be classified into numerous latent domains, where the actual domain categorizations are unknown. Furthermore, each patient domain exhibits distinct clinical characteristics, making it sub-optimal to train single model for all domains. To overcome these limitations, we propose SLDG, self-learning framework that iteratively discovers decoupled domains and trains personalized classifiers for each decoupled domain. We evaluate the generalizability of SLDG across spatial and temporal data distribution shifts two real-world public EHR datasets: eICU and MIMIC-IV. Our results show that SLDG achieves up to 11% improvement in the AUPRC score over the best baseline.
Introduction
Deep learning techniques have been increasingly popular in clinical predictive modeling with elec- tronic health records (EHRs) [12
58 2). However, these models typically assume that the training (source) data and testing (target) data share the same underlying data distribution (i.e., domain). This assumption become problematic when models are applied to domains, such as data from different hospitals or future time points [17 [59 20, (37.. In these situations, domain shifts caused by variations in patient cohorts, clinical standards, and terminology adoption can significantly degrade the model's performance.
47
This paper aims to develop clinical predictive model the source data that effectively handles potential domain shifts when applied to the target data. Domain generalization (DG) [7] methods have been widely utilized to address such problems, including techniques like domain alignment (32
24 25
45
31 51 62), meta-learning [23 27 26 5
22 29 and ensemble learning 9 42 43
However, when applied in healthcare settings, these methods encounter the following limitations:
Reliance on domain IDs. Most DG methods depend on the presence of domain IDs, which indicate
the domain which each sample belongs, to guide the model training (24 16 42 However, shown in Fig. [ patients can be divided into numerous latent domains based on features such as age, medical history, treatment, and symptoms. The actual categorization of these latent domains can be difficult to obtain and vary across different tasks [14 Consequently, existing DG methods often resort to broad domain categorizations, such as hospital or timestamp, which provide limited
information (59
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Age
Viral Infoction Fever
Fever normal
Medical
Fever probiematic
Hotury
Heart Failure Fever
Tresement
Somptor
Conventional Domain Generalization
Medical Generalization
Known domain IDs
Unknown domain IDs
Sharable characteristics ecross domains
Distinct characteristics across domains
Figure Conventional domain generalization methods typically rely on domain IDs and shared characteristics across domains train single generalized model. However, in the medical field, patients can be classified into numerous latent domains that are not directly observable. Additionally, each patient domain exhibits unique clinical characteristics, making it sub-optimal train a single model for all domains.
Attempt to train a single model. While some recent DG methods have attempted to alleviate the reliance domain labels [62 29], they try to train single model that generalizes across all domains. However, patients from different domains possess distinct characteristics and require different treatment approaches [2 [58 For example, shown in Fig. fever is considered normal symptom for patients with viral infections as it helps stimulate the immune system. On the other hand, it be bad signal for patients with cardiovascular disease, leading complications. Thus, training a single model for all domains is challenging and can lead to sub-optimal performance.
To overcome these limitations, we propose SLDG, self-learning framework for domain generalization that iteratively discovers decoupled domains and trains customized classifiers for each discovered domain. Specifically, SLDG consists of the following iterative steps:
Decoupled domain discovery. While domain labels are not initially available, we posit that they be recovered by clustering the learned latent representations. However, identifying all fine- grained domains across various clinical features (e.g., demographics, diagnosis, and treatments) can be challenging. Instead, we propose decouple these clinical features and discover the clusters separately for each type of feature. To achieve this, we maintain a distinct latent space for each type of features using feature-specific patient encoder. Within each latent space, we perform hierarchical clustering independently to discover the domain categorizations. By adopting this approach, we effectively reduce the number of domains from exponential linear the number of feature types.
Domain-specific model customization. To account for the unique characteristies of patients in different domains, our approach involves training customized classifiers for each domain. To ensure parameter efficiency, we extract domain representations from the learned clusters and utilize them to parameterize the domain-specifie classifiers. For given patient, we determine the closest domain by comparing the patient's representations with the domain representations, and subsequently select the corresponding classifier for accurate inference.
To assess the generalizability of SLDG across spatial and temporal shifts, we conduct experiments on two publicly available EHR datasets: eICU 38 and MIMIC-IV [8]. Our results demonstrate that SLDG outperforms the best baseline by up 11 % in terms of AUPRC score. We also conduct detailed analyses and ablation studies investigate the factors contributing to the performance gain achieved by SLDG.

---

Iteration
Age
Classifier
Age
Feature
Classifier
Medical History
Specific
Symptom
Prediction
Patient
Encoders
Classifier
Treatmentt
Treatmentt
Symptom
Classifier
Medical History
Source Patient Data
Decoupled Domain Discovery
Domain-Specific Model Customization
Figure 2: An illustration of the SLDG framework. The feature- speciffc patient encoder maps each patient into multiple latent spaces, with each space capturing patient characteristics from specific perspective. Next, SLDG iteratively performs decoupled clustering to identify latent domains and learns domain-specific classifiers customized for each domain.
2 Preliminaries
In EHR data, patient's hospital visit is represented by sequence of events, denoted as [e1, C2...., Cm]. where m is the total number of events in the visit. Each event characterizes features of certain type such diagnosis, prescription, and lab tests. This mapping is denoted by + T, where & and T denote the sets of all events and types, respectively. For example, a patient visit can be |Acute embolism (182. 40), Atrial fibrillation (148.91). Ultrasound (76700), CT scan (G0296), ECG (93042), Heparin IV (5224)). And each event corresponds to a specific type (e.g., diagnosis, procedure, medication). The main objective of clinical predictive modeling is to predict the occurrence of future events, such as 15-day hospital readmission and 90-day mortality, denoted as y {+, -+, based the patient's current visit z.
a function T(e)
Existing clinical prediction works typically train model f. with parameter 6 by minimizing a loss function I(-) source training data sampled from distribution Ptr, as in Eq. ①
arg min (a.,)) (z)) v)],
(I)
with the hope that the trained model can perform well on the target test data distributed according to Pte. However, in real-world settings, the source and target distributions can differ due spatial and temporal shifts, i.e., Per * Pt.. Consequently, the model trained on source data may experience a drop in performance when applied to the target data.
3 The SLDG Approach
In this paper, our goal is to train model (-) on source data Per that can generalize to target data Pte despite potential domain shifts. Existing DG algorithms face limitations due to their reliance on domain IDs and attempts to train single model for all domains. To overcome these limitations, propose to iteratively discover latent domains and train customized classifiers for each domain. However, we face the challenge of dealing with large number of latent domains, which not only makes domain discovery difficult but also results in exponential increase in the number of model parameters with respect to the number feature types. the following sections, we will describe how our method SLDG addresses this challenge through decoupled domain discovery and domain- specific model customization. Additionally, we will introduce the training and inference strategy. Fig. illlustrates the SLDG framework.
3.1 Decoupled Domain Discovery
Although domain labels are not initially available, we hypothesize that the domain information is encoded in the learned latent representations and can be recovered with the clustering technique. How-

---

ever, patients can be categorized into thousands of latent domains, determined by various features such as age, medical history, treatment, and symptoms. For instance, a patient can fall into the fine-grained domain of older male patients with a history of smoking and diagnosis of type 2 diabetes. Identi- fying all such tineggained domains can be challenging, as clustering methods may either overlook smaller domains or result in an excessive number of domains that would inflate the number of param- eters in subsequent steps. To address this, we propose decoupling these clinical features and indepen- dently discovering clusters for each feature type. For example, the patient above can simultancously belong to the decoupled domains of older, male, history of smoking, and diagnosis of type 2 diabetes. This approach effectively reduces the number of domains from exponential to linear with respect to the number of feature types.
Concretely, we maintain a distinct latent space for each type of feature. When given an input patient visit v, SLDG maps to the latent space corresponding to the feature type t E using a feature-specific patient encoder Fee.),, as in Eq. (2)
h (x),, h, ER
where h denotes the hidden dimension. Next, within each latent space of type t, SLDG gathers all patient representations (h نل where Ner the number of source training data, and performs clustering to discover the domain categorizations, as in Eq. (3)
M == Cluster((h) (1) ),T) M, E {0,1) M..K.
(3)
where K2 represents the number of discovered domains in the latent space of type M, denotes the learned domain assignment, where M,(i, k] is equal to one if and only if (i.f.f.) the patient (f()) is assigned to the k-th domain. We will describe this procedure in detail in the following.
Seature-Specific Patient Encoding. This module is responsible for mapping each patient into multiple latent spaces, each capturing the patient's health status of specific feature type. This enables subsequent modules to decouple the representations of different feature types. For a patient's s hospital visit x with a list of events (ei,... ........ em SLDG computes the contextualized representation for each event by applying the embedding function E(.), as in Eq.
[ei,..., cm E((eI....... em)), Gj E RRA
where e, is the contextualized representation for event éj with dimension h. We model E(-) using three-layer Transformer (48) framework. To ensure that there no unseen events in the target data, we initialize the event embedding look-up table with ClinicalBERT (3] embeddings of the event name and then project it down to our hidden dimension of size h. The embedding look-up table is fixed during training.
Next, SLDG aggregates the contextualized event representations [er, (e1...., eml based their types, such as family history, diagnosis, and treatments. For each type t E T, the type-specific representation h is computed by averaging the representations of all events of that type, in Eq.
h Average((e, [Tee)) = ()/=), h, E 12
(5)
where T(e:) indicates the type of event e;. If no events belong to a certain type, the pooled sequence representation is used substitute. Consequently, each patient's 'ss hospital visit is represented by set of vectors [he hee, with each vector capturing the patient's ' health status from a specific type of events. These decoupled patient representations are then utilized perform per-feature-type domain clustering, described next.
Hierarchical Domain Clustering. This module is responsible for clustering patient representations in each latent space to discover latent domains, enabling subsequent modules to customize the classifier for each domain. In the previous step, we obtain set of patient representations. (h, (t) No. R=i
for each latent space of type t. To perform clustering, standard clustering techniques such as k-Means and Gaussian Mixture Model (GMM) require specifying the number of clusters, which less ideal as the number of clusters can be difficult to choose and may vary across latent spaces. Inspired by GEORGE [46. we adopt a fully automated hierarchical clustering technique by monitoring the Silouette score 440

---

Specifically, in each latent space, SLDG first applies UMAP 30 for dimensionality reduction. Then, runs k-Means with f2.... 10] identify the optimal number of clusters based on the highest Silhouette score. Subsequently, SLDG further split each cluster into five sub-clusters. However, only sub-clusters surpassing the Silhouette score the original cluster and containing at least 500 patients are retained. The final number of clusters in the latent space of type is denoted as K₃- The cluster assignment is represented by a binary matrix M, of size Nr x Kt, where Mc(i, k] is set to one i.f.f. the patient (+() is assigned to the k-th cluster. This automated approach allows us to effectively select the number of clusters in each latent space, balancing between discovering overly coarse or fine-grained clusters.
3.2 Domain-Specific Model Customization
To accommodate the unique characteristics of patients in different domains, we propose to train customized classifiers for each decoupled domain. Given an input patient visit z and its multi-vector representations {hr her, SLDG computes the predicted probability o of a specific event occurring by employing a weighted combination of domain-specific classifiers in each latent space E T, as in Eq. (6).
E Ex K; IT IET -
1
Gro,(he)) (he( OE R.
(6)
gate
classifier
where Cr)+(-) refers to the customized classifier for the discovered domain & in the latent space of type t. while G+,x(-) corresponds the gating function. In the following, we will elaborate on how SLDG leverages the clustering results to efficiently parameterize the domain-specific classifier and effectively determine the gating weights.
To efficiently parameterize the domain-specific classifier C,,((+) for the k-th discovered domain in the latent space of type I, we define two learnable weight vectors of size h: w and Wi which represent the prototypes of the positive and negative classes, respectively. The predicted probability of specific event occurring is computed based on the relative distance between the patient representation h, and the positive and negative prototypical weights, as in Eq. 77.
exp( exp(-d(w,",. h.)) (he) E.e(t.-) exp(- d( kw h.)) (h,) E R,
where d(.;) )) is the Euclidean distance. To facilitate efficient learning, we initialize the two prototypical weight vectors w and We with the average representations of patients from the corresponding classes, as in Eq.
Init( w k.k) Average( (hh
1 =)) E (+,-),
(8)
where Ms] includes only patients assigned to the k-th domain in the latent space of type
We adopt a similar approach to the gating function G.(-). For each discovered domain the latent space of type t, we introduce learnable prototypical weight vector w,, E R², The gating weights are determined based on the distance between the patient representation h, and the corresponding prototypical weights, as in Eq. (9)
(h, Softmax (1 ((-d(wek) bo)) G.(h,) E ERK
where the prototypical weight vectors (wex)) are initialized as the average representations of patients that domain, as in Eq. (T).
Init(w w.,)) = Average ((h,) M. = 1), k= 1,.,.,,, Kr.
(10)
3.3 Training and Inference
To train SLDG, begin by utilizing pre-trained patient encoder for decoupled domain discovery. Then, we iteratively update the model weights and re-generate the clusters every 20 epochs. In each
In practice, we pre-train the patient encoder on the same clinical predictive task for 40 epochs. This means that SLDG undergoes total of 100 epochs of training, which is consistent with other baselines.
5

---

iteration, we re-initialize the classifier and gating parameters. This iterative process is repeated three times enhance the model's performance. During training, we minimize the binary cross-entropy loss. For inferencing, given a target patient visit, SLDG first maps it to multiple decoupled latent spaces with the feature-specific patient encoders (.).). Subsequently, in each latent space of type t, the gating function (-) determines the weight combinations used to aggregate the predictions from domain-specific classifiers (C),(()))= The final prediction is obtained by averaging the predictions from all latent domains. The pseudocode SLDG can be found in Appx.
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate SLDG on two publicly available real-world EHR datasets: eICU MIMIC-IV [18), which are described follows:
38 and
eICU B8 covers 200K visits for 139K patients admitted to the intensive care unit (ICU) in one of the 208 hospitals across the United States. The data was collected between 2014 and 2015. The 208 hospitals can be further categorized into four groups based their location (Midwest, Northeast, West, and South). We use age, gender, and ethnicity as patient demographic information, and leverage the diagnosis, treatment, medication, and lab tables to gather patient visit information.
MIMIC-IV 18 covers over 431K visits for 180K patients admitted to the ICU the Beth Israel Deaconess Medical Center. The data was collected between 2008 to 2019. The approximate actual year each admission revealed as one the four-year groups (2008-2010, 2011-2013, 2014-2016, and 2017-2019). We use age, gender, and ethnicity patient demographic information, and leverage diagnoses, procedures, and prescriptions to gather patient admission information.
We elaborate on the cohort selection process and provide comprehensive dataset statistics in Appx. B.I In the end, extract 149227 visits from 116075 patients in the eICU dataset, and 353238 visits from 156549 patients in the MIMIC-IV dataset.
Clinical Predictive Tasks, We focus on two common clinical predictive tasks: (1) Readmission prediction, which aims to determine whether patient will be readmitted within the next 15 days following discharge. (2) Mortality prediction, which aims to predict whether a patient will pass away upon discharge in the eICU setting, within 90 days after discharge in the MIMIC-IV setting. A detailed explanation of this setting can be found in Appx. B.2
Data Split. We evaluate the performance of our model across spatial gaps using the eICU dataset. For this purpose, we select the target testing data the group (Midwest) that demonstrated the largest performance gap in pilot study. The remaining groups (Northeast, West, and South) are used the source training data. To assess the model's performance across temporal gaps, utilize the MIMIC-IV dataset. Patients admitted after 2014 are used as the target testing data, while all preceding patients are included in the source training data. We elaborate more the data split in Appx. B.3
Baselines. We compare SLDG against three categories baselines. (1) The first category consists of naive baselines, including Oracle, trained directly on the target data, and Base, trained solely on the source data, (2) The second category comprises DG methods that require domain IDs. These include DANN [16] and MLDG [22), which coarse regional and temporal groups as the domain definition, and ManyDG (56), which treats each patient as a unique domain. (3) The last category
consists of DG methods that do not rely on domain IDs, including IRM [4, MMLD [29]. and
DRA 14] detailed explanation of all the baselines can be found in Appx. B.4
Evaluation Metries. Both readmission prediction and mortality prediction are binary classification tasks. To evaluate the performance of the models, we calculate the Area Under the Precision-Reca. Curve (AUPRC) and the Area Under the Receiver Operating Characteristic Curve (AUROC) scores. For each metric, report the average scores and standard deviation by performing bootstrapping

---

Table 1: Results of domain generalization on the eICU and MIMIC-IV datasets. An asterisk (*) indicates that SLDG achieves a significant improvement over the best baseline method, with p-value smaller than 0.05. The experimental results demonstrate that SLDG exhibits robustness against spatial (elCU) and temporal (MIMIC-IV) domain shifts.
eICU
MIMIC-IV
Method
Readmission
Mortality
Readmission
Mortality
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
Oracle
0.219 (0.01) 0.677 (0.01) 0.271 (0.01) 0.839 (0.01)
0.282 (0.01)
0.693 (0.00)
0.428 (0.00) 0.898 (0.01)
Base
0.104 (0.02)
(0.00) 0.230 (0.01) 0.803 (0.01)
0.237 (0.01)
0.665 (0.01)
0.374 (0.01)
0.861 (0.00)
DANN
0.135 (0.01) 0.538 (0.01) 0.245 (0.01)
0.808 (0.01)
0.247 (0.01)
0.673 (0.01)
0.380 (0.02)
0.873 (0.02)
MLDG
0.104 (0.01) 0.525 (0.01) 0.224 (0.01)
0.797 (0.01)
0.205 (0.01)
0.637 (0.02)
0.360 (0:01)
0.857 (0.01)
ManyDG
0,150 (0.01)
(0.01) 0.259 (0.01)
0.814 (0.01)
0.249 (0.01)
0.676 (0.01) 0.388 (0.01)
0,880 (0.01)
IRM
0.136 (0.01) 0.538 (0.01) 0.252 (0.02)
0,811 (0.01)
0.242 (0.00)
0.668 (0.01)
0.387 (0.01)
0.876 (0.01)
MMLD
0.167 (0.01) 0.578 (0.00) 0.256
0,818 (0.01)
0,250 (0.02)
0.679 (0.01)
0.393 (0.01)
0.887 (0.01)
DRA
0.148 (0.01) 0.551 (0.01) 0.249 (0.01) 0.810 (0.01)
(0.01)
0.670 (0.01) 0.387 (0.01)
0.875 (0.01)
SLDG
0.186 (0.01)" 0.623 (0.01)* 0.268 (0.01)* 0.824 (0.01)"
0.274 (0.01)*
0.690 (0.01)"
0.416 (0.00)*
0.899 (0.01)*
1000 times. Additionally, we conduct independent two-sample t-tests to assess whether SLDG achieves a significant improvement over the baseline methods.
the same Transformer 48 architecture as the backbone encoder. Patient demographics features (age, gender, and ethnicity) are embedded with an embedding look-up table. We also embed the timestamps with sinusoidal positional encoding. The medical, patient demographics, and temporal embeddings are added together to form the overall sequence embedding. All models are trained for 100 epochs, and the best model is selected based on the AUPRC score monitored on the source validation set. For SLDG, UMAP [30] from UMAP- learn 41 is used with components, 10 neighbors, and 0 minimum distance; and k-Means from Scikit-learn (35) is used with the default aaperr-arramter.. Further information regarding the detailed implementations can found in Appx. B.5
Implementation Details. For all baselines,
4.2 Main Results
Table 1 presents the domain generalization results the eICU (38 and MIMIC-IV [8] datasets. Firstly, we observe a significant performance gap between the Oracle and Base methods, indicating the presence of substantial spatial and temporal domain gaps. This supports the of the DG setting. Notably, the readmission tasks exhibit larger domain gaps, which is reasonable since hospitals across different locations and timestamps may have varying criteria for patient readmission. Secondly, we note that the two DG methods, DANN [66] and MLDG (22). utilizing coarse domain partitions such region and timestamp, achieve minimal
improvements. This outcome is expected because the domain partitions are too coarse, making it challenging to identify consistent domain features. In comparison, Ma (56) achieves better performance by considering each individual patient as a unique domain. Among the remaining three baseline methods that do not rely on domain IDs, IRM 4 demonstrates the slightest improvement. DRA [14] performs better due to the usage of multi-head networks, which share a similar intuition as SLDG. MMLD (29 attains the highest performance among all baselines, showcasing the advantages of explicit domain discovery. Lastly, SLDG outperforms basclines for all tasks. Specifically, in terms of the AUPRC score, SLDG achieves an 11% relative improvement in eICU readmission prediction. 3% in eICU mortality prediction, 10% in MIMIC-IV readmission prediction, and 6% in MIMIC-IV mortality prediction.
4.3 Quantitative Analysis
This section provides quantitative analyses to elucidate the performance enhancements achieved by SLDG. The analyses encompass the evaluation of clustering results, ablation studies on the clustering algorithm, the impact of the number of clusters and iterations, and a runtime comparison.
7

---

Evaluation of clustering results. First, we evaluate the domain recovery ability of DG methods that do not rely on domain IDs, namely MMLD [29]. DRA [14,, and the proposed SLDG.
0.30
Sihooutte Score
0.25
Since the actual latent domain categorizations are unavailable, we assess the separability of the learned clustering results with the Silhouette score [40]. Note that the reported Silhouette score is calculated on the testing set, while the hyper-paramene parameters are chosen based on the Silhouette score on the training set. As depicted in Fig. 3 DRA achieves the lowest score, which aligns with expectations as it solely learns latent domain categorizations through multi-head networks without explicit clustering. In contrast, MMLD generates more distinct clusters due its iterative clustering and training setup. However, it still necessitates manual specification of the number of clusters. In comparison, SLDG obtains the highest score using an automated hierarchical clustering technique.
0.20
0:15
MMLD
0.10
DRA
SLDG
0.05
eICU Readmission
eICU Mortality
MIMIC-IV
MIMIC-IV Mortality
Readmission
Figure 3:
Performance of latent domain cluster- ing. higher Silhouette score indicates improved cluster separability.
Influence of the clustering algorithm. Next, assess the influence of different clustering algorithms. Naive k-Means and GMM require manual specification the number of clusters, which set to the same value as SLDG's. The results can be found in Tab. 2 We observe that naive k-Means and GMM achieve similar perform similarly to the best baseline methods in Tab. [ This outcome is reasonable since the success of SLDG relies on both the accurate discovery of latent domains and customized models for each domain. Naive clustering techniques often fail to identify subtle yet important latent domains. In contrast, SLDG, utilizing the automatic hierarchical clustering technique, achieves the highest score.
Table 2: Ablation study on the influence of the clustering algorithm.
eCCU
MIMIC-IV
Method
Readmission
Mortality
Readmission
Mortality
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
SLDG k-Means
0,148 (0.01) 0.553 (0.01) 0.249 (0.01) 0.814 (0.00)
0.250 (0.01) 0.670 (0.01)
0.388 (0.01) 0.886 (0.01)
SI.DG+ + GMM
0.143 (0.01) 0.549 (0.01)
0.240 (0.01)
0.808 (0.01)
0.250 (0.01)
0.688 (0.01)
0.390 (0.00)
0.888 (0.00)
SLDG
0.116 (0.01)) 0.623 (0.01)" 0.268 (0.01)* 0.824 (0.01)* 0.274 (0.01)* 0.690 (0.01)* 0.416 (0.01)* 0.899 (0.01)*
Influence of the number of clusters and iterations. Next, we amalyze the impact of the number of clusters and iterations on the eICU readmission
0.20
prediction task. We also compare our results with
AUPRC
0.18
the best-performing baseline, MMLD [29]. The results can be found in Fig. The upper panel of the figure shows that the model's 's performance initially improves with increasing number of clusters. This improvement can be attributed to the finer granularity of clustering, which enables better identification of domains and customization of experts. However, as the number of clusters continues to increase, the model's 's performance starts to decline. This decline is caused by the growing number of model parameters, making training more challenging and leading to overfitting on suspicious samples. Similarly, the trend observed in the lower panel of the figure for the number of iterations aligns with the number of clusters. The performance initially improves
0.16
40
50
Number er clusters
AUPRC
0.18
0.16
0.14
SLDG
MMLD
Number of itterations
Figure 4: Results on the in influence of the num- ber of clusters and iterations.
8

---

the number of iterations increases, allowing the model to learn more from the data. However, after certain point, the model starts overfitting on specific clusters, leading decreased performance.
Runtime comparison. Lastly, compare the training time of SLDG with the naive Base baseline. All runtimes are measured on single NVIDIA A6000 GPU. The results can be found in Tab. 3 The use of the UMAP 33 dimensionality reduction technique enables SLDG to perform clustering quickly. As result, the training time overhead of SLDG is reasonably low overall (18% on eICU and 20% on MIMIC- IV) compared the significant performance improvement achieved (up to 79% relative improvement on AUPRC score on eICU and up 15% on MIMIC-IV).
Table 3: Runtime comparison.
Method
eICU
MIMIC-IV
Basc SLDG
67 min
93 min
79 min
112 min
4.4
Case Study
Latent Space
Frequent Clinical Events from the Top-2 Clusters
Rena Failure
Trauma
Diagnosis
Sepsis, Infection, Kidney failure
Cardiac
Arrast
Cardiac Arrest, Congestive Heart Failure
Preummniia
Treatment
Heart valve procedures, Cardiovascular monitoring
Sepsis
Gastroinfestinal Bleed
Ventilation, Respiratory intubation
Medication
Propofal, Lorazepam, Fentanyl
Infarction
Diabetes
Amiodarone, Noradrenaline
Stroke
Figure 5: Left: Common clinical events observed in two largest domains identified from each latent
space. Right: Learned similarity among the identified domains within the diagnostic latent space.
For the case study, we first examine the recovered domains within each latent space. The results are presented in the left panel of Fig. 5 is evident that the common events identified within the same domains are consistent. For instance, propofol, lorazepam, and fentanyl is frequently used together, serving the purpose of anesthesia (pre-surgery), sedation (in-surgery), and pain management (post-surgery). Furthermore, the right panel of Figure 5 illustrates visualization of the learned domain similarity, i.e., the distances between the domain prototypical weights {Wr,,) k- Notably, our method (SLDG) successfully captures meaningful relationships among the latent domains. For instance, strong relationship is learned between pneumonia and sepsis. In clinical practice, when pneumonia is severe or the infection spreads beyond the lungs, it can enter the bloodstream and trigger systemic response, leading sepsis [8). Another example is the observed strong association between cardiac arrest and infarction. In practice, if a significant portion of the heart muscle is damaged, it can disrupt the heart's 's electrical system, potentially leading to cardiac arrest [9]
K.
5 Related Work
Domain Generalization The goal of DG is to learn model using data from multiple source domains in order to achieve effective generalization to distinct target domain (7] To achieve this, domain alignment approaches try to match the feature distributions among multiple source domains
with techniques such moments minimization [32 25 contrastive learning [31], adversarial
learning 25 45), regularizers 4 244,, and augmentation
60]. Meta-learning frameworks
have also been utilized to simulate new domain scenarios during training 223 27
26
5
22
29].
Additionally. -speifc model ensemble techniques have been employed 9
42
43
However, these conventional DG methods assume the availability of domain IDs, which may not be feasible in healthcare settings where patients can belong to numerous unobserved domains.
Recent advancements in DG have attempted to alleviate the reliance on domain IDs 29 62 10
33
MMLD [29] the most relevant prior work to ours. simultaneously discovers latent domains and learns domain-invarian features through adversarial learning. However, it focuses on training

---

single model that generalizes across all domains. Given that patients from different domains exhibit distinct characteristics and require different treatment approaches [2 58), training a single model for all domains poses challenges and can result in sub-optimal performance.
Clinical Predictive Modeling. The main objective of clinical predictive modeling is to predict the occurrence of future events, such 15-day hospital readmission and 90-day mortality, based on existing patient information. Deep learning models have been widely used in clinical predictive modeling with EHR data [55 577. These models are designed capture temporal patterns patient data
36 6 28 model structural information in medical codes [3] 52), augment the model using pre- pre-training [39,, or leverage patient similarities for better decision making [5. However, these models typically assume an unchanged test domain and may suffer from degraded performance with
domain shift. To address this issue, AutoMap [53] solves the feature space shift issue by learning
an auto-mane mapping function without considering any distribution shift. Medl .ink [54] aggregates
de-identified patient data from different sites enable joint training. ManyDG [56] tackles patient
covariate shift by treating each patient unique domain and disentangling domain variant and
invariant features. However, maintaining a large number of domains is unnecessary, as similar patients often exhibit similar clinical behavior and can share common domain.
6 Conclusion
Clinical predictive models often exhibit degraded performance when applied to data from new regions or future periods due distribution shifts. To address this, we propose SLDG, self-learning framework that iteratively identifies decoupled domains and trains customized classifiers for each domain. We evaluate SLDG two medical datasets, and our results show that it outperforms all baseline methods. In addition, we provide detailed qualitative analyses and case studies to support our findings.
Acknowledgments
This work was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479. This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center.
10

---

References
[1] Davies Adeloye, Stephen Chua, Chinwei Lee, Catriona Basquill, Angeliki Papana, Evropi Theodoratou, Harish Nair, Danijela Gasevic, Devi Sridhar, Harry Campbell, Kit Yee Chan, Aziz Sheikh, and Igor Rudan. Global and regional estimates of COPD prevalence: Systematic review and meta-analysis. Journal of global health, 5(2):020415, December 2015. ISSN 2047-2978 2047-2986. doi: 10. 10.71 10.7189/jogh.05.00 05.020455.
Place: Scotland.
[2] Henrique Aguiar, Mauro Santos, Peter Watkinson, and Tingting Zhu. Learning of cluster-basee feature importance for electronic health record time-series. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 161-179. PMLR, 17-23 Jul 2022. URL httlhtttp:://prceeeeli mlr A.r.press/vie2. ppress//162//gulaa2222 agwiar htm
[3] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota. USA. June 2019. Association for Computational Linguistics. doi: 10.18653/vL/119-11
10te//1911900 1909
[4] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[5] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularivattonn In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc., 2018. URL http:ttts:////oceeeiiiii neur .ct.cu/ppccc////// paper_filese
[6] Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou. Patient subtyping via time-aware 1stm networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 65-74, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. 0097983.3097997, URL htt.hhttts///o..... org/ 1145/3097983 7444//3399983330999977
doi: 10.1145/
[7] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to new unlabeled sample. In J. TaylerTaylor R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL /ttpbttps///rrrrcedinnn.ccc http://igg...rriippppppcaapppppe files/ [pperr//A./.tiina/eeeeeeeeeeeeer
[8] Klaus F. Bodmann. Current guidelines for the treatment of severe pneumonia and sepsis. Chemotherapy, 51(5):227-333, Aug 2005, doi: 10. 10.1159/000087452. URL https://doi org/10. 1111/119900000 00159//00087452
[9] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22405-22415 Curran Associates, 2021, URL https: httpps://proceeedingg. wwwpeediiiinnnneuuunne neurips. cc/ cct wwwppppppp../// filerffilllppeeee boca.cccccBB666@aa1177766622220
Saaperr.fieee/pppp//////////// http:/paapeee/////pp//2 .df
[10] Zhihong Chen, Taiping Yao, Kekai Sheng, Shouhong Ding, Ying Tai, Jilin Li, Feiyue Huang, and Xinyu Jin. Generalizable Representation Learning for Mixture Domain Face Anti-Spoofing. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2): May 2021. ISSN 2374-3468, 2159-5399. doi: 10. 10..169/aaai..355..1119999 URL httptttts//:saaaaat org/
index.
[11] Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 3512-3520, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978 10033888.
11

---

[12] Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informaties Association, (4(2):3613370 08 2016. ISSN 1067-5027. doi: 10.1093/jmia/ocw 12. URL https://doi..rge/ 10. 100///00.000//11 1tt00///amm////11111//
[13] Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. Gram: Graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD *17, page 787-795, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874.
doi: 10. (0.1144/3097983.. 3098719 URL. httpt::ttpt.o...coo /rgp:////ooooo//.../. org/10. 1145/
3097983. 3098963 3098126
[14] Lucas Deecke, Timothy Hospedales, and Hakan Bilen. Latent domain learning with dynamic residual adapters, 2020.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors. Proceed- ings of the 34th International Conference on Machine Learning. volume 70 of Proceed- ings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017. URL nttps://proceeddin: wttp://innn..m...eeeee/////////e
mlr
html
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain- aaveriiraaver training of neural networks. J. Mach. Learn. Res., 17(1):2096- 2030, jan 2016. ISSN 1532-4435.
[17] Lin Lawrence Guo, Stephen Pfohl, Jason Fries, Alistair E. W. Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific Reports, 12(1):2726, February 2022. ISSN 2045- doi: 10. 100100031155555 10044459--222 022-06484-1 1. URL
httpp://wwwt http:://wwww.aauur.eemm nature. www.uureeccmmeeoo wwwom//rticlee///oooooooooaeeelt 21/////4111992-04-444802 06484464441
[18] Alistair W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, freely accessible electronic health record dataset. Scientific Data, 10(1):1, January 2023. ISSN 2052-4463. doi: 10. 11.1088/41197-022--1189999 URL httt[ht/s://dooi org/ 10. /038//4159971111 022-11557--222288---
[19] Kim, F. So, Kim, K. S. Seo, and M. Karm. Cardiac arrest due to an unexpected acute myocardial infarction during head and neck surgery: A case report. J Dent Anesth Pain Med. 18(1):57-64, 64, 2018. doi: 10. (l.1727/ddapm.2000 2018. 18.1 URL /attss:/::/// .rg 10. 17245/
1tapm.2018 18. 1.57
[20] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of wn-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR, 18-24 Jul 2021. URL https://proceediing
mlr wttpss/vee//eeeeeeeeeeeee 1119///hh/aa..... html
[21] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization, 2017.
[22] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization, In Proceedings of the Thirty-Secone AAAI Confer- ence on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI 18/AAA 18/EAAI 18. AAAI Press, 2018 ISBN 978-1-57735-800-
[23] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), October 2019.
12

---

[24] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
[25] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 00.160//aaivv321.111.. URL htttttt////sss. aaai
org/index: php/
[26] Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3915-3924, PMLR, 09-15 Jun 2019. URL https::/prroceedingg mlr wwwppppeeer//.p..
[27] Quande Liu, Qi Dou, and Pheng-Am Ahnn Heng. Shape-award meta-learning for generalizing prostate mri segmentation to unseen domains. In Anne L.. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, Kevin Zhou, Daniel Racoceanu, and Leo Joskowiez, editors, Medical Image Computing and Computer Assisted Intervention - MICCA/ 2020, pages 475-485, Cham, 2020. Springer International Publishing. ISBN 978-3-030-59713- 9.
[28] Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. Dipole: Diagnosis prediction in healthcare via attention- bidirectional recurrent neural networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1903-1911, Computing Machinery. ISBN 978 4503488774
New York, NY, USA, 2017. Association for doi: 10. 10.1145/3399988333999608 3098008. URIhttpss
//doi.org/10. 1095//199///39883111 309889
[29] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07))11749) 1 11756, Apr. 2020. doi: 10. 10.1609/aaai. wtwaaaai.i3i4...... wtt.v33341.44444 URL https: aaai org /egg/indxx
[30] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018
[31] Saeid Motiian, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[32] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invari- ant feature representation. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' 13, page 1-10-1-18. JMLR.org, 2013.
[33] Li Niu, Wen Li, and Dong Xu. Visual recognition by learning from web data: weakly supervised domain generalization approach. In 2015 IEEE Conference on Computer Vivion and Pattern Recognition (CVPR), pages 2774-2783, Boston, MA, USA, June 2015. IEEE. ISBN URL http:://eeeeeppporre
978- 978-1-4673-6964400 doi: 20L.10//CVP.2201....72.
org//docueent/22988///
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL prR_httpp//pppprs.eerrppss wwwripe...... cc/paper/rraoaaec 9015 111----ytorc------e-ea- an-imperative wtiheeessseeee---e-eeeeeeeeeeeee hwweeeehhhhhhheeeeeeee wtwp:/orrnnnne-eeee---eeeaaaeeee -inaannnnnnniaarrry pdf
[35] Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Seikit-learn: Machine learning Python, Journal of Machine Learning Research, 12:2825-2838 201
13

---

[36] X. Peng, G. Long. T. Shen, S. Wang, J. Jiang, and C. Zhang. Bitenet: Bidirectional temporal encoder network predict medical outcomes. In 2020 IEEE International Conference on Data Mining (ICDM), pages 412-421, , Los Alamitos, CA. USA, nov 2020. IEEE Computer Society. doi: 10.1 100000000 2020.00050. Utt. httptpppe//b..ieeecccttteeeeeeee wewecoopttersociitt........ 10. 10.11099 1101/19///00 MW..0//IDDM0008.... 2020.00050
[37]
Christian S. Perone, Pedro Ballester, Rodrigo C. Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with seff-ensembling. Neurolmage, 194: URL
1-11. 2019. ISSN 1053-81 8 19. doi:
wttp://wwww.m http://wwwwaaaacccooooo//oooaoaa
[38] Tom J. Pollard, Alistair W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, freely available multi-center database for critical care research. Scientific Data, 5(1):180178, 78, September 2018. ISSN 2052-4463. doi: 1030//dataa.....a sdata. 2018. 2018.178
1038/sdata 8. 178. URI. htt.httss://do.oooo org/ 1038/
[39] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextu- alized embeddings on large-scale structured electronic health records for discase prediction. npj Digital Medicine, 4(1):86, May 2021. ISSN 2398-6352. doi: 10. 10.1008/44414460011-0445554
URL https:://oo.... org 10. 1038 s4174 14144 021 00455-y
[40] Peter J. Rousseeuw. Silhouettes: graphical aid the interpretation and validation of cluster
Journal of Computationnl and Applied Mathemulics, 20:53-65, 1987. ISSN 0377-0427. Journul doi: http:://bbboogr//000066607777/71
URL https://::/// //wp///wkk.ooo scienedirrectt
Com/science//nan /tt.a//c/err.ccee/////////777772
[41] Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for
representation and semisupervised learning. Neural Computation, (3(11))2881-18 2021.
[42] Mattia Segu, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. Pattern Recognition. 135:109115, 2023. ISSN 0031-3203. doi:
URL https: /wwp::///wwwwww//./// www. www.eeedreeee.ecccommm. com/
[43] Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning optimize domain specific normalization for domain generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Mac-Mil Frahm, editors, Computer Vision ECCV 2020, pages 68-83. Cham, 2020. Springer International Publishing. ISBN 978-3-030- 58542-6.
[44] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri. Precthi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In International Conference on Learning Representations, 2018. URL httpss///ppperreiiee wwwpreeiiiiieene/fforreee net/forum?
rdx1fXffbWWW
[45] Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C. Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[46] Nimit Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-gru Fine-grained robustness in cearse-grained classification problems. In Proceedings of the 34th International Conference on Neural Information P Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc, ISBN 9781713829546.
[47] Yanchao Tan, Chengjun Kong, Leisheng Yu, Pan Li, Chaochao Chen, Xiaolin Zheng, Vicki Hertzberg, and Carl Yang. 4sdrug: Symptom-based set-to-set small and safe drug recom- mendation. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD '22, page 3970-3980, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450033850. https:://ooi org/10 /
doi: 10.1 10.1145/3534678.555 3539998 URL
14

---

[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 0781510860964
[49] Claus F. Vogelmeier, Kenneth R. Chapman, Marc Miravitlles, Nicolas Roche, Jørgen Vestbo, Chau Thach, Donald Banerji, Robert Fogel, Francesco Patalano, Petter Olsson, Konstantinos Kostikas, and Jadwiga A. Wedzicha. Exacerbation heterogeneity in COPD: subgroup analyses from the FLAME study. International journal of chronic obstructive pulmonary disease, 13: 1125-1134, 2018. ISSN 1178-2005 1176-9106. Zealand.
doi: 10.2147/COPD.S160D) Place: New
[50] Riccardo Volpi, Hongscok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL URL/atttss//prrcceddnng neurips cc wappappr_... paaer_fflest wwwes/llaaaae httppppppp/e/////////// pdf
[51] Ziqi Wang, Marco Loog, and Jan van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 9756-9763, 2021. doi: 10.1 109111C99488 2021
2021.9417777 .9412797.
[52] Zhenbang Wu, Huaxiu Yao, Zhe Su, David M Liebovitz, Lucas M Glass, James Zou, Chelsea Finn, and Jimeng Sun. Knowledge-driven new drug recommendation, 2022.
[53] Zhenbang Wu, Cao Xiao, Lucas M. Glass, David M. Liehovitz, and Jimeng Sun. Automap: Automatic medical code mapping for clinical prediction model deployment. In Massih- Reza Amini, Stéphane Canu, Asja Fischer, Tias Guns, Petra Kralj Novak, and Grigorios Tsoumakas, editors, Machine Learning and Knowledge Discovery in Databases, pages 505-520, Cham, 2023. Springer International Publishing. ISBN 978-3-031-263900
[54] Zhenbang Wu, Cao Xiao, and Jimeng Sun. Medlink: De-identified patient health record linkage. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 2672-2662 New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030.
doi: 10. 10.11445/388005... 35994477. URL https
//doi.org/10. 00845/5553335... 3599427
[55] Chao Xiao, Edward Choi, and Jimeng Sun. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics Association, 25(10):1419-1428, Oct 2018. doi: 10.1093/jamia/ ocy068.
[56] Chaoqi Yang, M Brandon Westover, and Jimeng Sun. ManyDG: Many-domain generalization for healthcare applications. The Eleventh International Conference on Learning Representations, 2023. URL /tttpss//openree
net htttt/oormmidd=lsst www.cSfirnllpp.
[57] Chaoqi Yang, Zhenbang Wu, Patrick Jiang. Zhen Lin, Junyi Gao, Benjamin P. Danek, and Jimeng Sun. Pyhealth: A deep learning toolkit for healthcare applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23. page 7788-7789, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305, 3599178. URL http::://paoo org/ 10. 1145/ 3580805 3599178
[58] Chaohe Zhang, Xin Gao, Liantao Ma. Yasha Wang. Jiangtao Wang, and Wen Tang. Grasp: Generic framework for health status representation learning based on incorporating knowledge from similar patients. Proceedings of the AAAI Conference Artificial Intelligence, 35(1): 715-723, May 2021. doi: 10. (0.1609/aaii.v35// 13511.1615... URL http:://t/////// aaai.
www.aarr/aaaa.aaaaa....... orgg /indexxxxx
hhtp/AAA//articlle//ii
[59] Haoran Zhang, Natalie Dullerud, Laleh Seyyed-Kalantee Quaid Morris, Shalmali Joshi, and Marzych Ghassemi. An empirical framework for domain generalization in clinical settings. In
15

---

Proceedings of the Conference Health, Inference, and Learning, pages 279-290, Virtual Event USA, April 2021. ACM. ISBN 978- 778-1-4503-83594 doi: 10.1 0..144/3450049934511777 https://:: acm
0ttp:///0....111/4554544999999 3451188
[60] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Viston ECCV 2020, pages 561-578, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58517-4.
[61] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions Image Processing, 30:8008-8018, 2021. doi: 10. 10.1109/T1P:021.3111
[62] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang, Domain generalization with mixstyle. In International Conference on Learning Representations, 2021. URL https:///peeereiiie (attoorrmmmddxxi33331313
16

---

Contents of Appendix
Pseudocode of SLDG
18
Additional Experimental Setup
18
Datasets
18
B
Clinical Predictive Tasks
18
B.3
Data Split
19
Baselines
20
B.5
Implementation Details
20
Limitations and Broader Impacts
20
D Notations
21
Additional Illustrations
21
17

---

A Pseudocode of SLDG
Algorithm 1: Training and Inference for SLDG.
1: II Training
Require: Source training data from Ptr
2: Pre-train patient encoder E-) on the same task with binary cross-entropy loss for 40 epochs
3: for iteration ranging from 1 to do
4: Perform decoupled domain discovery with the encoder E(-) by Eq
5: Initialize gating and classifier weights We, Wie We. by Eq.
10
8
6: for epoch ranging from 1 to 20 do
7:
for each patient (Ir y) ~ Ptr do
8:
Obtain decoupled patient representations (ET by Eq
9:
Compute domain-specific predictions C₁ (h,)(h;) by Eq.
Compute gating weights G; by Eq. (
11:
Obtain final prediction by Eq. (6)
Update model parameters with binary cross-entropy loss
13: end for
end for
15: end for
Inference
Require: Target testing data from Pte
17:
for each patient (x, ~ PLE do
18:
Obtain decoupled patient representations (he by Eq. 4).
19:
Compute domain-speciffi predictions (h,) by Eq. 7
20: Compute gating weights G,,),) by Eq. (9)
21: Obtain final prediction o by Eq. (6)
22: end for
B Additional Experimental Setup
B.1
Datasets
For both datasets, select our cohorts by filtering out visits of patients younger than 18 older than 89 years old, visits that last longer than 10 days, and visits with data from less than or more than 256 timestamps. In the case of the eICU dataset, additionally exclude visits lasting shorter than 12 hours, as the predictions are made 12 hours after admission. Similarly, for the MIMIC-IV dataset, we exclude visits where the patient ultimately passed away, as the predictions are made upon discharge. Tab, 4 provides detailed statistics of the two datasets.
B.2 Clinical Predictive Tasks
We focus on two common clinical predictive tasks: readmission prediction and mortality prediction.
In the case of the eICU dataset, the predictions prediction aims to determine whether a patient will be readmitted within the next 15 days following discharge. Mortality prediction, on the other hand, aims to predict whether patient will pass away upon discharge. The overall prevalence for these tasks is 15% for readmission and 4% for mortality.
made 12 hours after admission. Readmission
For the MIMIC-IV dataset, the predictions are made at the time of discharge. Similar to the eICU dataset, the readmission prediction task is defined as predicting whether a patient will be readmitted within 15 days after discharge. To prevent information leakage, the mortality prediction task for MIMIC-IV is defined as predicting whether patient will pass away within 90 days after discharge. The overall prevalence for these tasks is 14% for readmission and 4% for mortality.
18

---

Table 4: Dataset statistics.
Item
eCCU
MIMIC-IV
APatients
116075
156549
#Admissions
149227
353238
Readmission Rate
0.15
0.14
Mortality Rate
0.04
0.04
Region: Midwest
Year: 2008-2010
#Patients
29767
37328
#Admissions
35989
56433
Readmissiou
Mortality Rate
0.03
0.04
Age
56
Gender
F: 0.46, M: 0.54
F: 0.53, M: 0.47
Race
American: 0.09, Asian: 0.01, Caucasian: 0.83,
African American: 0.15, Asian: 0.03, Caucasian: 0.71,
Hispanic: 0.01, Native American: 0.01. Other: 0.04
Hispanic: 0.06, Native American: 0.00, Other: 0.04 31.87
Average #Events
90.01
Region: Northeast
Year: 2011-2013
#Patients
5886
39125
#Admissions
6958
62586
Readmission Rate
0.17
0.15
Mortality Rate
0.06
0.04
Age
62
57
Gender
F: 0.44, M: 0.56
F: 0.53, M. 0.47
Race
African
Caucasian: 0.92,
African American:
Caucasian: 0.66,
Hispanic: 0.01, Native American: 0.00, Other: 0.03
Hispanic: 0.07, Native American: 0.00, Other: 0.07
Average #Events
104.54
Region: South
Year: 2014-2016
#Patients
27584
41737
#Admissions
33033
64592
Readmission Rate
0.11
0,14
Mortality Rate
0.04
0.04
Age
62
57
Gender
F: 0.46, M: 0.54
F: 0.52, M: 0.48
Race
African American: 0.21, Asian: 0.01, Cancasian: 0.68,
African American: 0.17, Asian: 0.04, Caucasian: 0.66,
Hispanic: 0.05, Native American: 0.00, Other: 0.03 84.28
Hispanic: 0.06, Native American: 0.00, Other: 0.07 36.53
Average #Events
Region: West
Year: 2017-2019
#Patients
17670
40496
#Admissions
19803
62654
Readmission Rate
0.29
0.14
Mortality
0.04
Age
63
58
Gender
F: 0.45, M: 0.55
F: 0.52, M: 0.48
Race
African American: 0.05, Asian: 0.03, Caucasian: 0.77,
African American: 0.17, Asian: 0.04, Caucasian: 0.65,
Hispanic: 0.05, Native American: 0.02, Other: 0.08 85.29
Hispanic: 0.06, Native American: 0.00, Other: 0.07 36.95
Average #Events
B.3
Data Split
The eICU dataset comprises data collected from hospitals across the United States, while the MIMIC- IV dataset spans period of ten years. Therefore, utilize the eICU dataset evaluate the model's performance across spatial gaps, and the MIMIC-IV dataset to assess its performance across temporal gaps.
For the eICU dataset, we divide it into four spatial groups based on regions: Midwest, Northeast, West, and South. Each group is then split into 70% for training, 10% for validation, and 20% for testing. We evaluate the gap between groups by comparing the performance of the backbone model trained on data from within the same group and data from outside the group. The target testing data selected as the group (Midwest) that exhibits the largest performance gap, while the remaining groups (Northeast, West, and South) are used as the source training data.
Regarding the MIMIC-IV dataset, we divide into four temporal groups: 2008-2010, 201 1-2013, 2014-2016 and 2017-2019. Each group is further split into training, validation, and testing sets with a ratio of 70%, 10%, and 20% respectively. We consider patients admitted after 2014 as the target testing data, while all preceding patients are included in the source training data.
19

---

B.4 Baselines
We first compare SLDG to two naive baselines.
Oracle:
We directly train a backbone model on the training set of the target domain, select the best model on the target validation set, and evaluate its performance on the target testing set. This model is trained with in-domain data and can be seen upper bound for all domain generalization method.
Base: We train backbone model on the training set of the source domain, select the best model on the source validation set, and evaluate its performance on the target testing set. This model is trained with out-domain data and should act as a performance lower bound.
We then compare SLDG to both classic and recent domain generalization methods. For fair compar- isons, all the methods below are trained on the source training set, selected on the source validation set, and tested on the target testing set.
DANN [6]
Domaiin-Avvrsaae Adversarial Neural Networks leverage a domain classifier and a gradient reversal layer to extract domain-invann invariant representations. This method uses the coarse regional and temporal groups as the domain definition.
MLDG [22]: Meta-Learning for Domain Generalization adopts the Aonel-Agnostic Meta-Learning (MAML) 15] framework and simulates the new domain scenario during training. This method also uses the coarse regional and temporal groups as the domain definition.
ManyDG [5)] Many-Domant Generalization disentangles domain variant and invariant features through mutual reconstruction and orthogonal projection. This method treats each patient as unique domain.
IRM Invariant Risk Minimization learns domain-invarian representations by minimizing bound on the expected generalization error under domain shifts. It acts as regularizer and does not require domain IDs.
Domain Generalization using Mixture of Multiple Latent Domains iteratively assigns pseudo domain labels via clustering and trains the domain-invariant feature extractor through adversarial learning. This method does not rely on domain IDs.
MMLD [29]:
Latent Domain Learning with Dynamic Residual Adapters uses layer-wise multi-head correction networks with a gating mechanism and residual connection to enhance model learning. This method does not rely on domain IDs.
DRA 14:
B.5
Implementation Details
For all baselines, we use the Transformer as the backbone encoder. The number of layers is 3, the embedding dimension is 128, the number of attention heads is 2. The event embedding look-up table is initialized with ClinicalBERT 3 embeddings of the event name and then project down to 128 dimension with a linear layer. Patient demographics features (age, gender, and ethnicity) are separately embeded with another embedding look-up table. We also embed the timestamps with sinusoidal positional encoding. The medical, patient demographics, and temporal embeddings are added together to form the overall sequence embedding. For SLDG, UMAP [30] from UMAP- learn [41] is used with 2 components, 10 neighbors, and O minimum distance; and k-Means from Scikit-learn (35] is used with the default parameer We apply a dropout of rate 0.2. We use Adam the optimizer with learning rate of le-4 and weight decay of le-5. All models trained for 100 epochs. The batch size is 256. We select the best model by monitoring the AUPRC score on the source validation set (except for the Oracle baseline, where directly use the target validation set). We implement SLDG using Py (34 1.11 and Python 3.8. The model is trained on CentOS Linux machine with 128 AMD EPYC 7513 32-Core Processors, 512 GB memory, and eight NVIDIA RTX A6000 GPUs.
C Limitations and Broader Impacts
In terms of limitations, it important to acknowledge that our work operates under the assumption that the target testing data still exhibit some similarities with the source training data. If there is
20

---

significant distribution shift, the knowledge acquired from the source training data may become irrelevant. In such cases, neither the DG baselines nor our proposed method can effectively address the problem. It would be more appropriate to explore transfer learning train a new model to obtain a better solution. Further, we propose SLDG to tackle two main challenges: (1) unknown domain IDs and (2) distinct characteristics across domains. In the scenario when the domain IDs are given and clearly separable (e.g., photo, art painting, cartoon, and sketch the PACS [2I] dataset), SLDG domain discovery approach might be unnecessary. Existing DG methods directly utilizing the domain IDs could be better solution.
In terms of broader impacts, our work tackles practical and prevalent issue in healthcare known as the domain shift problem. We aim inspire future research in this area: both by investigating the existence of domain shift under various scenarios, and by contributing to the development of effective solutions for this real-world challenge.
D Notations
Notation
Meaning
a patient's hospital visit
[el,
em]
sequence of events type of an event
T()
mapping function from event its type set of events
T
set of all event types
E (+.-}
label, i.e., the occurrence certain future event overall clinical predictive model
fo()
model parameter loss function training and testing data distribution
((-)
Per Pie E,()
feature-specific patient encoder for event type
he
patient representation in latent space of type hidden dimension
h
(h N.
all patient representations in latent space of type total number of training samples number of discovered domains in the latent space of type domain assignment matrix
N
K;
M,
len..... eml B()
contextualized representation for event sequence [ei..... Cml embedding function
(hi)eT
multi vector representations for a single patient
CLa())
customized classifier for the discovered domain A: in the latent space of type / the gating weight for the customized classifier C.,(()) model output
G.
with Wi,k
learnable prototype weight vectors of the positive and negative classes for the k-th discovered domain in the latent space of type
d(--))
Euclidean distance
We.k
learnable prototypical weight vector for the discovered domain in the latent space
of type
E Additional Illustrations
21

---

Gender: M Age: 77
Patient
Atrial libillattion (148.91)
(15 Min, Diagnosis)
Respiratory fallure (196. 90)
(28 Min, Diagnosis)
(25 Min, Procedure) Respiraton cuspiratory ventilation (E0466)
(60 Min, Procedure) ECG (93012)
CT stan (G029c)
(93 Min, Procerdrre)
Heparin (V (5224)
[132 Min. Medication)
(187 Min, Admin) Discharge
Figure 6: illustration patient visit as input.
Hours
eICU Dataset
ICU Admission
Prediction Time
ICU Discharge
1. Whether the patient will decease at the end of current ICU stay?
2. Whether the patient will be readmitted to ICU within 14 days?
MIMIC-IV Dataset
Hospital Admission
Hospital Discharge
Prediction Time
Whether the patient will decease within days?
Whether the patient will readmitted hospital within days?
Figure 7: An illustration the task definitions in the eICU and the MIMIC-IV datasets.
Decoupled Patient Representations
h
h
Type-Specific Aggregation
e
e2
e
Three-Layer Transformer
22
p
Patient Visit x
Figure 8: The architecture of the feature-specific patient encoder.
22


Reviews:
[{"review_id": "AGTIJ7eR8p", "reviewer": "Reviewer_cG9z", "summary": "* This paper proposes a learning framework for improved predictive performance for EHR data under distribution shift.\n* The key problem tackled here is that many existing domain generalization algorithms rely on data points having domain identifiers, which can then be used in a learning algorithm to build a generalizable model. However, in clinical settings, patients may fall into many domains, which may also be latent (and not explicitly known), making the application of these existing methods challenging.\n* In this work, the proposed framework iterates (1) domain discovery and (2) domain-specific classifier training, to build predictors that are effective for the different domains considered.\n* This framework involves:  obtaining encoded representations for patient visits, clustering visits using a hierarchical procedure to discover domains, and building a weighted nearest neighbour styled classifier within each domain (using prototypes) to predict a label. \n* The paper evaluates the method and several baselines on two EHR datasets: eICU and MIMIC-IV. Two tasks are considered: readmission, and mortality prediction. On both datasets, the method performs well, and improves on baselines.\n* The paper also includes ablations on the number of iterations of the procedure and clusters, providing some more intuition on what drives performance. \n", "questions": "* Could you say more about the specific dimensionality and form of the input sequence that the model sees? And as above, adding a clearer figure to represent the data and the notational abstraction used in the paper. Adding figure to specify the architecture, and how the encoding phase works in more detail.\n* Can you answer my question re. the oracle, as discussed in Weaknesses? Specifically, on the proposed method being (sometimes substantially) worse than the oracle, and how to think about this?\n\nOverall, if the authors can say more about the above two points, I will increase my score.\n\n\n# After rebuttal\nAuthors have answered my main questions, so I will increase my score. ", "limitations": "* Limitations and broader impact are discussed in the appendix. I think however that the limitations could be fleshed out some more, maybe in terms of acknowledging that only two predictive tasks and two datasets are considered.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Clear presentation of the problem and limitations of existing methods e.g., the need for domain IDs, or training a single model across domains.\n* As far as I can tell, method appears to be original and also well motivated based on the problem at hand -- discovering latent regions of similarity and developing specialized classifiers for those regions.\n* Good evaluation setup: fairly comprehensive set of baselines, evaluation on two distinct datasets with varied characteristics\n* Results are encouraging (though a question/comment on this below) -- consistent improvements in AUROC/AUPRC on the two tasks on both datasets.\n* Useful additional results studying importance of clustering algorithm and understanding the nature of the latent spaces.", "weaknesses": "* Overall I was confused by the methodological presentation. Particularly, a diagram presenting the data analogously to the abstraction  (medical record is a set of visits, each with events) would help a lot. The dimensionality of different data/features at points in the pipeline was unclear, and a figure might help this.\n* Similar to the above, Figure 2 showing the method was useful, but I think too high level to properly internalize what was happening in the method. An architecture diagram for the encoder (for example) could help, and may address the point above too.\n* In general, Section 3.1 was tricky to understand -- perhaps this is because the high level presentation at the start was difficult to map to the more specific descriptions later (namely, the Feature specific patient encoding and hierarchical domain clustering paragraphs). Some of this confusion might stem from not understanding the input data very well, particularly the different feature types $t$ and encoder $E_t$, and embedding function $E$. A re-writing of this section might help, together with clear figures for each of the pieces (particularly the encoding section).\n* On the quantitative results: I appreciate that the proposed method improves on the different baselines -- that is encouraging to see. However, the fact that it does not improve on the Oracle method (and in some cases does substantially worse) may be something to discuss further -- why would we not just use the oracle if we have enough data from that domain?  Would finetuning the proposed method on the training set from the target domain help here (for some definition of finetuning? I appreciate it's a bit tricky here given the way the classifiers are parameterized)? In addition, could you show results when you have very limited data from the target domain (and specifically demonstrate that the Oracle no longer performs well)? As I see it, if the Oracle method that is trained on the labelled target data can do well in small target data regimes, then these results are not as convincing. \n* On evaluation of clustering results: Since the proposed method directly looks at the Silhouette score when doing clustering (and not sure the others do), I'm not sure this is very convincing -- we'd expect the proposed method to do better when evaluating Silhouette score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We would like to take this opportunity to clarify some potential misunderstandings regarding our manuscript.\n\n\n**Q1. More explanation on the input data structure and model architecture.**\n\nA1. In our setting, each input sample is defined as a patient’s hospital/ICU visit, which can be represented as a sequence of clinical events. For example, a patient visit can be [Acute embolism (I82. 40), Atrial fibrillation (I48.91), Ultrasound (76700), CT scan (G0296), ECG (93042), Heparin IV (5224)]. In our dataset, each event corresponds to a specific type (e.g., diagnosis, procedure, medication) and a specific timestamp (i.e., when the event occurred). Additionally, we add special <cls> and <eos> tokens at the beginning and end of the event sequence. To embed the patient visit, we use a three-layer Transformer encoder which essentially takes each medical code as a token and performs the self-attention mechanism across different codes. \n\nPlease see the attached PDF file for a detailed illustration. We will also incorporate some examples in the manuscript for better understanding.\n\n\n**Q2. Why is the proposed method weaker than the Oracle method?**\n\nA2. We would like to clarify that the Oracle method is directly trained on the target domain, while the rest of the baselines (base and DG methods) are only trained on the source domain and later tested on the unseen target domain. As all methods use the same backbone encoder, the Oracle method can be viewed as an upper bound of the model performance. As a result, the goal of all DG methods is to match the performance of the in-domain-trained oracle without seeing any in-domain data. And the performance gap between Oracle and DG methods should be the main indicator of model effectiveness. \n\nIn real-world clinical practice, due to privacy and legal concern, it is difficult to obtain the target data (e.g., from other hospitals) to train the Oracle method. Thus, DG solutions that can match the performance of the oracle but without the need of accessing target data are preferred.\n\n**Q3. Evaluation of clustering on the Silhouette score.**\n\nA3. We would like to clarify that during the clustering step, the hyper-parameters are chosen based on the Silhouette score on the training set; while during the evaluation, the Silhouette score is calculated on the testing set. Further, the model is later fine-tuned with the domain-specific classifiers. So there is no unfair advantage of SLDG. Nevertheless, we report the Calinski-Harabasz score (the higher, the better) of SLDG and the best baseline MMLD below.\n\n*Table: Results with additional cluster metric (Calinski-Harabasz score, the higher, the better).*\n\n| Method | eICU readmission | eICU mortality | MIMIC-IV readmission | MIMIC-IV readmission |\n|---|---|---|---|---|\n| MMLD | 137.9  | 125.8 | 169.8 | 264.7 | \n| SLDG (Ours) | **246.5** | **302.5** | **233.1** | **355.9** |\n\n\nThanks for your time during the rebuttal phase. We hope the updates can clarify the misunderstandings and address the reviewer’s concerns."}, {"review_id": "nSIM9Lc8Ku", "reviewer": "Reviewer_Cuot", "summary": "This paper introduces a transformer-based supervised learner (SDLG) for two healthcare domain adaptation adaptation tasks (one across geographic region, and one across time).  The pipeline consists of feature sequence embeddings into type-specific vector spaces via fine-tuned contextualized embedding approach, clustered and subclustered, and then passed through a distance-based gating layer.  The model outperform multiple methods from the domain adaptation literature in two electronic health records datasets.\n\nI have read the rebuttal, and while the clarifying comments are elucidating, I maintain my reservations about the work in its current form.", "questions": "See weaknesses above.", "limitations": "A high-level set of limitations are described in the Appendix.  While the data used are \"real-world\", the simplistic extraction used for the event sequence, and experimental setup suggest that the review should likely focus on the methods provided rather than value for the use case.  The methods can still be improved, perhaps with the use of synthetic data where theory and/or optimality could be established.  ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: The machine learning modeling approach combines multiple standard models in deep learning and uses appropriate fine-tuning methods.  It is applied to an extract of challenging real world (EHR) data.\n\nQuality: The figures (1,2) and equations effectively communicate the technical approach.\n\nClarity: The paper is mostly clear and the descriptions are straightforward to follow.\n\nSignificance: The results suggest substantial improvement over other baseline models, suggesting that, at least for these tasks, this decoupled approach works well when generalizing to extracts from clinical event sequences in a second domain.", "weaknesses": "It seems there could be a slight mismatch in the intuition of having separate latent domains and using the types shown in Fig 2. E.g. toe numbness (symptom), insulin (treatment), and history of diabetes (medical history) are all associated with the condition diabetes, but appear to be encoded separately into spaces/clusters/subclusters.\n\nHow UMAP is used is not entirely clear.  Is it performed on the training data only? If it is applied to the training and test data, is it not leaking information from the test in producing the embedding? If not, when applied to the test data, what is the mapping for test data points, and how are they assigned cluster/subcluster memberships?  The \"hierarchical clustering\" used, if as described, perhaps should be named differently as it does not appear to be the usual hierarchical clustering based on linkage (or details are missing).\n\nThe model pipeline includes many design choices, but is tested only in two particular settings.  In those settings some of the details are not entirely clear.  I was also unable to determine the list of event types used, or how the EHR data (typically thousands of measurements) were converted into sequences roughly 100 and 35 events long, respectively.  Which events/tables were used?  How the timestamps are embedded should be described precisely somewhere (e.g. is it time from first event in some unit of time?).\n\nFor eICU and given the number of sequential design choices, it seems like leave-out-one-domain cross validation would be a better measure of performance, than a single assessment for the held-out midwest region.  For MIMIC, the Appendix claims each temporal range is split into training/validation/test sets, but also says the evaluation is on patients admitted after 2014, which is confusing.\n\nFrom an applied perspective, it is unclear what AUCs are competitive or how they would influence decision making.  Are readmission AUCs of 0.62 compelling (or 0.68 for the oracle)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We would like to take this opportunity to clarify some potential misunderstandings regarding our manuscript.\n\n\n**Q1. Different types of medical codes can be related to each other but are encoded separately.**\n\nA1. We wish to clarify the confusion regarding our feature-specific patient encoding module. Specifically, for a patient’s hospital visit $x = [e_1, . . . , e_m]$, we first pass it through a Transformer encoder to obtain the contextualized event representations, as $[ \\mathbf{e}_1, \\dots, \\mathbf{e}_m ] = E ( [ e_1, \\dots, e_m ] )$. The Transformer encoder employs a self-attention mechanism to capture relationships among event pairs. Subsequently, we aggregate the contextualized event representations based on their respective types. This aggregation results in a distinct patient representation that takes into account different medical codes. Consequently, each medical code contributes to the interaction with all other codes within the same patient visit.\n\n\n**Q2. How UMAP is used?**\n\nA2. UMAP and hierarchical domain clustering is restricted to the training set. Specifically, we start by employing the UMAP algorithm to reduce the embedding dimension of each training sample to 2. Subsequently, a hierarchical k-Means algorithm operates in the dimension-reduced latent space, clustering the training samples. During testing, a test sample is mapped to specific clusters based on the distance between its embedding and the mean cluster embeddings.\n\n\n**Q3. Additional experimental details.**\n\nA3. Thank you for the feedback. We will enhance our manuscript to include more implementation and experiment details. We mainly follow existing works [1,2,3,4] in terms of task definition and feature selection. Specifically, for the MIMIC-IV dataset, we use the diagnosis, procedures, and prescriptions events, extracted from “diagnoses_icd”, “procedures_icd”, and “prescriptions” tables; For the eICU dataset, we use the diagnosis, procedures, prescriptions, and lab events extracted from the “diagnosis”, “treatment”, “medication”, “lab” tables. For both datasets, we also include patient demographics information, including age, gender, and ethnicity, extracted from the “patients” and “admissions” tables for the MIMIC-IC dataset, and from the “patient” table for the eICU dataset. \n\n**Q4. The rationale behind the proposed domain split.**\n\nA4. In our original experiment, for both datasets, the target domain is selected as the domain that exhibits the largest performance gap (with a model trained on other domains) in our pilot study. Additionally, we also split each domain into 70%, 10%, and 20% train, val, and test splits. This is to ensure that the oracle, base, and DG methods are comparable. Specifically, the oracle is trained on the target training and validation set, and tested on the target testing set. In contrast, the base and DG methods are trained on the source training and validation set and then tested on the target testing set. \n\n**Q5. Are readmission AUCs of 0.62 compelling?**\n\nA5. The readmission prediction task is very critical in real-world clinical practice to assist hospital planning and improve patient outcomes. This task is studied in several existing works [4,5,6] and grants (NIH 5K23HL153582-03, NIH 1R01HL119664-01A1, NIH 1R01HL130828-01A1). In our study, we adopt a stricter setup following Guo et al. [7]. For example, for the eICU dataset, the readmission prediction is made 12 hours after ICU admission (compared to the end to current admission in ManyDG [6]). It is important to note that we use the same setup across all methods for a fair comparison. The performance gap between the baseline and oracle method should be the most important indicator of model effectiveness.\n\n\n[1] Edward Choi et al. RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism. NIPS 2016.\n\n[2] Edward Choi et al. Using recurrent neural network models for early detection of heart failure onset. JAMIA 2017.\n\n[3] Phuoc Nguyen et al. Deepr: A Convolutional Net for Medical Records. IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 2017.\n\n[4] Chaohe Zhang et al. GRASP: Generic Framework for Health Status Representation Learning Based on Incorporating Knowledge from Similar Patients. AAAI 2021.\n\n[5] Rich Caruana et al. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD 2015.\n\n[6] Chaoqi Yang et al. MANYDG: MANY-DOMAIN GENERALIZATION FOR HEALTHCARE APPLICATIONS. ICLR 2023.\n\n[7] Lin Lawrence Guo et al. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific Reports 2022.\n\n\nThanks for your time during the rebuttal phase. We hope the updates can clarify the misunderstandings and address the reviewer’s concerns."}, {"review_id": "4v9PmbFWBj", "reviewer": "Reviewer_RZ7L", "summary": "The paper presents a novel method for domain generalization in healthcare, named Subspace Latent Domain Generalization (SLDG). The authors argue that traditional domain generalization methods are not suitable for healthcare applications due to the heterogeneity of patient data. The SLDG method discovers latent domains in patient data and uses them to improve the performance of predictive models. The authors provide empirical evidence to support their claims, demonstrating that the SLDG method outperforms several other methods in predicting patient readmission and mortality on two datasets, eICU and MIMIC-IV.", "questions": "* How does the SLDG method handle situations where the latent domains are not clearly separable? Could this affect the performance of the method?\n* Was the model evaluated in smaller datasets? What are the scenarios in which this method should or should not be applied?", "limitations": "The authors have provided a clear and detailed description of their method and have presented empirical evidence to support their claims. However, they could further discuss the limitations of their method, particularly in terms of the generalizability of the results. The paper focuses on two specific datasets, and it's unclear how well the method would perform on other datasets or in other healthcare contexts. The authors could also discuss the potential challenges of applying the SLDG method in practice, such as the need for large amounts of high-quality data.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Novel Approach: the paper presents a novel method (SLDG) for domain generalization in healthcare. This method, which discovers latent domains in patient data, could potentially improve the performance of predictive models in healthcare, where data is often heterogeneous and collected from various sources.\n* Performance: the proposed method surpassed several other strategies on both readmission and mortality for both datasets used for evaluation.\n* Method description: the proposed approach is extensively described with the provided pseudocode. Similarly the methodology employed is properly explained in the paper, with additional material containing the specificities of each section. ", "weaknesses": "Even though it performed well on the two evaluated datasets, it isn't clear how it would perform on smaller datasets with large amounts of missing data. It is possible the method won't perform so well in a dataset that is smaller than the ones where it was evaluated (1/10th of its size). There is a lack of discussion on the proposed method's limitations and possible challenges within the application. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive feedback. We have made the necessary updates to our manuscript. Below we would like to take this opportunity to respond to your questions.\n\n\n**Q1. How would SLDG perform on smaller datasets? Or when the latent domains are not clearly separable?**\n\nA1. It is difficult to provide a general performance conclusion of SLDG on unseen datasets (“No Free Lunch” theory). However, in our model design, we chose an efficient cluster-based parameterization initialization that can better accommodate smaller datasets. In terms of domain separability, we adopt a hierarchical clustering approach based on the Silhouette score which will automatically decide the number of clusters.\n\n\n**Q2. What are the scenarios in which this method should or should not be applied?**\n\nA2. We propose SLDG to tackle two main challenges: (1) unknown domain IDs and (2) distinct characteristics across domains. In the scenario when the domain IDs are given and clearly separable (e.g., photo, art painting, cartoon, and sketch in the PACS dataset), SLDG’s domain discovery approach might be unnecessary. Existing DG methods directly utilizing the domain IDs might be a better solution.\n\nThank you for taking the time during the rebuttal phase. We hope that these updates address your questions."}, {"review_id": "86zeweDnfE", "reviewer": "Reviewer_36fo", "summary": "the authors present a novel self-learning framework, sldg, designed to address the domain shift problem in deep learning models, particularly in the context of clinical decision-making. the authors posit that reliance on domain ids is a key limitation in existing domain generalization algorithms. to overcome this, the sldg framework iteratively decouples domains and trains personalized classifiers for each domain. the results demonstrate improvements in the auprc score over baselines.\n\nedit august 19, 2023: increase score from 5 to 6 in light of authors' rebuttal.", "questions": "1. can you elaborate on the process of decoupled domain discovery in sldg?\n\n2. how could the meaningfulness of each domain sldg identifies be evaluated more thoroughly?\n\n3. why were certain related approaches not included in the comparison with sldg?\n\n4. could you provide a more comprehensive contextualization with existing dg methods?\n\n5. could you provide more details on the computational complexity of sldg vis-a-vis the other dg baselines you employ in your experiments?\n\n6. the experiments seem focused on ehr data, but the title suggests broader applicability. could you clarify?\n\n7. how would sldg be adapted for domains beyond ehr?\n\n8. how does sldg relate to techniques like federated learning with clients that have non-iid data?\n\n9. how does sldg relate to disentangled representation learning?\n\n10. these methods also tackle domain generalization, often without domain ids. could you discuss any potential connections or distinctions?", "limitations": "it would be great to discuss more connections to related work from connected disciplines working on similar problems and include a clear discussion of limitations in the main text, which is missing at the moment.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **innovative approach to domain generalization**: the sldg framework introduces a novel way to handle the domain shift problem in deep learning models. unlike existing methods that rely on domain ids, sldg iteratively discovers decoupled domains and trains personalized classifiers for each domain. the approach is clearly explained.\n\n2. **significant performance improvement**: the paper demonstrates that the sldg framework can achieve a substantial improvement in the auprc score over the best baseline, indicating the promise of the proposed method.\n\n3. **applicability in healthcare settings**: the authors have focused on the application of their model in clinical decision-making, a field where domain shift is a common issue. the ability of the sldg framework to handle potential domain shifts when applied to target data is a significant strength, as it makes the model highly relevant and potentially impactful in real-world healthcare settings.\n\n4. **comprehensive technical details**: the inclusion of a detailed pseudocode for the sldg algorithm in the appendix is a strength. it provides a clear understanding of the steps involved in training and inference, which can be beneficial for researchers and practitioners looking to implement or build upon this work.\n\n5. **robust testing**: the authors have tested their algorithm on two different datasets for readmission and mortality prediction tasks. this robust testing adds credibility to the results and demonstrates the versatility of the algorithm. the filtering of datasets to exclude certain visits also shows a thoughtful approach to ensuring the quality and relevance of the data used.", "weaknesses": "1. **domain discovery process**: the paper presents an iterative process for decoupled domain discovery (section 3.2). a detailed explanation of how the algorithm ensures the discovered domains are meaningful and distinct from each other. a more thorough discussion or a visual representation of the discovered domains could enhance the understanding of this process.\n\n2. **baseline comparison**: the paper mentions that sldg outperforms the best baseline (section 4.1). however, many related approaches are not mentioned and no explanation is provided why they should be discarded or how they relate to sldg. including more comparisons, empirically or at least conceptually, with state-of-the-art models could strengthen the paper's claims about sldg's novelty. a good general overview can be found at pwc https://paperswithcode.com/task/domain-generalization.\n\n3. **algorithm complexity**: the pseudocode provided in the appendix outlines the steps of the sldg algorithm, but the paper does not discuss the computational complexity of these steps beyond base and sldg. providing runtime information for the other dgs as well as flops or macs information could help readers understand the feasibility of implementing sldg in practice.\n\n4. **generalizability**: while the paper demonstrates the effectiveness of sldg in healthcare settings, it does not discuss its applicability to other fields. a discussion on how the proposed method could be adapted for other domains would increase the paper's impact and reach. similarly, the title suggests application to medicine broadly, while experiments are performed on ehr data only.\n\n5. **relation to connected related fields**: the authors do not discuss in detail and only briefly at the end how the proposed sldg framework relates to or differs from other relevant techniques. some important ones, such as federated learning and disentangled representation learning, are not mentioned at all despite similar problems having been tackled in these fields (see for example https://ieeexplore.ieee.org/abstract/document/9174890 or https://aclanthology.org/P19-1041/, among many other works*). given that these methods also deal with domain generalization, often without domain ids such as in private distributed clients, a comparison or discussion could provide valuable context and highlight the unique contributions of sldg.\n\n*i have no affiliation to examples of related work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive feedback. We have made the necessary updates to our manuscript. Below we would like to take this opportunity to respond to your questions.\n\n\n**Q1. Can you elaborate on the process of decoupled domain discovery and the meaningfulness of each domain?**\n\nA1. The hierarchical domain clustering is only applied to the training set during model training. Specifically, we first use the UMAP algorithm to reduce the embedding dimension of each sample to 2. Then, we run a hierarchical k-Means algorithm to cluster the samples in the dimension-reduced latent space. During testing, the test sample is mapped to different clusters based on the distance between its embedding and the mean cluster embeddings. In section 4.4, we provide a case study and visualization regarding the meaningfulness of the discovered domain.\n\n\n**Q2. Additional comparison with existing dg method.**\n\nA3. Below, we provide additional experiments with two recent DG methods: GroupDRO [2] and GMoE [3].\n\n*Table: Comparison between SLDG and two additional DG methods. The SLDG archives the best performance in all tasks.*\n\n| Method | eICU readmission | eICU mortality | MIMIC-IV readmission | MIMIC-IV readmission |\n|---|---|---|---|---|\n| GroupDRO | 0.158 | 0.252 | 0.244 | 0.397 |\n| GMoE | 0.169 | 0.246 | 0.248 | 0.385 |\n| SLDG (Ours) | **0.186** | **0.268** | **0.274** | **0.416** |\n\n\n**Q3. Computational complexity of SLDG.**\n\nA3. The main computation overhead of SLDG happens during the training phase, where we need to perform an additional hierarchical clustering step. However, the computation overhead is minimized due to the usage of UMAP dimension reduction. As shown at the end of section 4.3, the training time of SLDG only increases by 12 minutes compared to the most efficient base method. In terms of inference efficiency and model complexity, SLDG is very similar to the other DG methods not requiring domain IDs (i.e., MMLD and DRA). Below we compare model parameters and inference efficiency between these methods.\n\n*Table: Additional comparison of model parameter size and inference time.*\n\n| Method | \\# Params | Inference Time (ms) | \n|---|---|---|\n| Base | 1.3M | 33 |\n| MMLD | 1.5M | 54 |\n| DRA | 1.5M | 63 |\n| SLDG (Ours) | 1.5M | 68 |\n\n\n**Q4. Broader application of SLDG.**\n\nA4. Our main insight is to iteratively discover latent domains and train personalized classifiers for each latent domain. This idea should be applicable in other scenarios facing similar challenges (i.e., no domain ids and distinct characteristics across domains). Below, we show an additional experiment on the medical imaging classification task. Specifically, we use the MIMIC-CXR data, consisting of chest X-ray data collected between 2008 and 2019. We train a ResNet backbone model on the X-rays before 2014 and test it on X-rays after 2014. \n\n*Table: Additional experiment on chest X-ray classification with temporal domain shift. SLDG can still outperform other DG baselines.*\n\n| Method | ROC-AUC |\n|---|---|\n|Oracle | 0.8011 |\n| Base | 0.7462 |\n| MMLD | 0.7580 | \n| DRA | 0.7446 |\n| SLDG (Ours) | **0.7736** |\n\n\n**Q5. Comparison to federated learning / disentangled representation learning.**\n\nA5. SLDG aims to tackle two main challenges: (1) unknown domain IDs and (2) distinct characteristics across domains. These challenges can be seen in the federated learning setting where different clients have non-iid data. We believe that some of our modules (e.g., decoupled representation learning and latent domain discovery) can also be leveraged in the federated learning setting.\n\nIn terms of disentangled representation learning, our method shares similar high-level intuitions. That is, we decouple the patient representation into different latent spaces to discover the view-specific latent domains. In the main experiment, we also include ManyDG, a SOTA disentangled representation learning baseline.\n\n\n[1] Huaxiu Yao et al. Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time. NeurIPS 2022.\n\n[2] Shiori Sagawa et al. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR 2020.\n\n[3] Bo Li et al. SPARSE MIXTURE-OF-EXPERTS ARE DOMAIN GENER- ALIZABLE LEARNERS. ICLR 2023.\n\nThank you for taking the time during the rebuttal phase. We hope that these updates address your questions."}, {"review_id": "kKuoprVDjh", "reviewer": "Reviewer_1qdB", "summary": "In this work, the authors present SLDG: a framework for dealing with domain shifts in clinical event data. They propose two steps in this framework. First, an unsupervised latent-domain clustering which automatically clusters different subjects into different feature-level clusters via monitoring the Silhouette score for optimal clusters, followed by individually trained classifiers for each cluster group, where the final probabilistic prediction is the weighted combination of the predictions from within each cluster group. The individually trained classifier is the softmax-ed distance between a patient's latent representation and the prototypical representations for the latent space. The authors find that by training this method by reinitializing the clusters every set number of epochs they can beat existing domain adaptation methods.", "questions": "- What is meant on line 121 with the embedding lookup table?\n- What is the intuition for using sinusoidal positional embeddings? I mean to say, why does it make sense here to embed timestamps with sinusoidal positional encodings when the patient visits would only be going forwards in time? Maybe I misunderstood this piece and would need some clarification.\n- The method for reducing the clusters from exponential to linear makes sense, but shouldn't the label types be coupled to some degree? Many of those features shouldn't be independent of each other so making an independence assumption is strange.\n- Related to my point of weakness with evaluation, I would definitely consider raising my score if there was additional necessary context concerning the capacity of the proposed method (with additional parameters per clustering included) in comparison to the baseline methods. I think that the clustering method is sufficiently novel and interesting that it would be helpful for others to know about, the only thing keeping my score at a borderline except at the moment is that I am unsure if the comparison to the baselines is fair.\n", "limitations": "The authors don't explicitly discuss the potential negative social impact of their work in the main body of the text, right now it is in the appendix, and I believe that it should be moved to the main body. It does address, however, the limitations of the work and places it meaningfully in context.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "- The idea and method are communicated in a clear manner and provide a good understanding of the underlying problem.\n- The method has clear benefits over baseline methods, sometimes with large jumps in improvement (Readmission on eICU).\n- The work does offer a unique and interesting method for self-supervised clustering of latent features. I really liked the proposed method and thought that it was well explained.\n- The ablation of the clustering algorithm demonstrates that the added complexity of the hierarchical clustering indeed shows benefits over traditional methods, something which other works could potentially build on top of.\n", "weaknesses": "- Customized classifiers for each domain are a very inefficient method of scaling to new domains even with the efficient parameterization of each domain. Although, this is definitely more problematic if the latent dimension is larger or smaller and there doesn't seem to be any kind of ablation for the size of the classifier dimension. I believe this might be a useless result to have.\n- The formulation of the prototypical vectors I don't think is very well motivated and is confusing why there is a positive and negative representation of each latent space t. Additionally, I'm not exactly sure why it would make sense to average the predictions from all latent domains to make new predictions given that for the downstream prediction task, not all clusters are probably equally helpful with making predictions. This seems like it would introduce a lot of noise into predictions.\n- The biggest weakness I find with the analysis of this work is that it is unclear how large the baseline methods are in comparison to the proposed method. The paper lacks an analysis of parameter numbers which is problematic because as you develop more clusters, it is my understanding that you will also balloon up your parameter count, but also we don't know how big the transformer architecture is in comparison to other baseline methods. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe would like to thank the reviewer for the positive feedback. We have made the necessary updates to our manuscript. Below we would like to take this opportunity to respond to your questions.\n\n\n**Q1. What is meant on line 121 with the embedding look-up table?**\n\nA1. The embedding look-up table is essentially a mapping from the event to a static embedding vector. For example, for the event “acute heart failure”, we obtain its static embedding by feeding the string “acute heart failure” through the ClinicalBERT and using the pooled sequence representation as the event embedding (dimension 768). In practice, we stack the embeddings for all events together into a matrix and store it in the `nn.Embedding()` layer.\n\n\n**Q2. What is the intuition for using sinusoidal positional embeddings?**\n\nA2. In both datasets, each event is associated with a timestamp denoting the time when the event occurred. We transform the absolute time to the relative time after patient admission (in hours) and embed the hour value using sinusoidal positional encoding. The intuition behind the temporal embedding is that the timestamps can provide an important indication of the speed of the patient's health progression (e.g., acute v.s. chronic).\n\n\n**Q3. The label types can be coupled to some degree?**\n\nA3. We wish to clarify the confusion regarding our feature-specific patient encoding module. Specifically, for a patient’s hospital visit $x = [e_1, . . . , e_m]$, we first pass it through a Transformer encoder to obtain the contextualized event representations, as $[ \\mathbf{e}_1, \\dots, \\mathbf{e}_m ] = E ( [ e_1, \\dots, e_m ] )$. The Transformer encoder employs a self-attention mechanism to capture relationships among event pairs. Subsequently, we aggregate the contextualized event representations based on their respective types. This aggregation results in a distinct patient representation that takes into account different medical codes. Consequently, each medical code contributes to the interaction with all other codes within the same patient visit.\n\n**Q4. The capacity of the proposed method (with additional parameters per clustering included) in comparison to the baseline methods.**\n\nA4. The main computation overhead of SLDG happens during the training phase, where we need to perform an additional hierarchical clustering step. However, the computation overhead is minimized due to the usage of UMAP dimension reduction. As shown at the end of section 4.3, the training time of SLDG only increases by 12 minutes compared to the most efficient base method. In terms of inference efficiency and model complexity, SLDG is very similar to the other DG methods not requiring domain IDs (i.e., MMLD and DRA). Below we compare model parameters and inference efficiency between these methods.\n\n*Table: Additional comparison of model parameter size and inference time.*\n\n| Method | \\# Params | Inference Time (ms) | \n|---|---|---|\n| Base | 1.3M | 33 |\n| MMLD | 1.5M | 54 |\n| DRA | 1.5M | 63 |\n| SLDG (Ours) | 1.5M | 68 |\n\nThank you for taking the time during the rebuttal phase. We hope that these updates address your questions."}]

Author rebuttal:
Additional PDF illustrating the input data structure, task definitions, and encoder architecture.

Meta-review: This paper proposes a domain generalization framework relevant to healthcare. The key idea is that "domains" in health are not as clear as other applications. Thus the authors provide an iterative framework that learn the domains using an embedding+hierarchical clustering. The domain generalization framework + latent domain identification together forms their self-learning framework. Authors demonstrate experiments on MIMIC and eICU, standard health benchmarks.

Reviewers overall have not found major technical issues, although several clarity issues have been identified. Authors have addressed major concerns from the reviewers including additional baselines, which I think should have been part of the original contribution (such as GroupDRO). My own view of the paper is that UMAP and related methods are fairly unreliable but might be acceptable for a downstream classification task. As such I expect the authors to include limitations of their unsupervised part of the framework. 

Besides that I would like to encourage authors to be more precise for clarity and include all additional empirical evaluation in the final version. Assuming the authors will deliver on this, I recommend an accept.
Decision: Accept (poster)