Title: An Iterative Self-Learning Framework for Medical Domain Generalization
Venue: NeurIPS 2023
Accepted: yes
Keywords: healthcare

clinical predictive model

domain generalization
Abstract: Deep learning models have been widely used to assist doctors with clinical decision-making. However, these models often encounter a significant performance drop when applied to data that differs from the distribution they were trained on. This challenge is known as the domain shift problem. Existing domain generalization algorithms attempt to address this problem by assuming the availability of domain IDs and training a single model to handle all domains. However, in healthcare settings, patients can be classified into numerous latent domains, where the actual domain categorizations are unknown. Furthermore, each patient domain exhibits distinct clinical characteristics, making it sub-optimal to train a single model for all domains. To overcome these limitations, we propose SLGD, a self-learning framework that iteratively discovers decoupled domains and trains personalized classifiers for each decoupled domain. We evaluate the generalizability of SLGD across spatial and temporal data distribution shifts on two real-world public EHR datasets: eICU and MIMIC-IV. Our results show that SLGD achieves up to 11% improvement in the AUPRC score over the best baseline.

An Iterative Self-Learning Framework for Medical Domain Generalization

Abstract

Deep learning models have been widely used to assist doctors with clinical decisionmaking. However, these models often encounter a significant performance drop when applied to data that differs from the distribution they were trained on. This challenge is known as the domain shift problem. Existing domain generalization algorithms attempt to address this problem by assuming the availability of domain IDs and training a single model to handle all domains. However, in healthcare settings, patients can be classified into numerous latent domains, where the actual domain categorizations are unknown. Furthermore, each patient domain exhibits distinct clinical characteristics, making it sub-optimal to train a single model for all domains. To overcome these limitations, we propose SLDG , a self-learning framework that iteratively discovers decoupled domains and trains personalized classifiers for each decoupled domain. We evaluate the generalizability of SLDG across spatial and temporal data distribution shifts on two real-world public EHR datasets: eICU and MIMIC-IV. Our results show that SLDG achieves up to 11% improvement in the AUPRC score over the best baseline.

1 Introduction

Deep learning techniques have been increasingly popular in clinical predictive modeling with electronic health records (EHRs) [12, 11, 47, 58, 2]. However, these models typically assume that the training (source) data and testing (target) data share the same underlying data distribution (i.e., domain). This assumption can become problematic when models are applied to new domains, such as data from different hospitals or future time points [17, 59, 20, 37]. In these situations, domain shifts caused by variations in patient cohorts, clinical standards, and terminology adoption can significantly degrade the model's performance.

This paper aims to develop a clinical predictive model on the source data that effectively handles potential domain shifts when applied to the target data. Domain generalization (DG) [7] methods have been widely utilized to address such problems, including techniques like domain alignment [32, 24, 25, 45, 31, 51, 62], meta-learning [23, 27, 26, 5, 22, 29], and ensemble learning [9, 42, 43, 61]. However, when applied in healthcare settings, these methods encounter the following limitations:

Reliance on domain IDs. Most DG methods depend on the presence of domain IDs, which indicate the domain to which each sample belongs, to guide the model training [24, 16, 4, 9, 42]. However, as shown in Fig. 1, patients can be divided into numerous latent domains based on features such as age, medical history, treatment, and symptoms. The actual categorization of these latent domains can be difficult to obtain and vary across different tasks [1, 49]. Consequently, existing DG methods often resort to broad domain categorizations, such as hospital or timestamp, which provide limited information [59, 17].

•

Known domain IDs

•

Sharable characteristics across domains

Medical

History

Symptom

Treatments

Age

Viral Infection + Fever

Heart Failure + Fever

Fever is normal

Fever is problematic

Medical Domain Generalization

•

Unknown domain IDs

•

Distinct characteristics across domains

Attempt to train a single model. While some recent DG methods have attempted to alleviate the reliance on domain labels [62, 29], they try to train a single model that generalizes across all domains. However, patients from different domains possess distinct characteristics and require different treatment approaches [2, 58]. For example, as shown in Fig. 1, fever is considered a normal symptom for patients with viral infections as it helps stimulate the immune system. On the other hand, it can be a bad signal for patients with cardiovascular disease, leading to complications. Thus, training a single model for all domains is challenging and can lead to sub-optimal performance.

To overcome these limitations, we propose SLDG , a self-learning framework for domain generalization that iteratively discovers decoupled domains and trains customized classifiers for each discovered domain. Specifically, SLDG consists of the following iterative steps:

Decoupled domain discovery. While domain labels are not initially available, we posit that they can be recovered by clustering the learned latent representations. However, identifying all finegrained domains across various clinical features (e.g., demographics, diagnosis, and treatments) can be challenging. Instead, we propose to decouple these clinical features and discover the clusters separately for each type of feature. To achieve this, we maintain a distinct latent space for each type of features using a feature-specific patient encoder. Within each latent space, we perform hierarchical clustering independently to discover the domain categorizations. By adopting this approach, we effectively reduce the number of domains from exponential to linear to the number of feature types.

Domain-specific model customization. To account for the unique characteristics of patients in different domains, our approach involves training customized classifiers for each domain. To ensure parameter efficiency, we extract domain representations from the learned clusters and utilize them to parameterize the domain-specific classifiers. For a given patient, we determine the closest domain by comparing the patient's representations with the domain representations, and subsequently select the corresponding classifier for accurate inference.

To assess the generalizability of SLDG across spatial and temporal shifts, we conduct experiments on two publicly available EHR datasets: eICU [38] and MIMIC-IV [18]. Our results demonstrate that SLDG outperforms the best baseline by up to 11% in terms of AUPRC score. We also conduct detailed analyses and ablation studies to investigate the factors contributing to the performance gain achieved by SLDG .

Source Patient Data

Decoupled Domain Discovery

Classifier

Domain-Specific Model Customization

Classifier

Classifier

+

Prediction

Medical

History

Symptom

Treatments

Age

Feature

Specific

Patient

Encoders

Classifier

Age

Symptom

Treatments

Medical History

Iteration

2 Preliminaries

In EHR data, a patient's hospital visit is represented by a sequence of events, denoted as x = [ e 1 , e 2 , . . . , e m ] , where m is the total number of events in the visit. Each event e characterizes features of a certain type t , such as diagnosis, prescription, and lab tests. This mapping is denoted by a function T ( e ) : E → T , where E and T denote the sets of all events and types, respectively. For example, a patient visit can be [Acute embolism (I82. 40), Atrial fibrillation (I48.91), Ultrasound (76700), CT scan (G0296), ECG (93042), Heparin IV (5224)]. And each event corresponds to a specific type (e.g., diagnosis, procedure, medication). The main objective of clinical predictive modeling is to predict the occurrence of future events, such as 15-day hospital readmission and 90-day mortality, denoted as y ∈ { + , -} , based on the patient's current visit x .

Existing clinical prediction works typically train a model f ϕ ( · ) with parameter ϕ by minimizing a loss function l ( · ) on source training data sampled from distribution P tr , as in Eq. (1),

with the hope that the trained model can perform well on the target test data distributed according to P te . However, in real-world settings, the source and target distributions can differ due to spatial and temporal shifts, i.e., P tr = P te . Consequently, the model trained on source data may experience a drop in performance when applied to the target data.

̸

3 The SLDG Approach

In this paper, our goal is to train a model f ϕ ( · ) on source data P tr that can generalize to target data P te despite potential domain shifts. Existing DG algorithms face limitations due to their reliance on domain IDs and attempts to train a single model for all domains. To overcome these limitations, we propose to iteratively discover latent domains and train customized classifiers for each domain. However, we face the challenge of dealing with a large number of latent domains, which not only makes domain discovery difficult but also results in an exponential increase in the number of model parameters with respect to the number of feature types. In the following sections, we will describe how our method SLDG addresses this challenge through decoupled domain discovery and domainspecific model customization. Additionally, we will introduce the training and inference strategy. Fig. 2 illustrates the SLDG framework.

3.1 Decoupled Domain Discovery

Although domain labels are not initially available, we hypothesize that the domain information is encoded in the learned latent representations and can be recovered with the clustering technique. How-

ever, patients can be categorized into thousands of latent domains, determined by various features such as age, medical history, treatment, and symptoms. For instance, a patient can fall into the fine-grained domain of older male patients with a history of smoking and a diagnosis of type 2 diabetes . Identifying all such fine-grained domains can be challenging, as clustering methods may either overlook smaller domains or result in an excessive number of domains that would inflate the number of parameters in subsequent steps. To address this, we propose decoupling these clinical features and independently discovering clusters for each feature type. For example, the patient above can simultaneously belong to the decoupled domains of older , male , history of smoking , and diagnosis of type 2 diabetes . This approach effectively reduces the number of domains from exponential to linear with respect to the number of feature types.

Concretely, we maintain a distinct latent space for each type of feature. When given an input patient visit x , SLDG maps it to the latent space corresponding to the feature type t ∈ T using a feature-specific patient encoder E t ( · ) , as in Eq. (2),

where h denotes the hidden dimension. Next, within each latent space of type t , SLDG gathers all patient representations { h ( i ) t } N tr i =1 , where N tr is the number of source training data, and performs clustering to discover the domain categorizations, as in Eq. (3),

where K t represents the number of discovered domains in the latent space of type t . M t denotes the learned domain assignment, where M t [ i, k ] is equal to one if and only if (i.f.f.) the patient x ( i ) is assigned to the k -th domain. We will describe this procedure in detail in the following.

Feature-Specific Patient Encoding. This module is responsible for mapping each patient into multiple latent spaces, each capturing the patient's health status of a specific feature type. This enables subsequent modules to decouple the representations of different feature types. For a patient's hospital visit x with a list of events [ e 1 , . . . , e m ] , SLDG computes the contextualized representation for each event by applying the embedding function E ( · ) , as in Eq. (4),

where e j is the contextualized representation for event e j with dimension h . We model E ( · ) using a three-layer Transformer [48] framework. To ensure that there are no unseen events in the target data, we initialize the event embedding look-up table with ClinicalBERT [3] embeddings of the event name and then project it down to our hidden dimension of size h . The embedding look-up table is fixed during training.

Next, SLDG aggregates the contextualized event representations [ e 1 , . . . , e m ] based on their types, such as family history, diagnosis, and treatments. For each type t ∈ T , the type-specific representation h t is computed by averaging the representations of all events of that type, as in Eq. (5),

where T ( e i ) indicates the type of event e i . If no events belong to a certain type, the pooled sequence representation is used as a substitute. Consequently, each patient's hospital visit is represented by a set of vectors { h t } t ∈T , with each vector capturing the patient's health status from a specific type of events. These decoupled patient representations are then utilized to perform per-feature-type domain clustering, described next.

Hierarchical Domain Clustering. This module is responsible for clustering patient representations in each latent space to discover latent domains, enabling subsequent modules to customize the classifier for each domain. In the previous step, we obtain a set of patient representations { h ( i ) t } N tr i =1 for each latent space of type t . To perform clustering, standard clustering techniques such as k-Means and Gaussian Mixture Model (GMM) require specifying the number of clusters, which is less ideal as the number of clusters can be difficult to choose and may vary across latent spaces. Inspired by GEORGE [46], we adopt a fully automated hierarchical clustering technique by monitoring the Silouette score [40].

Specifically, in each latent space, SLDG first applies UMAP [30] for dimensionality reduction. Then, it runs k-Means with k ∈ { 2 , . . . , 10 } to identify the optimal number of clusters based on the highest Silhouette score. Subsequently, SLDG further split each cluster into five sub-clusters. However, only sub-clusters surpassing the Silhouette score of the original cluster and containing at least 500 patients are retained. The final number of clusters in the latent space of type t is denoted as K t . The cluster assignment is represented by a binary matrix M t of size N tr × K t , where M t [ i, k ] is set to one i.f.f. the patient x ( i ) is assigned to the k -th cluster. This automated approach allows us to effectively select the number of clusters in each latent space, balancing between discovering overly coarse or fine-grained clusters.

3.2 Domain-Specific Model Customization

To accommodate the unique characteristics of patients in different domains, we propose to train customized classifiers for each decoupled domain. Given an input patient visit x and its multi-vector representations { h t } t ∈T , SLDG computes the predicted probability o of a specific event occurring by employing a weighted combination of domain-specific classifiers in each latent space t ∈ T , as in Eq. (6),

where C t,k ( · ) refers to the customized classifier for the discovered domain k in the latent space of type t , while G t,k ( · ) corresponds to the gating function. In the following, we will elaborate on how SLDG leverages the clustering results to efficiently parameterize the domain-specific classifier and effectively determine the gating weights.

To efficiently parameterize the domain-specific classifier C t,k ( · ) for the k -th discovered domain in the latent space of type t , we define two learnable weight vectors of size h : w + t,k and w -t,k , which represent the prototypes of the positive and negative classes, respectively. The predicted probability of a specific event occurring is computed based on the relative distance between the patient representation h t and the positive and negative prototypical weights, as in Eq. (7),

where d ( · , · ) is the Euclidean distance. To facilitate efficient learning, we initialize the two prototypical weight vectors w + t,k and w -t,k , with the average representations of patients from the corresponding classes, as in Eq. (8),

where M t [ i, k ] = 1 includes only patients assigned to the k -th domain in the latent space of type t .

We adopt a similar approach to the gating function G t ( · ) . For each discovered domain k in the latent space of type t , we introduce a learnable prototypical weight vector w t,k ∈ R h . The gating weights are determined based on the distance between the patient representation h t and the corresponding prototypical weights, as in Eq. (9),

where the prototypical weight vectors { w t,k } K t k =1 are initialized as the average representations of patients in that domain, as in Eq. (10),

3.3 Training and Inference

To train SLDG , we begin by utilizing a pre-trained patient encoder 1 for decoupled domain discovery. Then, we iteratively update the model weights and re-generate the clusters every 20 epochs. In each

iteration, we re-initialize the classifier and gating parameters. This iterative process is repeated three times to enhance the model's performance. During training, we minimize the binary cross-entropy loss. For inferencing, given a target patient visit, SLDG first maps it to multiple decoupled latent spaces with the feature-specific patient encoders E t ( · ) . Subsequently, in each latent space of type t , the gating function G t ( · ) determines the weight combinations used to aggregate the predictions from domain-specific classifiers { C t,k ( · ) } K t k =1 . The final prediction is obtained by averaging the predictions from all latent domains. The pseudocode of SLDG can be found in Appx. A.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate SLDG on two publicly available real-world EHR datasets: eICU [38] and MIMIC-IV [18], which are described as follows:

eICU [38] covers over 200K visits for 139K patients admitted to the intensive care unit (ICU) in one of the 208 hospitals across the United States. The data was collected between 2014 and 2015. The 208 hospitals can be further categorized into four groups based on their location (Midwest, Northeast, West, and South). We use age, gender, and ethnicity as patient demographic information, and leverage the diagnosis, treatment, medication, and lab tables to gather patient visit information.

MIMIC-IV [18] covers over 431K visits for 180K patients admitted to the ICU in the Beth Israel Deaconess Medical Center. The data was collected between 2008 to 2019. The approximate actual year of each admission is revealed as one of the four-year groups (2008-2010, 2011-2013, 2014-2016, and 2017-2019). We use age, gender, and ethnicity as patient demographic information, and leverage diagnoses, procedures, and prescriptions to gather patient admission information.

We elaborate on the cohort selection process and provide comprehensive dataset statistics in Appx. B.1. In the end, we extract 149227 visits from 116075 patients in the eICU dataset, and 353238 visits from 156549 patients in the MIMIC-IV dataset.

Clinical Predictive Tasks. We focus on two common clinical predictive tasks: (1) Readmission prediction, which aims to determine whether a patient will be readmitted within the next 15 days following discharge. (2) Mortality prediction, which aims to predict whether a patient will pass away upon discharge in the eICU setting, or within 90 days after discharge in the MIMIC-IV setting. A detailed explanation of this setting can be found in Appx. B.2.

Data Split. We evaluate the performance of our model across spatial gaps using the eICU dataset. For this purpose, we select the target testing data as the group (Midwest) that demonstrated the largest performance gap in a pilot study. The remaining groups (Northeast, West, and South) are used as the source training data. To assess the model's performance across temporal gaps, we utilize the MIMIC-IV dataset. Patients admitted after 2014 are used as the target testing data, while all preceding patients are included in the source training data. We elaborate more on the data split in Appx. B.3.

Baselines. We compare SLDG against three categories of baselines. (1) The first category consists of naive baselines, including Oracle , trained directly on the target data, and Base , trained solely on the source data. (2) The second category comprises DG methods that require domain IDs. These include DANN [16] and MLDG[22] , which use coarse regional and temporal groups as the domain definition, and ManyDG [56] , which treats each patient as a unique domain. (3) The last category consists of DG methods that do not rely on domain IDs, including IRM [4] , MMLD [29] , and DRA [14] . A detailed explanation of all the baselines can be found in Appx. B.4.

Evaluation Metrics. Both readmission prediction and mortality prediction are binary classification tasks. To evaluate the performance of the models, we calculate the Area Under the Precision-Recall Curve (AUPRC) and the Area Under the Receiver Operating Characteristic Curve (AUROC) scores. For each metric, we report the average scores and standard deviation by performing bootstrapping

1000 times. Additionally, we conduct independent two-sample t-tests to assess whether SLDG achieves a significant improvement over the baseline methods.

Implementation Details. For all baselines, we use the same Transformer [48] architecture as the backbone encoder. Patient demographics features (age, gender, and ethnicity) are embedded with an embedding look-up table. We also embed the timestamps with sinusoidal positional encoding. The medical, patient demographics, and temporal embeddings are added together to form the overall sequence embedding. All models are trained for 100 epochs, and the best model is selected based on the AUPRC score monitored on the source validation set. For SLDG , UMAP [30] from UMAPlearn [41] is used with 2 components, 10 neighbors, and 0 minimum distance; and k-Means from Scikit-learn [35] is used with the default hyper-parameter. Further information regarding the detailed implementations can be found in Appx. B.5.

4.2 Main Results

Table 1 presents the domain generalization results on the eICU [38] and MIMIC-IV [18] datasets. Firstly, we observe a significant performance gap between the Oracle and Base methods, indicating the presence of substantial spatial and temporal domain gaps. This supports the use of the DG setting. Notably, the readmission tasks exhibit larger domain gaps, which is reasonable since hospitals across different locations and timestamps may have varying criteria for patient readmission. Secondly, we note that the two DG methods, DANN [16] and MLDG [22], utilizing coarse domain partitions such as region and timestamp, achieve minimal or no improvements. This outcome is expected because the domain partitions are too coarse, making it challenging to identify consistent domain features. In comparison, ManyDG [56] achieves better performance by considering each individual patient as a unique domain. Among the remaining three baseline methods that do not rely on domain IDs, IRM [4] demonstrates the slightest improvement. DRA [14] performs better due to the usage of multi-head networks, which share a similar intuition as SLDG . MMLD [29] attains the highest performance among all baselines, showcasing the advantages of explicit domain discovery. Lastly, SLDG outperforms baselines for all tasks. Specifically, in terms of the AUPRC score, SLDG achieves an 11% relative improvement in eICU readmission prediction, 3% in eICU mortality prediction, 10% in MIMIC-IV readmission prediction, and 6% in MIMIC-IV mortality prediction.

4.3 Quantitative Analysis

This section provides quantitative analyses to elucidate the performance enhancements achieved by SLDG . The analyses encompass the evaluation of clustering results, ablation studies on the clustering algorithm, the impact of the number of clusters and iterations, and a runtime comparison.

Evaluation of clustering results. First, we evaluate the domain recovery ability of DG methods that do not rely on domain IDs, namely MMLD[29], DRA [14], and the proposed SLDG . Since the actual latent domain categorizations are unavailable, we assess the separability of the learned clustering results with the Silhouette score [40]. Note that the reported Silhouette score is calculated on the testing set, while the hyper-parameters are chosen based on the Silhouette score on the training set. As depicted in Fig. 3, DRA achieves the lowest score, which aligns with expectations as it solely learns latent domain categorizations through multi-head networks without explicit clustering. In contrast, MMLD generates more distinct clusters due to its iterative clustering and training setup.

eICU

Readmission

eICU

Mortality

MIMIC-IV

Readmission

MIMIC-IV

Mortality

0.05

0.10

0.15

0.20

0.25

0.30

MMLD

DRA

SLDG

However, it still necessitates manual specification of the number of clusters. In comparison, SLDG obtains the highest score using an automated hierarchical clustering technique.

Influence of the clustering algorithm. Next, we assess the influence of different clustering algorithms. Naive k-Means and GMM require manual specification of the number of clusters, which we set to the same value as SLDG 's. The results can be found in Tab. 2. We observe that naive k-Means and GMM achieve similar perform similarly to the best baseline methods in Tab. 1. This outcome is reasonable since the success of SLDG relies on both the accurate discovery of latent domains and customized models for each domain. Naive clustering techniques often fail to identify subtle yet important latent domains. In contrast, SLDG , utilizing the automatic hierarchical clustering technique, achieves the highest score.

Influence of the number of clusters and iterations.

Next, we analyze the impact of the number of clusters and iterations on the eICU readmission prediction task. We also compare our results with the best-performing baseline, MMLD [29]. The results can be found in Fig. 4. The upper panel of the figure shows that the model's performance initially improves with an increasing number of clusters. This improvement can be attributed to the finer granularity of clustering, which enables better identification of domains and customization of experts. However, as the number of clusters continues to increase, the model's performance starts to decline. This decline is caused by the growing number of model parameters, making training more challenging and leading to overfitting on suspicious samples. Similarly, the trend observed in the lower panel of the figure for the number of iterations aligns with the number of clusters. The performance initially improves as

5

10

25

40

50

Number of clusters

0.14

0.16

0.18

0.20

AUPRC

1

2

3

4

5

Number of iterations

0.14

0.16

0.18

0.20

AUPRC

SLDG

MMLD

dl4health.org the number of iterations increases, allowing the model to learn more from the data. However, after a certain point, the model starts overfitting on specific clusters, leading to decreased performance.

Runtime comparison. Lastly, we compare the training time of SLDG with the naive Base baseline. All runtimes are measured on a single NVIDIA A6000 GPU. The results can be found in Tab. 3. The use of the UMAP [30] dimensionality reduction technique enables SLDG to perform clustering quickly. As a result, the training time overhead of SLDG is reasonably low overall (18% on eICU and 20% on MIMICIV) compared to the significant performance improvement achieved (up to 79% relative improvement on AUPRC score on eICU and up to 15% on MIMIC-IV).

4.4 Case Study

For the case study, we first examine the recovered domains within each latent space. The results are presented in the left panel of Fig. 5. It is evident that the common events identified within the same domains are consistent. For instance, propofol , lorazepam , and fentanyl is frequently used together, serving the purpose of anesthesia (pre-surgery), sedation (in-surgery), and pain management (post-surgery). Furthermore, the right panel of Figure 5 illustrates a visualization of the learned domain similarity, i.e., the distances between the domain prototypical weights { w t,k } K t k =1 . Notably, our method ( SLDG ) successfully captures meaningful relationships among the latent domains. For instance, a strong relationship is learned between pneumonia and sepsis . In clinical practice, when pneumonia is severe or if the infection spreads beyond the lungs, it can enter the bloodstream and trigger a systemic response, leading to sepsis [8]. Another example is the observed strong association between cardiac arrest and infarction . In practice, if a significant portion of the heart muscle is damaged, it can disrupt the heart's electrical system, potentially leading to cardiac arrest [19].

5 Related Work

Domain Generalization The goal of DG is to learn a model using data from multiple source domains in order to achieve effective generalization to a distinct target domain [7]. To achieve this, domain alignment approaches try to match the feature distributions among multiple source domains with techniques such as moments minimization [32, 25], contrastive learning [31], adversarial learning [25, 45], regularizers [4, 24], and augmentation [62, 50, 44, 60]. Meta-learning frameworks have also been utilized to simulate new domain scenarios during training [23, 27, 26, 5, 22, 29]. Additionally, domain-specific model ensemble techniques have been employed [9, 42, 43, 61]. However, these conventional DG methods assume the availability of domain IDs, which may not be feasible in healthcare settings where patients can belong to numerous unobserved domains.

Recent advancements in DG have attempted to alleviate the reliance on domain IDs [29, 62, 14, 10, 33]. MMLD[29] is the most relevant prior work to ours. It simultaneously discovers latent domains and learns domain-invariant features through adversarial learning. However, it focuses on training a

single model that generalizes across all domains. Given that patients from different domains exhibit distinct characteristics and require different treatment approaches [2, 58], training a single model for all domains poses challenges and can result in sub-optimal performance.

Clinical Predictive Modeling. The main objective of clinical predictive modeling is to predict the occurrence of future events, such as 15-day hospital readmission and 90-day mortality, based on existing patient information. Deep learning models have been widely used in clinical predictive modeling with EHR data [55, 57]. These models are designed to capture temporal patterns in patient data [11, 36, 6, 28], model structural information in medical codes [13, 52], augment the model using pre-training [39], or leverage patient similarities for better decision making [58, 2]. However, these models typically assume an unchanged test domain and may suffer from degraded performance with domain shift. To address this issue, AutoMap [53] solves the feature space shift issue by learning an auto-mapping function without considering any distribution shift. MedLink [54] aggregates de-identified patient data from different sites to enable joint training. ManyDG [56] tackles patient covariate shift by treating each patient as a unique domain and disentangling domain variant and invariant features. However, maintaining a large number of domains is unnecessary, as similar patients often exhibit similar clinical behavior and can share a common domain.

6 Conclusion

Clinical predictive models often exhibit degraded performance when applied to data from new regions or future periods due to distribution shifts. To address this, we propose SLDG , a self-learning framework that iteratively identifies decoupled domains and trains customized classifiers for each domain. We evaluate SLDG on two medical datasets, and our results show that it outperforms all baseline methods. In addition, we provide detailed qualitative analyses and case studies to support our findings.

Acknowledgments

This work was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479. This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center.

References

Davies Adeloye, Stephen Chua, Chinwei Lee, Catriona Basquill, Angeliki Papana, Evropi Theodoratou, Harish Nair, Danijela Gasevic, Devi Sridhar, Harry Campbell, Kit Yee Chan, Aziz Sheikh, and Igor Rudan. Global and regional estimates of COPD prevalence: Systematic review and meta-analysis. Journal of global health , 5(2):020415, December 2015. ISSN 2047-2978 2047-2986. doi: 10.7189/jogh.05.020415. Place: Scotland.

Henrique Aguiar, Mauro Santos, Peter Watkinson, and Tingting Zhu. Learning of cluster-based feature importance for electronic health record time-series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 161-179. PMLR, 17-23 Jul 2022. URL https://proceedings. mlr.press/v162/aguiar22a.html .

Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop , pages 72-78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-1909. URL https://aclanthology.org/W19-1909 .

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2020.

Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/ paper_files/paper/2018/file/647bba344396e7c8170902bcf2e15551-Paper.pdf .

Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou. Patient subtyping via time-aware lstm networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '17, page 65-74, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/ 3097983.3097997. URL https://doi.org/10.1145/3097983.3097997 .

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011. URL https://proceedings.neurips.cc/paper_files/ paper/2011/file/b571ecea16a9824023ee1af16897a582-Paper.pdf .

Klaus F. Bodmann. Current guidelines for the treatment of severe pneumonia and sepsis. Chemotherapy , 51(5):227-233, Aug 2005. doi: 10.1159/000087452. URL https://doi. org/10.1159/000087452 .

Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 22405-22418. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ bcb41ccdc4363c6848a1d760f26c28a0-Paper.pdf .

Zhihong Chen, Taiping Yao, Kekai Sheng, Shouhong Ding, Ying Tai, Jilin Li, Feiyue Huang, and Xinyu Jin. Generalizable Representation Learning for Mixture Domain Face Anti-Spoofing. Proceedings of the AAAI Conference on Artificial Intelligence , 35(2):1132-1139, May 2021. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v35i2.16199. URL https://ojs.aaai.org/ index.php/AAAI/article/view/16199 .

Edward Choi, Mohammad Taha Bahadori, Joshua A. Kulas, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 3512-3520, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Using recurrent neural network models for early detection of heart failure onset. Journal of the American Medical Informatics Association , 24(2):361-370, 08 2016. ISSN 1067-5027. doi: 10.1093/jamia/ocw112. URL https://doi.org/10.1093/jamia/ocw112 .

Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, and Jimeng Sun. Gram: Graph-based attention model for healthcare representation learning. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '17, page 787-795, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/3097983.3098126. URL https://doi.org/10.1145/ 3097983.3098126 .

Lucas Deecke, Timothy Hospedales, and Hakan Bilen. Latent domain learning with dynamic residual adapters, 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1126-1135. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/finn17a.html .

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. J. Mach. Learn. Res. , 17(1):2096-2030, jan 2016. ISSN 1532-4435.

Lin Lawrence Guo, Stephen R. Pfohl, Jason Fries, Alistair E. W. Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific Reports , 12(1):2726, February 2022. ISSN 2045-2322. doi: 10.1038/s41598-022-06484-1. URL https://www.nature.com/articles/s41598-022-06484-1 .

Alistair E. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data , 10(1):1, January 2023. ISSN 2052-4463. doi: 10.1038/s41597-022-01899-x. URL https://doi.org/10.1038/s41597-022-01899-x .

J. Kim, E. So, H. J. Kim, K. S. Seo, and M. H. Karm. Cardiac arrest due to an unexpected acute myocardial infarction during head and neck surgery: A case report. J Dent Anesth Pain Med , 18(1):57-64, 2018. doi: 10.17245/jdapm.2018.18.1.57. URL https://doi.org/10.17245/ jdapm.2018.18.1.57 .

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5637-5664. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/koh21a.html .

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'18/IAAI'18/EAAI'18. AAAI Press, 2018. ISBN 978-1-57735-800-8.

Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11682. URL https://ojs.aaai. org/index.php/AAAI/article/view/11682 .

Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3915-3924. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/li19l.html .

Quande Liu, Qi Dou, and Pheng-Ann Heng. Shape-aware meta-learning for generalizing prostate mri segmentation to unseen domains. In Anne L. Martel, Purang Abolmaesumi, Danail Stoyanov, Diana Mateus, Maria A. Zuluaga, S. Kevin Zhou, Daniel Racoceanu, and Leo Joskowicz, editors, Medical Image Computing and Computer Assisted Intervention - MICCAI 2020 , pages 475-485, Cham, 2020. Springer International Publishing. ISBN 978-3-030-597139.

Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. Dipole: Diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '17, page 1903-1911, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/3097983.3098088. URL https: //doi.org/10.1145/3097983.3098088 .

Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. Proceedings of the AAAI Conference on Artificial Intelligence , 34(07):1174911756, Apr. 2020. doi: 10.1609/aaai.v34i07.6846. URL https://ojs.aaai.org/index. php/AAAI/article/view/6846 .

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software , 3(29):861, 2018.

Saeid Motiian, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , Oct 2017.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 , ICML'13, page I-10-I-18. JMLR.org, 2013.

Li Niu, Wen Li, and Dong Xu. Visual recognition by learning from web data: A weakly supervised domain generalization approach. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2774-2783, Boston, MA, USA, June 2015. IEEE. ISBN 978-1-4673-6964-0. doi: 10.1109/CVPR.2015.7298894. URL http://ieeexplore.ieee. org/document/7298894/ .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

X. Peng, G. Long, T. Shen, S. Wang, J. Jiang, and C. Zhang. Bitenet: Bidirectional temporal encoder network to predict medical outcomes. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 412-421, Los Alamitos, CA, USA, nov 2020. IEEE Computer Society. doi: 10.1109/ICDM50108.2020.00050. URL https://doi.ieeecomputersociety.org/ 10.1109/ICDM50108.2020.00050 .

Christian S. Perone, Pedro Ballester, Rodrigo C. Barros, and Julien Cohen-Adad. Unsupervised domain adaptation for medical imaging segmentation with self-ensembling. NeuroImage , 194: 1-11, 2019. ISSN 1053-8119. doi: https://doi.org/10.1016/j.neuroimage.2019.03.026. URL https://www.sciencedirect.com/science/article/pii/S1053811919302034 .

Tom J. Pollard, Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific Data , 5(1):180178, September 2018. ISSN 2052-4463. doi: 10.1038/sdata.2018.178. URL https://doi.org/10.1038/sdata.2018.178 .

Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine , 4(1):86, May 2021. ISSN 2398-6352. doi: 10.1038/s41746-021-00455-y. URL https://doi.org/10.1038/s41746-021-00455-y .

Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics , 20:53-65, 1987. ISSN 0377-0427. doi: https://doi.org/10.1016/0377-0427(87)90125-7. URL https://www.sciencedirect. com/science/article/pii/0377042787901257 .

Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation , 33(11):2881-2907, 2021.

Mattia Segu, Alessio Tonioni, and Federico Tombari. Batch normalization embeddings for deep domain generalization. Pattern Recognition , 135:109115, 2023. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2022.109115. URL https://www.sciencedirect.com/ science/article/pii/S0031320322005957 .

Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 , pages 68-83, Cham, 2020. Springer International Publishing. ISBN 978-3-03058542-6.

Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum? id=r1Dx7fbCW .

Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C. Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019.

Nimit Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Yanchao Tan, Chengjun Kong, Leisheng Yu, Pan Li, Chaochao Chen, Xiaolin Zheng, Vicki S. Hertzberg, and Carl Yang. 4sdrug: Symptom-based set-to-set small and safe drug recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 3970-3980, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393850. doi: 10.1145/3534678.3539089. URL https://doi.org/10.1145/3534678.3539089 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Claus F. Vogelmeier, Kenneth R. Chapman, Marc Miravitlles, Nicolas Roche, Jørgen Vestbo, Chau Thach, Donald Banerji, Robert Fogel, Francesco Patalano, Petter Olsson, Konstantinos Kostikas, and Jadwiga A. Wedzicha. Exacerbation heterogeneity in COPD: subgroup analyses from the FLAME study. International journal of chronic obstructive pulmonary disease , 13: 1125-1134, 2018. ISSN 1178-2005 1176-9106. doi: 10.2147/COPD.S160011. Place: New Zealand.

Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 1d94108e907bb8311d8802b48fd54b4a-Paper.pdf .

Ziqi Wang, Marco Loog, and Jan van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 9756-9763, 2021. doi: 10.1109/ICPR48806.2021.9412797.

Zhenbang Wu, Huaxiu Yao, Zhe Su, David M Liebovitz, Lucas M Glass, James Zou, Chelsea Finn, and Jimeng Sun. Knowledge-driven new drug recommendation, 2022.

Zhenbang Wu, Cao Xiao, Lucas M. Glass, David M. Liebovitz, and Jimeng Sun. Automap: Automatic medical code mapping for clinical prediction model deployment. In Massih-Reza Amini, Stéphane Canu, Asja Fischer, Tias Guns, Petra Kralj Novak, and Grigorios Tsoumakas, editors, Machine Learning and Knowledge Discovery in Databases , pages 505-520, Cham, 2023. Springer International Publishing. ISBN 978-3-031-26390-3.

Zhenbang Wu, Cao Xiao, and Jimeng Sun. Medlink: De-identified patient health record linkage. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 2672-2682, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305.3599427. URL https: //doi.org/10.1145/3580305.3599427 .

Chao Xiao, Edward Choi, and Jimeng Sun. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. Journal of the American Medical Informatics Association , 25(10):1419-1428, Oct 2018. doi: 10.1093/jamia/ ocy068.

Chaoqi Yang, M Brandon Westover, and Jimeng Sun. ManyDG: Many-domain generalization for healthcare applications. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=lcSfirnflpW .

Chaoqi Yang, Zhenbang Wu, Patrick Jiang, Zhen Lin, Junyi Gao, Benjamin P. Danek, and Jimeng Sun. Pyhealth: A deep learning toolkit for healthcare applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '23, page 5788-5789, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10.1145/3580305.3599178. URL https://doi.org/10.1145/ 3580305.3599178 .

Chaohe Zhang, Xin Gao, Liantao Ma, Yasha Wang, Jiangtao Wang, and Wen Tang. Grasp: Generic framework for health status representation learning based on incorporating knowledge from similar patients. Proceedings of the AAAI Conference on Artificial Intelligence , 35(1): 715-723, May 2021. doi: 10.1609/aaai.v35i1.16152. URL https://ojs.aaai.org/index. php/AAAI/article/view/16152 .

Haoran Zhang, Natalie Dullerud, Laleh Seyyed-Kalantari, Quaid Morris, Shalmali Joshi, and Marzyeh Ghassemi. An empirical framework for domain generalization in clinical settings. In

Proceedings of the Conference on Health, Inference, and Learning , pages 279-290, Virtual Event USA, April 2021. ACM. ISBN 978-1-4503-8359-2. doi: 10.1145/3450439.3451878. URL https://dl.acm.org/doi/10.1145/3450439.3451878 .

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 , pages 561-578, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58517-4.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing , 30:8008-8018, 2021. doi: 10.1109/TIP.2021.3112012.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations , 2021. URL https://openreview. net/forum?id=6xHJ37MVxxp .

Contents of Appendix

A Pseudocode of SLDG

B Additional Experimental Setup

B.1 Datasets

For both datasets, we select our cohorts by filtering out visits of patients younger than 18 or older than 89 years old, visits that last longer than 10 days, and visits with data from less than 3 or more than 256 timestamps. In the case of the eICU dataset, we additionally exclude visits lasting shorter than 12 hours, as the predictions are made 12 hours after admission. Similarly, for the MIMIC-IV dataset, we exclude visits where the patient ultimately passed away, as the predictions are made upon discharge. Tab. 4 provides detailed statistics of the two datasets.

B.2 Clinical Predictive Tasks

We focus on two common clinical predictive tasks: readmission prediction and mortality prediction.

In the case of the eICU dataset, the predictions are made 12 hours after admission. Readmission prediction aims to determine whether a patient will be readmitted within the next 15 days following discharge. Mortality prediction, on the other hand, aims to predict whether a patient will pass away upon discharge. The overall prevalence for these tasks is 15% for readmission and 4% for mortality.

For the MIMIC-IV dataset, the predictions are made at the time of discharge. Similar to the eICU dataset, the readmission prediction task is defined as predicting whether a patient will be readmitted within 15 days after discharge. To prevent information leakage, the mortality prediction task for MIMIC-IV is defined as predicting whether a patient will pass away within 90 days after discharge. The overall prevalence for these tasks is 14% for readmission and 4% for mortality.

B.3 Data Split

The eICU dataset comprises data collected from hospitals across the United States, while the MIMICIV dataset spans a period of ten years. Therefore, we utilize the eICU dataset to evaluate the model's performance across spatial gaps, and the MIMIC-IV dataset to assess its performance across temporal gaps.

For the eICU dataset, we divide it into four spatial groups based on regions: Midwest, Northeast, West, and South. Each group is then split into 70% for training, 10% for validation, and 20% for testing. We evaluate the gap between groups by comparing the performance of the backbone model trained on data from within the same group and data from outside the group. The target testing data is selected as the group (Midwest) that exhibits the largest performance gap, while the remaining groups (Northeast, West, and South) are used as the source training data.

Regarding the MIMIC-IV dataset, we divide it into four temporal groups: 2008-2010, 2011-2013, 2014-2016, and 2017-2019. Each group is further split into training, validation, and testing sets with a ratio of 70%, 10%, and 20% respectively. We consider patients admitted after 2014 as the target testing data, while all preceding patients are included in the source training data.

B.4 Baselines

We first compare SLDG to two naive baselines.

Oracle: We directly train a backbone model on the training set of the target domain, select the best model on the target validation set, and evaluate its performance on the target testing set. This model is trained with in-domain data and can be seen as a upper bound for all domain generalization method.

Base: We train a backbone model on the training set of the source domain, select the best model on the source validation set, and evaluate its performance on the target testing set. This model is trained with out-domain data and should act as a performance lower bound.

We then compare SLDG to both classic and recent domain generalization methods. For a fair comparisons, all the methods below are trained on the source training set, selected on the source validation set, and tested on the target testing set.

DANN [16]: Domain-Adversarial Neural Networks leverage a domain classifier and a gradient reversal layer to extract domain-invariant representations. This method uses the coarse regional and temporal groups as the domain definition.

MLDG[22]: Meta-Learning for Domain Generalization adopts the Model-Agnostic Meta-Learning (MAML) [15] framework and simulates the new domain scenario during training. This method also uses the coarse regional and temporal groups as the domain definition.

ManyDG [56]: Many-Domain Generalization disentangles domain-variant and invariant features through mutual reconstruction and orthogonal projection. This method treats each patient as a unique domain.

IRM [4]: Invariant Risk Minimization learns domain-invariant representations by minimizing a bound on the expected generalization error under domain shifts. It acts as a regularizer and does not require domain IDs.

MMLD [29]: Domain Generalization using a Mixture of Multiple Latent Domains iteratively assigns pseudo domain labels via clustering and trains the domain-invariant feature extractor through adversarial learning. This method does not rely on domain IDs.

DRA [14]: Latent Domain Learning with Dynamic Residual Adapters uses layer-wise multi-head correction networks with a gating mechanism and residual connection to enhance model learning. This method does not rely on domain IDs.

B.5 Implementation Details

For all baselines, we use the Transformer as the backbone encoder. The number of layers is 3, the embedding dimension is 128, the number of attention heads is 2. The event embedding look-up table is initialized with ClinicalBERT [3] embeddings of the event name and then project it down to 128 dimension with a linear layer. Patient demographics features (age, gender, and ethnicity) are separately embeded with another embedding look-up table. We also embed the timestamps with sinusoidal positional encoding. The medical, patient demographics, and temporal embeddings are added together to form the overall sequence embedding. For SLDG , UMAP [30] from UMAPlearn [41] is used with 2 components, 10 neighbors, and 0 minimum distance; and k-Means from Scikit-learn [35] is used with the default hyper-parameter. We apply a dropout of rate 0.2. We use Adam as the optimizer with a learning rate of 1e-4 and a weight decay of 1e-5. All models are trained for 100 epochs. The batch size is 256. We select the best model by monitoring the AUPRC score on the source validation set (except for the Oracle baseline, where we directly use the target validation set). We implement SLDG using PyTorch [34] 1.11 and Python 3.8. The model is trained on a CentOS Linux 7 machine with 128 AMD EPYC 7513 32-Core Processors, 512 GB memory, and eight NVIDIA RTX A6000 GPUs.

C Limitations and Broader Impacts

In terms of limitations, it is important to acknowledge that our work operates under the assumption that the target testing data still exhibit some similarities with the source training data. If there is

a significant distribution shift, the knowledge acquired from the source training data may become irrelevant. In such cases, neither the DG baselines nor our proposed method can effectively address the problem. It would be more appropriate to explore transfer learning or train a new model to obtain a better solution. Further, we propose SLDG to tackle two main challenges: (1) unknown domain IDs and (2) distinct characteristics across domains. In the scenario when the domain IDs are given and clearly separable (e.g., photo, art painting, cartoon, and sketch in the PACS [21] dataset), SLDG 's domain discovery approach might be unnecessary. Existing DG methods directly utilizing the domain IDs could be a better solution.

In terms of broader impacts, our work tackles a practical and prevalent issue in healthcare known as the domain shift problem. We aim to inspire future research in this area: both by investigating the existence of domain shift under various scenarios, and by contributing to the development of effective solutions for this real-world challenge.

D Notations

E Additional Illustrations

Patient id: 12853

Gender: M

Age: 77

•

(15 Min, Diagnosis)

Atrial fibrillation (I48.91)

•

(24 Min, Diagnosis)

Respiratory failure (J96. 90)

•

(25 Min, Procedure)

Respiratory ventilation (E0466)

•

(60 Min, Procedure)

ECG (93042)

•

(93 Min, Procedure)

CT scan (G0296)

•

(132 Min, Medication)

•

Discharge

eICU Dataset

ICU Admission

ICU Discharge

Prediction Time

12 Hours

Whether the patient will decease at the end of current ICU stay?

Whether the patient will be readmitted to ICU within 14 days?

MIMIC-IV Dataset

Hospital

Admission

Hospital Discharge

& Prediction Time

Whether the patient will decease within 90 days?

Whether the patient will be readmitted to hospital within 14 days?

Three-Layer Transformer

𝑒

!

𝑒

"

…

𝑒

#

Patient V

isit	x

𝒆

!

𝒆

"

…

𝒆

#

Decoupled Patient Representations

Type-Specific Aggregation

𝑒

$

𝒆

$

𝒉

%

…

𝒉

!

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present SLDG: a framework for dealing with domain shifts in clinical event data. They propose two steps in this framework. First, an unsupervised latent-domain clustering which automatically clusters different subjects into different feature-level clusters via monitoring the Silhouette score for optimal clusters, followed by individually trained classifiers for each cluster group, where the final probabilistic prediction is the weighted combination of the predictions from within each cluster group. The individually trained classifier is the softmax-ed distance between a patient's latent representation and the prototypical representations for the latent space. The authors find that by training this method by reinitializing the clusters every set number of epochs they can beat existing domain adaptation methods.\n\nSTRENGTHS:\n- The idea and method are communicated in a clear manner and provide a good understanding of the underlying problem.\n- The method has clear benefits over baseline methods, sometimes with large jumps in improvement (Readmission on eICU).\n- The work does offer a unique and interesting method for self-supervised clustering of latent features. I really liked the proposed method and thought that it was well explained.\n- The ablation of the clustering algorithm demonstrates that the added complexity of the hierarchical clustering indeed shows benefits over traditional methods, something which other works could potentially build on top of.\n\nWEAKNESSES:\n- Customized classifiers for each domain are a very inefficient method of scaling to new domains even with the efficient parameterization of each domain. Although, this is definitely more problematic if the latent dimension is larger or smaller and there doesn't seem to be any kind of ablation for the size of the classifier dimension. I believe this might be a useless result to have.\n- The formulation of the prototypical vectors I don't think is very well motivated and is confusing why there is a positive and negative representation of each latent space t. Additionally, I'm not exactly sure why it would make sense to average the predictions from all latent domains to make new predictions given that for the downstream prediction task, not all clusters are probably equally helpful with making predictions. This seems like it would introduce a lot of noise into predictions.\n- The biggest weakness I find with the analysis of this work is that it is unclear how large the baseline methods are in comparison to the proposed method. The paper lacks an analysis of parameter numbers which is problematic because as you develop more clusters, it is my understanding that you will also balloon up your parameter count, but also we don't know how big the transformer architecture is in comparison to other baseline methods.\n\nQUESTIONS:\n- What is meant on line 121 with the embedding lookup table?\n- What is the intuition for using sinusoidal positional embeddings? I mean to say, why does it make sense here to embed timestamps with sinusoidal positional encodings when the patient visits would only be going forwards in time? Maybe I misunderstood this piece and would need some clarification.\n- The method for reducing the clusters from exponential to linear makes sense, but shouldn't the label types be coupled to some degree? Many of those features shouldn't be independent of each other so making an independence assumption is strange.\n- Related to my point of weakness with evaluation, I would definitely consider raising my score if there was additional necessary context concerning the capacity of the proposed method (with additional parameters per clustering included) in comparison to the baseline methods. I think that the clustering method is sufficiently novel and interesting that it would be helpful for others to know about, the only thing keeping my score at a borderline except at the moment is that I am unsure if the comparison to the baselines is fair.\n\nLIMITATIONS:\nThe authors don't explicitly discuss the potential negative social impact of their work in the main body of the text, right now it is in the appendix, and I believe that it should be moved to the main body. It does address, however, the limitations of the work and places it meaningfully in context.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe authors present a novel self-learning framework, sldg, designed to address the domain shift problem in deep learning models, particularly in the context of clinical decision-making. the authors posit that reliance on domain ids is a key limitation in existing domain generalization algorithms. to overcome this, the sldg framework iteratively decouples domains and trains personalized classifiers for each domain. the results demonstrate improvements in the auprc score over baselines.\n\nedit august 19, 2023: increase score from 5 to 6 in light of authors' rebuttal.\n\nSTRENGTHS:\n1. **innovative approach to domain generalization**: the sldg framework introduces a novel way to handle the domain shift problem in deep learning models. unlike existing methods that rely on domain ids, sldg iteratively discovers decoupled domains and trains personalized classifiers for each domain. the approach is clearly explained.\n\n2. **significant performance improvement**: the paper demonstrates that the sldg framework can achieve a substantial improvement in the auprc score over the best baseline, indicating the promise of the proposed method.\n\n3. **applicability in healthcare settings**: the authors have focused on the application of their model in clinical decision-making, a field where domain shift is a common issue. the ability of the sldg framework to handle potential domain shifts when applied to target data is a significant strength, as it makes the model highly relevant and potentially impactful in real-world healthcare settings.\n\n4. **comprehensive technical details**: the inclusion of a detailed pseudocode for the sldg algorithm in the appendix is a strength. it provides a clear understanding of the steps involved in training and inference, which can be beneficial for researchers and practitioners looking to implement or build upon this work.\n\n5. **robust testing**: the authors have tested their algorithm on two different datasets for readmission and mortality prediction tasks. this robust testing adds credibility to the results and demonstrates the versatility of the algorithm. the filtering of datasets to exclude certain visits also shows a thoughtful approach to ensuring the quality and relevance of the data used.\n\nWEAKNESSES:\n1. **domain discovery process**: the paper presents an iterative process for decoupled domain discovery (section 3.2). a detailed explanation of how the algorithm ensures the discovered domains are meaningful and distinct from each other. a more thorough discussion or a visual representation of the discovered domains could enhance the understanding of this process.\n\n2. **baseline comparison**: the paper mentions that sldg outperforms the best baseline (section 4.1). however, many related approaches are not mentioned and no explanation is provided why they should be discarded or how they relate to sldg. including more comparisons, empirically or at least conceptually, with state-of-the-art models could strengthen the paper's claims about sldg's novelty. a good general overview can be found at pwc https://paperswithcode.com/task/domain-generalization.\n\n3. **algorithm complexity**: the pseudocode provided in the appendix outlines the steps of the sldg algorithm, but the paper does not discuss the computational complexity of these steps beyond base and sldg. providing runtime information for the other dgs as well as flops or macs information could help readers understand the feasibility of implementing sldg in practice.\n\n4. **generalizability**: while the paper demonstrates the effectiveness of sldg in healthcare settings, it does not discuss its applicability to other fields. a discussion on how the proposed method could be adapted for other domains would increase the paper's impact and reach. similarly, the title suggests application to medicine broadly, while experiments are performed on ehr data only.\n\n5. **relation to connected related fields**: the authors do not discuss in detail and only briefly at the end how the proposed sldg framework relates to or differs from other relevant techniques. some important ones, such as federated learning and disentangled representation learning, are not mentioned at all despite similar problems having been tackled in these fields (see for example https://ieeexplore.ieee.org/abstract/document/9174890 or https://aclanthology.org/P19-1041/, among many other works*). given that these methods also deal with domain generalization, often without domain ids such as in private distributed clients, a comparison or discussion could provide valuable context and highlight the unique contributions of sldg.\n\n*i have no affiliation to examples of related work\n\nQUESTIONS:\n1. can you elaborate on the process of decoupled domain discovery in sldg?\n\n2. how could the meaningfulness of each domain sldg identifies be evaluated more thoroughly?\n\n3. why were certain related approaches not included in the comparison with sldg?\n\n4. could you provide a more comprehensive contextualization with existing dg methods?\n\n5. could you provide more details on the computational complexity of sldg vis-a-vis the other dg baselines you employ in your experiments?\n\n6. the experiments seem focused on ehr data, but the title suggests broader applicability. could you clarify?\n\n7. how would sldg be adapted for domains beyond ehr?\n\n8. how does sldg relate to techniques like federated learning with clients that have non-iid data?\n\n9. how does sldg relate to disentangled representation learning?\n\n10. these methods also tackle domain generalization, often without domain ids. could you discuss any potential connections or distinctions?\n\nLIMITATIONS:\nit would be great to discuss more connections to related work from connected disciplines working on similar problems and include a clear discussion of limitations in the main text, which is missing at the moment.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for domain generalization in healthcare, named Subspace Latent Domain Generalization (SLDG). The authors argue that traditional domain generalization methods are not suitable for healthcare applications due to the heterogeneity of patient data. The SLDG method discovers latent domains in patient data and uses them to improve the performance of predictive models. The authors provide empirical evidence to support their claims, demonstrating that the SLDG method outperforms several other methods in predicting patient readmission and mortality on two datasets, eICU and MIMIC-IV.\n\nSTRENGTHS:\n* Novel Approach: the paper presents a novel method (SLDG) for domain generalization in healthcare. This method, which discovers latent domains in patient data, could potentially improve the performance of predictive models in healthcare, where data is often heterogeneous and collected from various sources.\n* Performance: the proposed method surpassed several other strategies on both readmission and mortality for both datasets used for evaluation.\n* Method description: the proposed approach is extensively described with the provided pseudocode. Similarly the methodology employed is properly explained in the paper, with additional material containing the specificities of each section.\n\nWEAKNESSES:\nEven though it performed well on the two evaluated datasets, it isn't clear how it would perform on smaller datasets with large amounts of missing data. It is possible the method won't perform so well in a dataset that is smaller than the ones where it was evaluated (1/10th of its size). There is a lack of discussion on the proposed method's limitations and possible challenges within the application.\n\nQUESTIONS:\n* How does the SLDG method handle situations where the latent domains are not clearly separable? Could this affect the performance of the method?\n* Was the model evaluated in smaller datasets? What are the scenarios in which this method should or should not be applied?\n\nLIMITATIONS:\nThe authors have provided a clear and detailed description of their method and have presented empirical evidence to support their claims. However, they could further discuss the limitations of their method, particularly in terms of the generalizability of the results. The paper focuses on two specific datasets, and it's unclear how well the method would perform on other datasets or in other healthcare contexts. The authors could also discuss the potential challenges of applying the SLDG method in practice, such as the need for large amounts of high-quality data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a transformer-based supervised learner (SDLG) for two healthcare domain adaptation adaptation tasks (one across geographic region, and one across time).  The pipeline consists of feature sequence embeddings into type-specific vector spaces via fine-tuned contextualized embedding approach, clustered and subclustered, and then passed through a distance-based gating layer.  The model outperform multiple methods from the domain adaptation literature in two electronic health records datasets.\n\nI have read the rebuttal, and while the clarifying comments are elucidating, I maintain my reservations about the work in its current form.\n\nSTRENGTHS:\nOriginality: The machine learning modeling approach combines multiple standard models in deep learning and uses appropriate fine-tuning methods.  It is applied to an extract of challenging real world (EHR) data.\n\nQuality: The figures (1,2) and equations effectively communicate the technical approach.\n\nClarity: The paper is mostly clear and the descriptions are straightforward to follow.\n\nSignificance: The results suggest substantial improvement over other baseline models, suggesting that, at least for these tasks, this decoupled approach works well when generalizing to extracts from clinical event sequences in a second domain.\n\nWEAKNESSES:\nIt seems there could be a slight mismatch in the intuition of having separate latent domains and using the types shown in Fig 2. E.g. toe numbness (symptom), insulin (treatment), and history of diabetes (medical history) are all associated with the condition diabetes, but appear to be encoded separately into spaces/clusters/subclusters.\n\nHow UMAP is used is not entirely clear.  Is it performed on the training data only? If it is applied to the training and test data, is it not leaking information from the test in producing the embedding? If not, when applied to the test data, what is the mapping for test data points, and how are they assigned cluster/subcluster memberships?  The \"hierarchical clustering\" used, if as described, perhaps should be named differently as it does not appear to be the usual hierarchical clustering based on linkage (or details are missing).\n\nThe model pipeline includes many design choices, but is tested only in two particular settings.  In those settings some of the details are not entirely clear.  I was also unable to determine the list of event types used, or how the EHR data (typically thousands of measurements) were converted into sequences roughly 100 and 35 events long, respectively.  Which events/tables were used?  How the timestamps are embedded should be described precisely somewhere (e.g. is it time from first event in some unit of time?).\n\nFor eICU and given the number of sequential design choices, it seems like leave-out-one-domain cross validation would be a better measure of performance, than a single assessment for the held-out midwest region.  For MIMIC, the Appendix claims each temporal range is split into training/validation/test sets, but also says the evaluation is on patients admitted after 2014, which is confusing.\n\nFrom an applied perspective, it is unclear what AUCs are competitive or how they would influence decision making.  Are readmission AUCs of 0.62 compelling (or 0.68 for the oracle)?\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nA high-level set of limitations are described in the Appendix.  While the data used are \"real-world\", the simplistic extraction used for the event sequence, and experimental setup suggest that the review should likely focus on the methods provided rather than value for the use case.  The methods can still be improved, perhaps with the use of synthetic data where theory and/or optimality could be established.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper proposes a learning framework for improved predictive performance for EHR data under distribution shift.\n* The key problem tackled here is that many existing domain generalization algorithms rely on data points having domain identifiers, which can then be used in a learning algorithm to build a generalizable model. However, in clinical settings, patients may fall into many domains, which may also be latent (and not explicitly known), making the application of these existing methods challenging.\n* In this work, the proposed framework iterates (1) domain discovery and (2) domain-specific classifier training, to build predictors that are effective for the different domains considered.\n* This framework involves:  obtaining encoded representations for patient visits, clustering visits using a hierarchical procedure to discover domains, and building a weighted nearest neighbour styled classifier within each domain (using prototypes) to predict a label. \n* The paper evaluates the method and several baselines on two EHR datasets: eICU and MIMIC-IV. Two tasks are considered: readmission, and mortality prediction. On both datasets, the method performs well, and improves on baselines.\n* The paper also includes ablations on the number of iterations of the procedure and clusters, providing some more intuition on what drives performance.\n\nSTRENGTHS:\n* Clear presentation of the problem and limitations of existing methods e.g., the need for domain IDs, or training a single model across domains.\n* As far as I can tell, method appears to be original and also well motivated based on the problem at hand -- discovering latent regions of similarity and developing specialized classifiers for those regions.\n* Good evaluation setup: fairly comprehensive set of baselines, evaluation on two distinct datasets with varied characteristics\n* Results are encouraging (though a question/comment on this below) -- consistent improvements in AUROC/AUPRC on the two tasks on both datasets.\n* Useful additional results studying importance of clustering algorithm and understanding the nature of the latent spaces.\n\nWEAKNESSES:\n* Overall I was confused by the methodological presentation. Particularly, a diagram presenting the data analogously to the abstraction  (medical record is a set of visits, each with events) would help a lot. The dimensionality of different data/features at points in the pipeline was unclear, and a figure might help this.\n* Similar to the above, Figure 2 showing the method was useful, but I think too high level to properly internalize what was happening in the method. An architecture diagram for the encoder (for example) could help, and may address the point above too.\n* In general, Section 3.1 was tricky to understand -- perhaps this is because the high level presentation at the start was difficult to map to the more specific descriptions later (namely, the Feature specific patient encoding and hierarchical domain clustering paragraphs). Some of this confusion might stem from not understanding the input data very well, particularly the different feature types $t$ and encoder $E_t$, and embedding function $E$. A re-writing of this section might help, together with clear figures for each of the pieces (particularly the encoding section).\n* On the quantitative results: I appreciate that the proposed method improves on the different baselines -- that is encouraging to see. However, the fact that it does not improve on the Oracle method (and in some cases does substantially worse) may be something to discuss further -- why would we not just use the oracle if we have enough data from that domain?  Would finetuning the proposed method on the training set from the target domain help here (for some definition of finetuning? I appreciate it's a bit tricky here given the way the classifiers are parameterized)? In addition, could you show results when you have very limited data from the target domain (and specifically demonstrate that the Oracle no longer performs well)? As I see it, if the Oracle method that is trained on the labelled target data can do well in small target data regimes, then these results are not as convincing. \n* On evaluation of clustering results: Since the proposed method directly looks at the Silhouette score when doing clustering (and not sure the others do), I'm not sure this is very convincing -- we'd expect the proposed method to do better when evaluating Silhouette score.\n\nQUESTIONS:\n* Could you say more about the specific dimensionality and form of the input sequence that the model sees? And as above, adding a clearer figure to represent the data and the notational abstraction used in the paper. Adding figure to specify the architecture, and how the encoding phase works in more detail.\n* Can you answer my question re. the oracle, as discussed in Weaknesses? Specifically, on the proposed method being (sometimes substantially) worse than the oracle, and how to think about this?\n\nOverall, if the authors can say more about the above two points, I will increase my score.\n\n\n# After rebuttal\nAuthors have answered my main questions, so I will increase my score.\n\nLIMITATIONS:\n* Limitations and broader impact are discussed in the appendix. I think however that the limitations could be fleshed out some more, maybe in terms of acknowledging that only two predictive tasks and two datasets are considered.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}