SPONGE: Competing Sparse Language Representations for Effective Knowledge Transfer
Authors: Jens-Michalis Papaioannou
Alexei Figueroa
Conor Fallon
Anna Capilla
Alexandra Bekiaridou
Stavros Zanos
Wolfgang Nejdl
Alexander Löser
Venue: Accepted by TMLR ()

Abstract: In domains with privacy constraints, most knowledge resides in siloed datasets, hindering the development of a model with all relevant knowledge for a task. 
Clinical NLP is a prime example of these constraints in practice. 
Research in this area typically falls back to the canonical setting of sequential transfer learning, where a model pre-trained on large corpora is finetuned on a smaller annotated dataset. 
An avenue for knowledge transfer among diverse clinics is multi-step sequential transfer learning since models are more likely to be shared than private clinical data. 
This setting poses challenges of cross-linguality, domain diversity, and varying label distributions which undermine generalisation. 
We propose SPONGE, an efficient prototypical architecture that leverages competing sparse language representations. 
These encompass distributed knowledge and create the necessary level of redundancy for effective transfer learning across multiple datasets. 
We identify that prototypical classifiers are critically sensitive to label-recency bias which we mitigate with a novel strategy at inference time. SPONGE in combination with this strategy significantly boosts generalisation performance to unseen data. 
With the help of medical professionals, we show that the explainability of our models is clinically relevant. 
We make all source code available.

Alexandra Bekiaridou Published in Transactions on Machine Learning Research (12/2025)
SPONGE: Competing Sparse Language Representations for Effective Cross-Lingual Knowledge Transfer in Healthcare
Jens-Michalii Papaioannou"
michalis. Segialasppppiionnnnn@@@hhtheh..
Berlin University of Applied Sciences and Technology
Leibniz University Hannover
Alexci Figucroa Rosero"
Berlin University of Applied Sciences and Technology
Leibniz University Honnover
Conor Fallon
efallon@bht-brrinn...
Berlin University of Applied Sciences and Technology
Anna Capilla
ornacapila@gmaailoco com
Independent Researcher, Berlin, Germany
abekiaridon.@nartwwelll.cm
Peinstein Instibutes for Medical Research, Northowlll Health
Stavros Zanos
saanos@northwwell..cm
Peinstein Institutes for Medical Research, Northwell Health
Wolfgang Nejdl
nejdI de
Leibniz University Hannover
Alexander Löser Berlin University of Applied Sciences and Technology
Aloeser@blt-ben lin de
Reviewed on OpenReview: https: // openreview. net/forunts id- DevFdPgk3h
"Equal contribution.
Abstract
In domains with privacy constraints, most knowledge resides in siloed datasets, hindering the development of a model with all relevant knowledge for a task. Clinical NL.P is prime example of these constraints practice. Research this area typically falls back to the canonical setting of sequential transfer learning, where a model pre-trained on large corpora finetuned on a smaller annotated dataset. An avenue for knowledge transfer among diverse clinics is multi-step sequential transfer learning since model!] are more likely to be shared Ihan private clinical data. This setting poses challenges of cross-linguality, domain diversity, and varying label distributions which undermine generalisation. We propose SPONGE an efficient prototypical architecture that leverages competing sparse language representations. These encompass distributed knowledge and create the necessary level of redundancy for effective transfer learning across multiple datasets. We identify that prototypical classifiers
Hugginglace blomedical models

---

Published in Transactions on Machine Learning Research (12/2025)
are critically sensitive to label-recency bias which we mitigate with novel strategy at infer- ence time. SPONGE in combination with this strategy significantly boosts generalisation performance to unseen data. With the help of medical professionals, we show that the explainability of our models is clinically relevant. We make all source code available.
1 Introduction
In real-world machine learning applications, access to labeled data is often limited, whether due to intellectual property reasons or privacy constraints. Clinical Natural Language Processing (NLP) tasks exemplify this challenge, where these limitations hinder the unification of all medical knowledge. A common approach to handling decentralised and private data is Federated Learning (FL). However, FL performance suffers when data are non-i.id. (Nguyen et al. 2023), which is prevalent for most of the clinically relevant outcome classes (diagnoses, medications, procedures). Sequential transfer learning (Howard & Ruder 2018) is well- established and simple alternative to FL when model checkpoints, rather than data, are used for knowledge transfer. Furthermore, this is particularly relevant. for the case of NTP models of the BERT family, since attacks on extracting their training data are significantly less successful than with generative models (GPT lamily) (Lehman al. 2021 Vakili & Dalianis) 2021 Huang et al., 2022), , therefore preserving privacy better. Sequential transfer learning is also the de facto standard for tasks with limited labeled data, usually involving two-step approach: pretraining on large data corpora and fine-tuning on smaller domain-specific dataset. Nevertheless, more than two steps (datasets) have also proven beneficial for knowledge transfer (Poth et al.
2021 Jeong et al.) 2020
At the core of clinical data are Electronic Health Records (EHR), which document the medical history of patients. Upon patient admission, predictions of diagnoses, procedures, medications, and mortality risk help optimize resource allocation in clinical facilities. Clinical Decision Support Systems (CDSS) increasingly rely on NLP methods to assist medical professionals. Models from various clinics and specialties have been made available by researchers in large hosting platforms like Huggingface (Wolf et al. 2020). For example. clinical BERT models trained by Zhongshan and Qingpu Hospital /Wang el al. 2023 or Charite Hospita Bressem et al. 2024) among others are publicly available and can be finetuned by other clinical facilities for downstream tasks.
In diagnoses prediction state-of-the-art. (SOTA) methods augment. Transformer representations (BioMed- BERT (Tinn et al. 2023 ) with prototypical networks (Figueroa et al. 2024 van Aken et al. 2022. Diag- noses prediction generally poses significant challenges: 1) In practice diagnoses occur with widely varying prevalence, exhibiting a pronounced long-tail distribution (Papaioannou et al. 2022): 2) Medical knowledge and data are unevenly distributed across regions and languages. For example, the higher incidence of gas- tric cancor in Japan (Naylor et al. 2006) or Chagas discase in South America (Martins- Moli et al. 2014): 3) Privacy concerns isolate patient data across medical facilities, hindering unified model training. These challenges are amplified by the fact that Transformer models are usually pretrained on large corpora in high-resource languages (mainly English) limiting performance in low-rcsourco languages.
Sequential transfer learning addresses the constraints of clinical data, namely privacy and data isolation, uneven diagnosis distribution, low-resource settings, and multilinguality. Multiple fine-tuning steps have also been shown to improve downstream performance on low-rcsource diagnoscs prediction (Papaioannou et al. 2022). Research has centered on finding optimal sequences of tasks (Lim et al. 2024 Poth et al. 2021). highlighting the performance sensitivity of Transformers to the task order. However, finding a training order for optimal knowledge transfer requires access to all datasets, which does not comply with the clinical data restrictions mentioned above. We argue that creating robust solutions subject to these constraints involves
1) A realistic simulation of knowledge transfer that reflects the challenges of working with clinical data, 2)
An architecture that is agnostic to the fine-tuning sequence.
DDownload the repository https://anonymous www.erouuuuuooon.ooo....e http:aacccaccc/////P00-----0-0
BERT from Zhongshan and Qingpu Hospitals.
BERT from Charite Hospital.
2

---

Published in Transactions on Machine Learning Research (12/2025)
We gather collection of six clinical datasets encompassing different clinics, writing styles, languages, number of patients, and diagnoses distributions. In Figure 1 we illustrate the label spaces of our five training datasets, which we complement with an additional dataset used only for testing, We train and evaluate SOTA architectures strictly following clinical data constraints, i.c. using model chockpoints rather than data as the medium of knowledge transfer (see Figure | agnostic performance, reflecting the realistic scenario which clinies are unaware of the data seen by publicly available models.
right). Additionally, we focus on assessing sequence-
(102.2k)
SemClinßr (0,5k)
Swedish
(1.6k)
Ahopa (1.6k)
Codi
(0.7k)
163
$
B
6
M
C
10
2
B
98
53
50
82
2
Figure 1: (Left) Label space relationships across five clinical datasets used for training. Legend: number of training samples for each dataset. (Right) Six (two-steps more) training sequences two 5-step permutations of the datasets that start with MIMIC (M). Each node is the end of sequence and results in a model checkpoint; e,g, C' indicates the model checkpoint for M + C. In our experiments, we ensure that the model checkpoints (and not data) are the only medium knowledge transfer.
generated from
We find that SOTA models for diagnoses prediction such as S-Proto (Figueroa et al. 2024) struggle under realistic clinical conditions. We argue that building non-interfering knowledge redundancy during sequential fine-tuning is key for knowledge transfer in this scenario. However, deep neural networks are usually dense, i.e, they utilize all parameters during prediction. In sequential settings, deep neural networks often suffer from knowledge overwriting, which impairs elfective transfer (Ling et. al. 2024).
We introduce SPONGE: an efficient architecture that boosts knowledge transfer in diagnoses prediction by creating end-to-end sparse language representations. We construct these by integrating concepts from parameter-efficien fine-tuning (PEFT) (Pfeilfer et al. 2020 Poth et al., 2023), prototypical networks (Snell let al.] 2017), and conditional computation (Bengio al. 2013) using non-parametric wimner-takes-al (Yuille & Grzywacz, 1989) mechanism. Inspired by biological ncural circuits, the sparse subnetworks in
SPONGE are fullede predictors that compete to predict diagnosis. While requiring only 222.6 of the trainable parameters, our method outperforms current SOTA in diagnoses prediction. We validate this extensively by analyzing both single and multi-step transfer learning scenarios.
Catastrophic forgetting (McCloskey Cohen 1989) during sequential fine-tuning impairs generalization. We find that prototypical classifiers tend to focus disproportionately on recent labels when compared traditional Transformers, reducing generalization to unseen datasets. However, explicitly preventing catas- trophic forgetting can degrade performance on target tasks (Pfeiffer et al. 2020). We propose alternative approach: Hydra, strategy that improves generalization to unseen datasets without modifying SPONGE's architecture. Hydra efficiently improves macro AUROC and PRAUC by 7.5 and 3.5 percentage points, re- spectively, with a negligible increase in inference parameters. SPONGE and Hydra function as adapters that can be loaded and fine-tundd by clinics as needed to boost knowledge transfer.
Additionally, our method inherits the explainability propertics of previous SOTA (Figueroa et al. 2024) by predicting in a latent prototypical space (Chen al., 2019). With input from medical professionals, we validate that the observed improvement in knowledge transfer is underpinned by clinically meaningful explanations van Aken et al. 2022
Contributions
3

---

Published in Transactions on Machine Learning Research (12/2025)
To our knowledge, our evaluation of multi-step sequential transfer learning is the largest to date in simulating knowledge transfer for diagnoses prediction with real-world clinical data constraints.
We propose a novel architecture of sparse competing subnetworks that achieves superior knowledge transfer in diagnoses prediction.
Our analysis identifies weaknessos in current SOTA approaches, which we address with an efficient strategy that mitigates catastrophic forgetting and improves generalization to unscen datasets.
2 Related Work
Diagnoses prediction from textual data. Transformer models have been widely applied to diagnoses
prediction (He et al., 2025 Roehr et al. 2024 van Aken et al. 2021 Rasmy et al. 2021 Yang et al. (2022)
employ prompt tuning combined with a contrastive loss to improve few-shot prediction of diagnoses codes,
while Liu et al. (221 propose a convolutional attention network that captures multi-scale representations
and integrates cross-entropy with focal loss to better handle rare labels. Zhang et al. (2022) incorporate structural information from medical documents and reconcile code emboddings to address class-imbalance and stylistic heterogeneity in clinical (exts. The work most closely related to ours, S-Proto (Figueroa et al., 2024) improves diagnoses prediction, particularly for rare codes, by augmenting Transformer with sparse prototypical network. The authors attribute the improved performance to the competing subnetworks in their prototypical layer. In contrast, we construct competing end-to-end subnetworks that integrate both the prototypical network and the Transformer to boost diagnoses prediction for high and low-resource datasets.
Cross-lingual transfer a popular topic in NLP (Hämmerl ct al. 2024 Philippy et al. 2023): Most research has focused on identifying optimal dataset sequences and configurations, extracting dataset prop- erties that enhance performance on downstream tasks (Lim et al. 2024 Protasov et. al. 2024 Lin et al., 2019 Malkin et al. 2022). Given clinical data constraints, Paapaioannou et al. (2022) find that sequential training for diagnoses prediction boosts downstream performance for low-resource datasets, however, only for optimal dataset sequences. In contrast, we propose a robust architecture that enables cross-lingual transfer independently of sequence order, and the dataset properties; i.c., size, labels, distribution, and language.
Parameter efficient fine-tuning (PEFT) and generalization. Liu et al. 2024); Lialin et al. (2023)
Chalkidis et al. (2021) highlight the importance of PEFT using adapters to improve zero-shot classification performance. In work, adapters play a crucial role in inducing sparsity. We use pool of adapters to generate multiple Transformer representations, which compete via a winner-takes-all (Yuille & Grzywacz, 1989) mechanism. Additionally, PEFT enables our model to scale the number of subnetworks efficiently.
Parameter selective training approachos have centered on using masks to update specific paramoters (Somayajula et al., 2024; Sung et al., 2021 Winter el al. 2022). Similarly, we selectively update only a subset of model parameters. However, rather than applying weight. level updates, we constrain updates subnetworks responsible for end-to-end representations. Moreover, Ansell et al.) (2022) use sparse fine- tuning to boost cross-lingual transfer using prior knowledge of the target task and language. In contrast, our subnetwork selection strategy operates without prior knowledge of downstream tasks.
Subnetwork routing and lateral inhibition. With ever-growing parameters in Transformers, a Mixture of Experts (MoE) offers an efficient path for scaling computation (Shazeer et al., 2017 Lepikhin et al. 2021
Fedus et al. 2022 Du et al., 2022 Chi et al. 2022). Our approach differs from MoE. in that 1) it does not fragment the input (tokens) 2) we use no additional routing loss functions or gating mcchanisms 3) our wubnet are independent predictors that compete to produce the best representation. In fact, we leverage conditional computation (Bengio et al. 2013). and are closer to single-layer biologically inspired neural networks (Figueroa et al. 2025 Bricken et al. 2023 Liang et al.) 2021 | to learn sparse representations through lateral inhibition. However, we extend this to deeper architectures integrating Transformer models with prototypical classification in a latent. metric space.
Sequential lcarning and catastrophic forgetting. Sequential learning presents significant challenges for deep neural networks. A well-established trade-off in this setting is the stability- stability-llasticity dilemma, which is central to continual learning (Biesialska et al. 2020 Huang al. 2021). h. In cross-lingual transfer, explicitly
4

---

Published in Transactions on Machine Learning Research (12/2025)
mitigating catastrophic forgetting (i.e., optimizing for stability) may result in reduced downstream perfor- mance (Pleiffer et al. 2020). We use winner-takes-all as non-parametric inhibition approach that creates non non-interfering parallel representations, improving both plasticity and stability. However, our objective is to maximize knowledge transfer under clinical data constraints; hence, we evaluate downstream performance on low-resource datasets of clinies.
3 Multilingual Clinical Datasets
We train and test our models on sequences constructed from five different clinical datascts across multiple languages (see Figure right). Additionally, we evaluate their generalization performance on multilingual dataset with the same EHRs in seven different languages in zero-shot setting. For all datasets, map the diagnoses to the Clinical Classifications Software Refined (CCSR P code space. CCSR offers advantages over International Classification of Discases (ICD-10), including improved clinical relevance and simplified calegorization.
MIMIC-IV (M) (Johnson et al. 2021 2023 is an English clinical dataset containing cases from the Intensive Care Unit (ICU) and other hospital departments. Admission and discharge notes are simulated from the EHRs as in van Aken et al. 2021 We use only the ICD-10 subset, which we map to the corresponding
CCSR categories.
CodiEsp (C) contains clinical case studies in Spanish, with patients from various medical specialties, including oncology, urology, cardiology, pneumology, and infectious diseases Miranda-Fiscalada et al. 2020
AHEPA-Cardio (A) are cardiology discharge summaries in Greek (Papaioannou et al.
2022
SemClinBr (B) comprises Portuguese clinical notes from multiple Brazilian medical institutions across various specialties (Oliveira et al. 2022
Stockholm University Gastrointestinal (S) Lamprrodds et al. 2023) consists of EHRs from gas- troenterology department. These are in Swedish and sourced from the Swedish Health Record Research
Bank Dalianis et al. 2015
Zero-Shot Datasets: DisTEMIST. We use the dataset of Miranda-Escaladaa el al. (2022) for zero-shot evaluation. It comprises the same EHRs expressed in Spanish, English, Catalan, Portuguese, French, Italian, and Romanion.
Data Splits and Label Space. For all datasets, we use stratified sampling (Sechidis et al. 2011 to create train, validation, and test splits (see Table 5 in Appendix A). We remove EHRs with less than four labels. Figure summarizes the label space and training sample counts.
4 Sequential Transfer Learning for Diagnoses prediction
Diagnoses prediction is a multilabel classification task with large label space, in our case, comprising 509 CCSR codes (98% of the full CCSR specification). We focus on knowledge transfer for this task using only model checkpoints for transfer to simulate real-world clinical constraints. Accordingly, we adopt the canonical sequential transfer learning setup involving both single and multi-step fine-tuning (Ruder) 2019). This involves sequential training with all permutations of our datasets, evaluating performance independently of any specific fine-tuning order.
We initialize all architectures on this task with MIMIC, our high-resource dalaset. Next, we construct all permutations of the four low-resource datasets to generate sequence variations, This process results in 64+1 distinct sequences (including the one-stop sequence on MIMIC, see Appendix E Figure 1 (right) illustrates
two example sequences.
CCSR categories
This research has been approved by Swedish Ethical Review Authority under permission no.2019-05679. The use of Stockholm EPR. Gastro ICD-10 Pseudo Corpus with the amendment 2022-02386-02.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Evaluation. The goal of our experiments is to assess how effectively architecture transfers relevant knowledge and generalizes to unseen data at each step of the sequence. For example, sequences involving one training dataset are evaluated on four unseen test datasets, those trained on two evaluated on three, and so on. DisTEMIST is used exclusively as a test set. We evaluate all resulting model checkpoints (64 + 1), measuring averages of macro AUROC, micro AUROC, and macro PRAUC to capture sequence agnostic scores.
MIMIC (High-resource). We report results this dataset it serves the standard benchmark for diagnosis prediction (Roehr et al., 2024 van Aken et al. 2021). This evaluation corresponds to the one-step sequential transfer learning scenario, i.c., the sole single-datasct sequence in our experiments.
Final datasets (Low-resource). We rank architectures based on the performance on the final dalasel in each sequence. This evaluation measures downstream performance independently of the datasets seen by the model during sequential training. This is consistent with the performance expected by clinics when fine-tuning models without explicit knowledge of their prior training dala. In this evaluation, we average all performances in all sequence configurations. We evaluate 64 sequences of two datasets or more, with total of 16 sequences for every target dataset (see Appendix
E
Zero-shot generalization. In contrast to the standard zero-shot cross-lingual transfer setting (uu et al. 2020j, we focus on models trained sequentially on multiple datasets. This implies that variations in data distributions, label spaces, and language typologies may contribute or interfere with every additional training dataset. We probe the generalization capabilities of all models on DisTEMIST in addition to all unseen datasets. We evaluate 64 + 1 checkpoints per model and report the average for each metric for 1) all unseen datasets, 2) unscen languages within DiSTEMIST, and 3) seen languages within DiSTEMIST.
5 Methods
Several works have lackled diagnoses prediction with Transformer networks and architectural augmenta- tions. van Aken al.) (2022) incorporate prototypical classifiers to use latent metric spaces, enhancing both the explainability and performance of model predictions. Figueroa et al. (2024) extend this approach in S-Proto by adding sparsity to the prototypical classification layer using winner-takes-all mechanism. Specifically, S-Proto leverages competing subnetworks to model distinct phenotypes of diagnoses, further boosting performance. These architectural enhancements to Transformer representations achieve state-of- the-art performance in diagnoses prediction. However, they struggle when exposed to multi-step sequential transfer learning, since sequentially updating a single Transformer representation creates inter-lask interfer- ence. We speculate that this is due to the competing subnetworks, which share large number parameters, leading entangled representatious. To address this, we propose extending sparsity throughout the entire network, rather than limiting it to the classification layer.
5.1 End- To-End Sparse Representation
The dense (non-sparse) modules of S-Proto FFigurroa et al. 2024 are the Transformer and the projection matrix to the prototypical space. We add competing subnetworks to these components to create end-to-end sparse representations, which is our main methodological contribution generalizing the work of Figueroa et al. (2024). This entails 1) creating in parallel multiple Transformer representations for the same input, 2) expanding the dimensionality of the projection layer to the prototypical space, and 3) selecting winner subnetwork, hence, the winner-takes taeee-t----all all mechanism must be generalized for the entire architecture. We show an overview of the resulting architecture in Figure 2
Creating multiple Transformer representations would require parallel embedding Transformers, which is computationally prohibitive. We opt for simpler solution using multiple adapters (Figure 2 left), which significantly improves efficiency. Trainable adapters combined with a language model have shown competitive performance on classification tasks (Houlsby et al. 2019). We add I' task adapters (Pfeiffer et al. 2020 modular components, freezing the rest of the Translormer while training.
T A (embb, r) U (DD +r
(1)
6

---

Published in Transactions on Machine Learning Research (12/2025)
Tack Adastarn
Iecheee spran
508 Diagnoses
Fra Frazen Parameters
Sperse Sperse-Autivetee Autivatedd Learnable Parameters
Cinical Note
Transformer
Down Projection
End-tp- End Network
Label-wise attention & Prototypes
Myocardial
Figure 2: Classifying clinical note with SPONGE: A pool of T adapters (left) create multiple transformer representations. P down-projection layers map these representations, which are then classified in latent metric space with sparse label-wise attention. & is the winner network among all competing subnetworks that map the input to the label space, resulting in sparse gradient updates.
D' and U are the task adapter down and up projections, emb is the encoder representation and r is the residual for a layer within the Transformer. We compute multiple token representations, one with every task adapter: rf Exx, where h is the hidden dimension of the encoder, and E is the encoder sequence length,
Expanding the dimensionality of the projection layer. Additionally, increase the number of down- projection layers L E Raxd to L E Pxxxd (Figure 2 center), where d is the hidden dimension and P is the number of projection layers. We project '/' which results in representation F = (nn,, , LP) E RExd, where I" Raxd is a single projection indexed by p, and E (1...., T) and p E {1,..., P} define a subnetwork. We refer to this as the input subnetwork, specified by t and p, which enables an end-to-end sparse encoding of the EHR and is core to our contributions.
Let 7 I' be an output subnetwork index in the prototypical layer, c E C be a specific class, therefore E T P I defines the subnetwork that spans through the entire model and involves the combination of 1,P,7 7- We first map the token vectors v
(WbT,WWT) RCXE
where W e R are the abblliise-attention vectors. A class score S = softmax(d5) is computed for every token in the EHR. The mapped EHR, for subnetwork in prototypical space is vf.
x = (SE PP e gCxd
The Euclidean distance from mapped EHR of to proto- type uf is denoted by & = ||u - 11l22 and the prediction of subnetwork ÿf a(-é) where or is the sigmoid function. We select the winner subnetwork & (Figure 2 golden path) with the winner-takes-all mechanism inhibiting all other subnetworks:
SPONGE: Generalizing winner-takess-dl
= 8
where d is the Kronecker 8 function. To compute the winning subnetwork we start by selecting the winner output subnetwork::
= argmax(($)
(3)
We use 7 f. to select the winner input subnetwork i, with the mode over the classes for an EHR.
1, p mode (argmaa (555
c 7,7 x..)) ...r)]
(4)

---

Published in Transactions on Machine Learning Research (12/2025)
Finally, the and-to-end winner subnetwork is:
{ = argmax (g (17)
We define the loss the binary cross-entropy between & (see: Equation 2 and the true labels y.
Subnetwork exploration. Training of sparse networks with wimmer-lakes-al may result in dead subnet- works that are never selected and, thus, never updated. We modify the approach in (Bricken et al. 2023 inspired by c- greedy (Sutton [ exploration, to ensure broader subnetwork activation during training. At the start of training, we explore the top subnetworks &, choosing one of these randomly. During this phase, we anneal & linearly until k 1. Training proceeds with the top-1 subnetwork in Equation This strategy helps initialize and train all subnetworks, avoiding dead units, while still limiting computation by updating only one subnet work per sample.
5.2 Boosting generalization
Prolotypical networks use latent distance to make predictions. Guided by the loss, these classifiers map tokens and shift prototypes in latent space. During sequential training, the learned prototypes become overly biased toward the most recent label distribution, severely restricting generalization different label spaces. We argue that distributed knowledge still exists in the input subnetworks of SPONGE due to the overall sparsity (we examine this further in Section
Hydra: exploiting distributed knowledge. To counter the label recency bias, we store the prototypice pical parameters (u and W) learned for each dataset in training sequence, using them only for inference. Impor- tantly, we only use parameters from past training steps for knowledge transfer, respecting the constraints of clinical data. Despite the growing number of output subnetworks, the winner subnetwork selection strategy of SPONGE remains applicable without requiring architectural changes. Although the number of paramoters for inference increases after each dataset in a sequence, the increase is small w.r.t. the Transformer's size. In our experiments, a fivo-dataset sequence, the added vectors amount 6.6M parameters or 2.4% of the Transformer's 270M. A key advantage of the wiinerr-laees-ee selection its compatibility with parameter accumulation, requiring architectural changes added complexity like weight consolidation (Kirkpatrick ct al. 2017 or knowledge distillation (Ermis et al. 2022
HSPONGE: Hydra facilitates effective knowledge transfer to previously unseen data distributions. Impor- tantly, SPONGE and Hydra scale very well to larger Transformers since the dimensionality of the parameters is independent of their hidden size This approach supports modular scaling to arbitrary dataset sequences during sequential training. Furthermore, in the case of growing label spaces, this modularity would enable our models to train only the new parameters.
6 Experiments
Although generative large language models (LLMs) have shown strong performance in many language tasks, they still fall short of SOTA in many classification scenarios (Yang et al.] 2024), particularly those with large label spaces. For diagnoses prediction, the best-performing approach remains fine-tuning Transformer
representations with a classification head, for both encoder (Roehr et al. 2024 Figueroa et al. 2024] and decoder models (Gema et al. 2024), meanwhile decoder methods st.ill fall short this task (Grundmann et al. 2025
We investigate how sparse subnetworks impact sequential transfer learning for diagnoses prediction. There- fore, we control for one Transformer architecture. We use XLM-R (Conncau et al. 2020) for all experiments,
as it has been extensively studied for cross-lingual transfer (Philippy et al. 2023 Pleiffer et al., 2020 Hu
le al. 2020 Choi et. al. 2020: Conneau et al. 2020). Although our methods are applicable to any Transformer architecture, we favor XLM-R over significantly larger Transformers, given the large number of experiments.
Baselines. We focus on the following architectures: 1) XLM-R, which cross-lingual model and the least sparse method (single dense network); 2) XLM-R + A, which extends the base model with single task
8

---

Published in Transactions on Machine Learning Research (12/2025)
adapter to ablate the effect of PEFT and an adapter in our architecture; and 3) S-PROTOxuxaa Figueroa et al. 2024), the current SOTA architecture for diagnoses prediction, which combines a dense Transformer with a sparse prototypical classifier. For the latter, we replace the BiomedBERT (Tinn et al. 2023 with XLM-R, since the former not a cross-lingual Transformer.
SPONGE Variations. We evaluate multiple configurations of sparse subnetworks by varying the number of adapters and projection matrices. We name each variant SPONGE" where T and P stand for the number of task adapters and projections rospectively. SPONGE establishes the effect of having no sparsity in the encoder. SPONGE displays the effect of having sparsity only in the projection stage before the prototypical layer. SPONGE"1 highlights the impact of no sparsity in the projection, but only in the Transformer using a pool of adapters. Given the effectiveness of adapters for cross-lingual transfer (He et al. 2021), our sparsest model, SPONGE uses more adapters than projections.
All prototypical models (SPONGE and S-PROTOXAAA) have five output subnetworks as in Figucroa et al. h024h. We detail hyperparameters in Appendix C
7 Results
Performance on MIMIC. This dataset is a standard benchmark for NLP models in diagnoses prediction. We present the performance of all models in Table ] (left). S-PROTO outperforms all dense base- lines (lack subnetworks): XLM-R and XLM-R + A. Our variant with the largest number subnetworks: SPONGE033 outperforms all evaluated methods.
Subnetwork exploration. We show the impact of this strategy in the bottom section of Table D (left). The models SPONGEL and SPONGE (marked with -x) indicate that they do not use exploration. In both cases, exploration improves PRAUC, benefiting the sparser SPONGE morc.
Table 1: (Left) model performance when trained solely on MIMIC. (Right) average performance on the last dataset each sequence. SPONGE"3 consistently outperforms across both settings.
MIMIC Only
Sequential Training micro AUROC
Model
macro AUROC
micro AUROC
macro PRAUC
macro AUROC
macro PRAUC
S- PROTO
87.51
93.89
28.08
80.08
86.62
37.01
SPONGELI
89.68
94.67
33.18
87.47
91.78
52.07
SPONGELS
86.63
93.75
22.94
83,57
89.71
43.71
SPONGE
86.55
90.42
31.42
86.19
90.71
48.93
SPONGE
89.70
94.70
34.17
88.58
92.09
54.24
XLM-R
86.21
93.67
27.27
83.10
88.89
45.92
XLM-R +A
85.74
93.31
26.38
83.81
89.10
45.11
SP
89.28
94.66
32.51
S
89.48
94.56
32.56
Sequential training performance. We show in Table I (right) the average performance of all models across the 64 training sequences (16 per target dataset; see Appendix E XLM-R+A performs similarly to XLM-R, reinforcing the effectiveness of PEFT. S-PROTO performs the worst. This emphasizes the inter-task interference created by updating single Transformer representation for sequential training. PEFT with output subnetworks shows significant gains. Specifically, 7.39 points in macro AUROC and 15.06 points in macro PRAUC when comparing SPONGE', to S-PROTO
Creating input subnetworks exclusively with either the adapter component or the projection is not beneficial as creating them with both simultaneously. This is evident when comparing SPONGE and SPONGE against SPONGE.3 In general, increasing sparsity via more subnetworks boosts PRAUC, aligning with findings from Figueroa et al. 2024).
S consistently demonstrates the best performance regardless of dataset order, language mix, label space diversity, or fine-tuning sequence length, while requiring s 2.6% of the training parameters of
9

---

Published in Transactions on Machine Learning Research (12/2025)
XLM-R or S-PROTOOUMER (sce Table 8 in Appendix Fj. We therefore focus SPONGE for all subsequent
analyses.
Downstream dataset performance. We further analyze sequential training results by examining all permutations ending with each specific target dataset. We present this in Table 2 (top) and highlight how SPONGE outperforms all methods for all metrics, showcasing the robustness to large distribution shifts (languages, dataset sizes, and labels). Generally, downstream dataset performance significantly benefits from multi-step sequential transfer learning, detail this in Appendix [
Although in practice clinical documentation is often decentralized and multilingual, we find that SPONGE also improves knowledge transfer in a standard monolingual transfer learning setting. We expand on this in Appendix G where we reproduce the experiments of Section for English translations of the datasets and SOTA English clinical Transformers.
Number of training datascts. We analyze the performance w.r.t. the number of datasets used in a training sequence. We present this in Table 2 (boltom). SPONGE also significantly outperforms all other methods regardless of the number of datasets. We observe that updating all Transformer parameters (XLM-R, and S-PROTOQMMAR) leads to performance degradation as the sequences increase in length.
Table 2: Classification performance per target dataset (top) and number of datasets (bottom) averaged over all permutations of sequences. SPONGE outperforms all methods in both set settings.
macro AUROC
miero AUROC
macro PRAUC
Model/Datase
A
A
S
PROTOO
81.83
70.75
83.33
84.42
88.18
80.81
88.51
88.96
42.22
23.83
34.66
47.33
SPONGE
90.05
82.65
91.03
90.62
94.18
86.68
93.97
93.53
59.93
42.16
55.22
59.67
XLM-R
85.78
73.30
85.67
87.63
91.39
81.78
89.70
92.69
58.85
27.73
39.20
57.90
XLM-R+A
84.83
76.93
86.38
88.08
90.27
83.05
90.33
92.76
55.65
28.54
38.77
57.52
Model/ # datassets
2
3
2
5
S- S-PROTOuEen
81.83
70.75
83.33
84,42
88.18
80.81
88.51
88.96
42.22
23.83
34.66
47.33
SPONGE
90.05
82.65
91.03
90.62
94.18
86.68
93.97
93.53
59.93
42.16
55.22
59.67
XLM-R
85.78
85.67
87.63
91.39
81.78
89.70
92.69
58.85
27.73
39.20
57.90
XLM-R+A
84.83
75.93
86.38
88.08
90.27
83.05
90.33
92.76
55.65
28.34
38.77
57.52
Performance on unseen data. In Table 3 (left) we present the results for all unseen datasets: DisTEMIST and all datasets absent in a training sequence. Additionally, we adapt S-PROTOOuMAR and SPONGE with the Hydra strategy. We denote this with H preceding the name of the model. S-PROTOOUAs- performs worse than XLM-R and XLM-R++ and fails to improve when employing Hydra. SPONGE performs better than XLM-R and XLM-R + A in PRAUC; however, falls behind on AUROC. Hydra proves effective, since HSPONGE'3 outperforms all other methods.
Performance DiSTEMIST. The clinical notes of this datasel are multiple versions of the same EHR in different languages, therefore it is good test bed for generalization. We distinguish between unseen and seen languages for all trained sequences. We present results in Table 3 center and right. Intuitively, for all models the performance on seen languages is higher than on unseen languages. HSPONGESS consistently outperforms in both cases.
8 Analysis and Discussion
Our experiments show that multi-step sequential transfer learning is generally beneficial for diagnoses pre- diction. SPONGE outperforms all other methods on all target datascts, regardless of the sequence. Never- theless, SPONGE struggles lo generalize to unseen datasets, often underperforning XLM-R and XLM-R + A (Table 3 However, we demonstrate that Hydra an effective strategy to address this. We attribute the increased performance of RSPONGE.5 in SPONGE induced by winner-takes-all which creates distributed Transformer representations; and 2) Hydra's ability to leverage these representations, alleviating label recency bias
two main factors: 1) the sparsity and the modular architecture
10

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: Average zero-shot performance of all sequences. Left: all datasets (DisTEMIST + those unseen
during training). Center, right:
DisTEMIST only, for both seen and unscen languages respectively.
Unseen datasets DiSTEMIST
DiSTEMIST unseen languages
DisTEMIST seen languages
micro AUROC
macro PRAUC
micro
macro PRAUC
macro AUROC
micro
macro
Model
AUROC
AUROC
AUROC
AUROC PRAUC
S- S-PROTOREEn
68.77
65.40
12.58
69.36
65.11
10.91
72.69
67.23
14.97
HS-PROTO
54.39
54.76
8.32
54.69
54.32
6.86
51.41
51.93
8.08
SPONGE
74.46
71.15
20.91
74.44
70.69
19.27
77.74
73.23
23.47
RSPONGE
81.99
77.86
24.40
82.22
77.26
22.21
85.80
81.11
27.28
XLM-R 75.15
73.38
19.03
75.40
72.60
16.42
81.52
78.23
25.17
XLM-R+A 76.18
72.65
18.67
77.14
72.82
17.51
80.50
75.69
21.97
Distributed Representations in SPONGE. We investigate the existence of distributed knowledge after sequential training. For this, we test on the first dataset (MIMI) after training on each the 64 dataset sequences. As before, we report the average performance of all checkpoints which we denote as Original Table g Additionally, we evaluate variant of cach chockpoint where we replace the classification layer with the one trained only on MIMIC; denote this M. This allows the model's feature-building network to better align existing knowledge with the MIMIC label space. In other words, we decouple the Transformer reprosentation from the classification layer and measure its impact. M favors the label space of MTMIC and thus mitigates label recency bias.
We observe that S-PROTOXIMER does not improve when employing M. We argue that this degradation in performance related to the dense projection L which is crucial map to the prototypical space. In contrast, XLM-R and XLM-R A benefit from M, hinting that the bare Transformer representations can recover knowledge from MIMIC, further emphasizing the need for sparsity in L for S-PROTOXUM.R Although SPONGE underperforms XLM-R and XLM-R + -A, it outperforms all methods when employing M. This significant boost in performance confirms that SPONGE retains MIMIC knowledge but is unable to access it once the prototypical parameters и and W have shifted after sequential-training The M strategy is a simplification of Hydra that explains how the latter enables access to distributed knowledge in the model for better generalization as seen in Table 3
Table 4: Average performance on MIMIC of all 64 checkpoints. Every checkpoint has been trained on two more datasets. Original denotes scores of the unmodified checkpoints; M corresponds to checkpoints modified by replacing their classification head with the one trained only MIMIC; A denotes the difference.
Macro AUROC
Micro AUROC
Macro PRAUC
Model
Original
M
A
Original
M
A
Original
M
A
S-PROTORMER
63.97
46.91
-17.06
68.08
63.34
-4.74
6.85
6.08
-0.77
SPONGESS
68.43
82.30
+13.87
79.62
83.50
+3.88
14.58
20.39
+5,81
XLM-R
76.87
78.91
+2.04
80.74
87.48
+6.74
16.68
18.53
+1.85
XLM-R+A 73.00
78.79
+5.79
76.68
87.04
+10.36
11.27
17.40
+6.13
tannerltakes-all and optimal subnetwork. We expand on the analysis of distributed knowledge by focusing on all subnetworks (full-fledged predictors) of SPONGE6.3 For every 4th subnetwork (90 total) of this model, we compute all metrics which we present in Figure 3 The Wimner-takes-all mechanism defines the order of the subnetworks indexed by k (x-axis in the figure). For instance, k 89 is the most inhibited subnetwork. In contrast, A 0 is the single winner subnetwork that yields prediction. We highlight with the best-performing subnetwork, which proves to be within the top-18 for all metrics. Generally, given the aggregation in Equation 4 predictions are bundled in groups of 18 where performance is very similar, stemming from the 6 adapters 7' and 3 projections L. This supports the idea that the inhibition mechanism in winner-takes-dl produces multiple non-interfering predictors that are almost optimal.
Hydra and label recency. SPONGE underperforms when not modified with Hydra (see SPONGE XLM-R and XLM-R A in Table 3 yet. HSPONGEE. outperforms all methods unseen data. We
11

---

Published in Transactions on Machine Learning Research (12/2025)
80-
70-
Score
macro AUROC AUROC marc PRAUC
50-
40-
30-
20 40 60
80
k-winner subnetwork
Figure 3: Scores achieved by SPONGE for all subnetworks, ordered by the wimmer-takes-all selection mechanism (see Equation (5)) The decreasing nature of the curves highlights how the top activated sub- networks highly correlate with classification performance, reinforcing the suitability of this selection strategy.
Asterisks point to the k with the highest scores which are close to the winner-takes-all selection at k 0.
investigate the impact of label recency prototypical models when exposed to unseen data. Therefore, compare HSPONGE and S- S-PROTOxuaeR since it is the SOTA architecture for this task. For fair comparison to S-PROTOXIMan we focus on its best-performing checkpoint on DiSTEMIST i.c., the sequence M-BB+C (see performance of S-PROTOxM.- in Table 7 in Appendix D and Table 3) We analyze the performance of both models w.r.t. the recency of the labels for this sequence. For each label, we define recency as the proportion of the most recently trained samples (C) w.r.t all samples M, B, and C.
Recency and PRAUC. We divide the labels into 8 logarithmically spaced recency groups, each containing at least 10 labels to ensure reliable estimation. Figure ! (bottom left) shows the label distribution across these recency groups. In Figure 4 (top left) we present the difference(A) between the two models in macro PRAUC for the labels sorted by recency (see z-axis). PSPPONGE performs significantly better on all but one label group, thus, A PRAUC is positive. This difference decreases as recency increases; notably, is significantly larger for labels with lower recency. This illustrates the positive effect of Hydra at mitigating the recency bias.
Recency and explainability. Prototypical methods create token saliencies that explain their predictions. Faithfulness quantifies how gradually masking the input according to these salicncies, deteriorates model performance. A lower faithfulness score indicates greater explainability (Atanasova el al.] 2020). We generate token saliencies for HSPONGE and S-PROTOXIMER, and compute faithfulness for macro AUROC and macro PRAUC for the aforementioned recency groups. Figure (right) presents the difference in faithfulness for HSPONGE and S-PROTOMM..-- We expand these scores for all DisTEMIST languages (rows in the Figure). Similar to A PRAUC, the largest difference is in the first (least recent) label group. This relationship appears to hold independently of the language.
Both classification (PRAUC) and explainability (Faithfulness) metrics confirm that the prototypical layer alone (S-PROTOQIMaa) struggles to generalize in sequential training, likely due to an overemphasis on the most recent labels. However, HSPONGE overcomes this in both classification and iaplainabilityy reinforcing the importance of combining the distributed representations of SPONGE with Hydra.
Analysis of saliencies with doctors. We further examine 30 patients and five diagnoses with the help of medical doctors. We compare the saliencies of E-PROTOXIMAL and HSPONGE5 for the previously analyzed sequence M-B->C We aim to understand how the performance gap between the two models manifests in qualitative differences from domain expert's perspective. We focus testing AHEPA (A), which is not included in the sequence and contains the typologically most distant language (Greek). observe the same performance gap in laithfulness on the least. recent labels as with the DisTEMIST languages. The physicians were tasked with ranking patients according to their relevance to describe a diagnosis. Remarkably, both models rank the patients similarly, i.c., they successfully identify prototypical patients. itative differences of the saliencies, doctors find that S-PROTO more sensitive to learning annotation artifacts (ellers et. al. 2019) like stopwords explicit mentions of acronyms of the diagnosis, explaining
terms of the qual-
12

---

Published in Transactions on Machine Learning Research (12/2025)
& Faithfulness PRAUC
& Faithfulness AUROC
APRAUC
96
EN
34
FR
8.0
#8 Labels
Protouu--n
PT
RD
1t-
ES
-05
Sample proportion [C / [M.B.C|
1w:0
26d0
2e.00
Ie<11
Tedd
2e-33
2e-12
1M01
Sample proportion |C 1/ [M.B.C]
Figure 4: Left top: A(HSPOOGE -EETICEES-S-PEEEEE S-PROTO in PRAUC over label recency. Bottom: number of labels by recency group. On average (gray dotted line) HSPONGE 6,3 generalizes better for all label groups. Right A(HSPONGE - S- S-PROTOXXXB) Difference in faichfulness for zero-shot. datasets (lower is better). Although faithfulness is similar for both models (A is roughly 0), S-PROTOxu>> is less explainable for the under-represented labels the last finc- inn-tuning (most negative left region).
its weakness in generalization to distinct languages. This issue is compounded by S- S-PROTOxLa-R assigning near-uniform relevance scores to most tokens. In contrast, HSPONGESS sharply highlights key terms re- lated to each diagnosis. These qualitative findings suggest that the zero-shot gencralization capabilitics of HSPONGE". is underpinned by clinically meaningful and explainable saliencies.
9
Conclusion
We address the challenges of sequential transfer learning when data is subject to privacy constraints by the example of the clinical domain. We focus on a multi-step sequential transfer learning setting, transferring knowledge via model checkpoints rather than raw data. We propose SPONGE, novel architecture that excels at knowledge transfer and achieves superior performance in diagnoses prediction. We generalize winner-takes-ale all, mechanism of neural inhibition inspired by nature, creating competing representations. Extensive analysis reveals that these representations combined with the ability of Hydra to mitigate label- recency bias, are key to strong generalization performance. We confirm with medical doctors that the explanations provided by SPONGE are supported by clinical relevance.
Future Work. We examine a set of datasets in nine different Indo- ando-European languages, in multi-step sequential transfer learning. While this evaluation is significant, is not exhaustive; further analysis will be conducted more clinical data becomes available. Although we focus Transformers are widely used for solving generative tasks. Given that our architecture may also enhance daseder-based Transformer models, future evaluation on generative tasks is warranted as LLMs become more computationally efficient.
multi-label classification task,
our evaluation, SPONGE performs strongly across high- and low- resource data, sequential training, and generalization tasks. However, the scores achieved still leave room for improvement, which is crucial for the clinical domain. We emphasize that this work does not aim to replace doctors; the applications of technologies to the clinical environment must meet stringent legal and ethical requirements.
Deployment considerations.
10 Acknowledgements
We thank the reviewers and the Action Editor for their constructive feedback, which improved this work. This work is funded by the Deutsche Forschunggemeinsccatt (DFG, German Research Foundation) Project-
ID 528483508 - FIP 12, as well as the European Union under the grant project 101079894 (COMFORT Improving Urologic Cancer Care with Artificial Intelligence Solutions), as well as by BMWe SOOFI, Grant Agrcement 13IPC040D. The views expressed are solely those of the authors and do not necessarily reflect those of the European Union or European Health and Digital Executive Agency (HaDEA); neither is responsible for them.
13

---

Published in Transactions on Machine Learning Research (12/2025)
References
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulic. Composable sparse fine-tuning for cross- lingual transfer. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 1778 1796. Association for Computational Linguistics, 2022. URL https: httpps://doit org////oooo////oooooo/ 10. 11653/v1/2022 aclglong..............oooooolll. 125
doi: 10. 201/3///1/0022ACL--22
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. In Bonnic Webber, Trevor Cohn, Yulan Hc, and Yang Liu (eds.); Proceedings of the 2020 Conference on Empirieal Melhods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 3256-3274. Association for Computational Linguistics, 2020. doi: 10.18653/VV C03655//V122222 /1220000EEE/222PMAAAAA 263. URL https://doi. org/ /0. 120.13/2//vv/ emnlp-main 263
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3433 2013.
Magdalena Bicsialska, Katarzyna Bicsialska, and Marta R. Costa-jussà. Contimual lifelong learning in natural language processing: A survey. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pp. 5523-6541, International Committoe on Computational Linguistics, 2020. doi: 10.18653/VI /220020..200.////////22 /TT////.....0ONGAAAAAAAIC COLING-MAIN.574 URL attps://doi.org// http://000.00///010018////11/110 http:///.........ooooeo....o ioolinggmain.... 10w.3/1161vv100
Keno K. Bressem, Jecs-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C. Adams, Leonhard Liu, Felix Busch, Lina Xu, Jan P. Loyen, Stefan Markus Niehues, Moritz Augustin, Lennart Grosser, Marcus R. Makowski, Hugo J. W. L.. Aerts, and Alexander Löser. medbert.de: A comprehensive german BERT model for the medical domain. Expert Syst. Appl., 237(Part C):121598, 2024. doi: 10.1016/J.
USWA.2023.121500 URL hattps://doi.. 1erg/10.1000 1016/j eswa. www.wwwwwwwwwwee.eoo..e...e 121598
Trenton Bricken, Xander Davies, Deepak Singh, Dmitry Krotov, and Gabriel Kreiman. Sparse distributed memory is a continual learner. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nef, 2023. URL huttps:/openrevviee net/ forum? coluumiic 1id=
Taas Chalkidis, Manos Forgadiotis, and Ton Androutsopoulos. Multicurlex A multi-lingual and multi- label legal document classification dataset for zero-shot cross-lingual transfer. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Puntta Cana, Domini- can Republic, 7-11 November, 2021, pp. 6974-6996. Association for Computational Linguistics, 2021. doi: 10. 10.18653/V1, /122211122222222222 WEENE--MANNNN.E 5599. URL https: ftttps://doi.org/1111 10. 10653/11/22221111 emnlp-main 559
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In Hanna M. Wallach, Hugo Larochelle, Alina Beygels- imer, Florence d' Alché-Buec Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 8928 8939, 2019. URL httttp::/ppooeeddiieeee 120122000111..0222222277--eeeeee Abstract..tttttala neurips.cc/pan pttpsc/c//pape//////aaa///s//// 22
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the reprosentation col- lapse of sparse mixture of experts. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, Novem- ber 28- December 9, 2022, 2022. URL http://papers. httppsccc//ppprrrcp htcc/ppprrfflle//pppeeeeeeeeeeep af4f371f 41171111199900000000111111100000 htwpraaaaaaiarrddd-b---rrraa---e wwwe---aaattttttaaeaeaeeaeee-eee Conferenoeeeeeeeeeeeee html
Hyunjin Choi, Judong Kim, Seongho Joe, Scungjai Min, and Youngunee Gwon. Analyzing zero-shot cross- lingual transfer in supervised NLP tasks. Tn 25th International Conference on Pattern Recognition, ICPR
14

---

Published in Transactions on Machine Learning Research (12/2025)
2020, Virtual Event / Milan, Italy, January 10-15, 2021, pp. 9608 9613. IEEE, 2020. doi: 10.1109/ 01188806:201.94111 URL httpt::td....orrrg/eeeeeooo 1109/ICPR48806. 2021 .4125700
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACL, pp. 8440 8451. Association for Computational Linguistics, 2020.
Hercules Dalianis, Aron Henriksson, Maria Kvist, Sumithra Velupillai, and Rebecka Weegar. Health bank workbench for data science applications in healthcare. In John Krogstie, Gustaf Juell-Skielse, and Vandana Kabilan (eds.), Proceedings of the CA1SE-2015 Industry Track co-located with 27th Conference on Advanced Information Systems Engineering (CAiSE 2015), volume Vol-1381, pp. 1-18, Stockholm, Sweden, June 11 2015. CEUR. URL http: //www/ dav. su. se/ se/healthbank
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wci Yu, Orhan Firat, Barret Zoph, Liam Fodus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 5547-55669 PMLR, 2022.
Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédric Archambeau. Memory effi- cient continual learning with transformers. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Bel- grave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL URL/ttsp:/ppapez nips wwwp.iccpppp.eeeeee wwtpccppppppleeeeeeeeeeeeeeeeeee files/paper/200/ 2t////22222////// 45226e4119ddb33444aa226666 www.:///cc--ssatat--a-aaaaa-a-e Centtrrct--onfeeeene html
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:3, 2022.
Alexei Figueroa, Jecs-lichalis Papaioannou, Conor Fallon, Alexandra Bekiaridou, Keno Bressem, Stavros Zanos, Felix Gers, Wolfgang Nejdl, and Alexander Löser. Boosting long-tail data classification with sparse prototypical networks. In ECML/PKDD (7), volume 14947 of Lecture Notes in Computer Science, pp. 434 449. Springer, 2024.
Alexei Figueroa, Justus Westerhoff, Golzar Atefi, Dennis Fast, Benjamin Winter, Felix Alexander Gers, Alexander Löser, and Wolfgang Nejdl. Comply: Learning sentences with complex weights inspired by fruit fly olfaction. In 2025 Neuro Inspired Computational Elements (NICE), pp. 1-10, 2025. doi: 10.1109/ NICE65350. 202.350..222.111 0w..22222....9909 11064982.
Aryo Gema, Pasquale Minervini, Luke Daines, Tom Hope, and Beatrice Alex. Parameter-efficient fine- tuning of LLaMA for the clinical domain. In Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, and Danielle Bitterman (eds.), Proceedings of the 6th Clinical Natural Language Processing Workshop, pp. 91 104, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653 00.18653/v1/2000
wrw.attooooooooooo/222/2// org/2024. clinicalnlp-1.9) URL http:///c...attoooo/oo/////aaaaa
Paul Grundmann, Dennis Fast, Jan Frick, Thomas Steffek, Felix A. Gers, Wolfgang Nejdl, and Alexander Löser. Clinibench: A clinical outcome prediction benchmark for generative and encoder-based language models. CoRR, abs/2509 2509.26666. 2025.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specifi language model pretraining for biomedical natural language processing. ACM Trans. Comput. Heal., 3(1):2:1-2:23, 2:23, 2022. doi: 10.1145/3458754. URL httpp:///oo.. htg/10.1145/34587/
Katharina Hämmerl, Jindrich Libovicky, and Alexander Fraser. Understanding cross-lingual alignment A survey. Tn ACL (Findings), pp. 11922-10945 Association for Computational Linguistics, 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. A survey of large language models for healtheare: from data, technology, and applications to accountability and ethics. Inf. Fusion, 118:102963, 2025. doi: D..1016/..NEPUU..200..00 inffus. 2025. 102963
URL
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia- Wei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter- adapter-based tuning for pretrained language model adaptation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Evenl, August 1-6, 2021, pp. 2208 2222. Association for Computational Linguistics, 2021. doi:
URL httpp:///do..oo org/ 10. 18653/v1/2022 wttp:///222..c11111101 -long. 172
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. -fffiietr-efficientt transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790 2799. PMLR, 2019. URL http://prroceedinss.oo mlr. wttpiiinnmmmm..ppe/s////./ press/v97/ houlsby19a. html
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text. classification. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, 10.18653/V1/P1888 VV/P18- URL https:
pp. 328 339. Association for Computational Linguistics, 2018. doi: 10.18653/
//aclanthology/ org /r///11-//////
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In JCLR. Openlevvew.net, 2022.
Junjic Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/20033 11080, 2020. URL
11080
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2038 2038-2047. Association for Computational Linguistics, 2022. doi: 10. 10.18653/V1/222 WADDIGGS-MMTLLE
URL htttss:/ttt...oooo wtt.://///...11////////00000 www.:///...iiidinnid... findings-emnllp 148
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tan- moy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- 2021, Online, June 6-11, 2021, pp. URL https:
Association for Computational Linguisties, 2021. doi: 11551/0011665//122222 wtwpe/2...aaaaaa.eee... naacl-main. 228
10. /10033/VV222111AAAACO
Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André F. T. Martins, François Yvon, and Hinrich Schütze. Glot500: Scaling multilingual corpora and language models to 500 languages. In ACL (1), pp. 1082 1117. Association for Computational Linguistics, 2023.
Minbyul Jeong, Mujeen Sung, Gangwoo Kim, Donghyeon Kim, Wonjin Yoon, Jaehyo Yoo, and Jaewoo Kang, Transferability of natural language inference to biomedical question answering. In Linda Cappellato, Carsten Eickhoff, Nicola Ferro, and Aurélic Névéol (eds.), Working Notes of CLEF 2020 Conference and Labs of the Evaluation Forum, Thessoloniki, Greece, September 22-25, 2020, volume 2696 of CEUR Workshop Proceedings. CEUR-WS.org 2020. URL https::/ceurrws www..oor/ooooooo.. /aap////ppprr_pe 1alpp__4..pp0 44.
16

---

11B/://///221.NAACIL Published in Transactions on Machine Learning Research (12/2025)
Alistair Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic- iv, 2021. URL
Alistair F. W. Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, Li-wei H. Lehman, Leo A. Celi, and Roger G. Mark. Minic-iv, a freely accessible electronic health record dataset. Scientifie Data, 10(1):1, Jan 2023. ISSN 2052-4463. doi: 00.1038/s41197--00 -22-1199-22--1-999 URL https://doi.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11 (14(13):3521-355 2017. doi: 10.1073/pnas.160 161183514.
URL https::/ //www. /ww. www.//wwwwpp.........m.om..aa... pnas. org/ /tt.:/aa.....1.......//./ pow./...............////
pnas 1611835114
Anastasios Lamproudis, Thomas Olsen Svenning, Trond Torsvik, Taridzo Chomutare, Andrius Budrionis, Phuong Dinh Ngo, Thomas Vakili, and Hercules Dalianis. Using a large open clinical corpus for improved ied-10 diagnosis coding. In Proceedings of AMIA 2023, Annual Symposium, New Orleans, LA, USA, November 11-15 2023. URL https://www http:/wwwwwcci...ccco////....... ncbi. nlm. nih. gttp:/pme/arriiccess////1166665/
Simon A. Lee, Anthony Wu, and Jeffrey N. Chiang. Clinical modernbert: An efficient and long context encoder for biomedical text. CoRR, httes/250..09990 2025. doi: 10.48550/ARXTV 2504..0394.. URL
https://doi.org org/ 10. 48550//00ii.....///// 2504. 03964
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does BERT pretrained clinical notes reveal sensitive data? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 946-959. Association for Computational Linguistics, 2021. doi: 10. 18653/v1/2221.. naacl-main. 73
URL https:
7/doi. orggoogg///00
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In ICLR. OpenReview.net, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficint prompt tuning, In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punto Cana, Dominican Republic, 7-11 November, 2021, pp. 3045-3059. Association for Computational Linguistics, 2021. doi: 10.18653/V1 /2022 /ttp:///V//222.EMM.PAMMAMA EMNTP- URL https: //dts:://di...o wtg/
2021. emnlp-mainnnnn 243
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In ACL/IJCNLP (1), pp. 4582 4597. Association for Computational Linguistics, 2021.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter- efficient. fine-tuning. CoRR, abs/2303. 15442333,55677, 2023. doi: 10.48550/ARXT 2303.15647. URL https:
//doi.org/10 48550/8r0iv...././/// www.a..ww................. 15647
Yuchen Liang, Chaitanya K. Ryali, Benjamin Hoover, Leopold Grinberg, Saket Navlakha, Mohammed J. Zaki, and Dmilry Krotov. Can fruit fly learn word embeddings? In ICLR. OpenReview.ne,, 2021.
Seong Hoon Lim, Taejun Yun, Jinhyeon Kim, Jihun Choi, and Taeuk Kim. Analysis of multi-source language training in cross-lingual transfer. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 712-725. Association for Computational Linguistics, httt::://aoi..ogg/0..11///////// wwwp:00244clll1000
2024. doi: 10. 10.18653/VV 20133/112200 211/2024.AA/ ACI.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios A Anastasopoulos, Patrick Littell, and Graham Neubig. Choosing transfer languages for cross-lingual learning, In ACL (1), pp. 3125 3135. Association for Computational Linguistics, 2019.
Charles X Ling, Ganyu Wang, and Boyu Wang. Sparse and expandable network for google's pathways. Front. Big Data, 7:1348030, August 2024.
Chen Liu, Jonas Pfciffor, Ivan Vulic, and Iryna Gurevych. FUN with fisher: Improving generalization of adapter-based based lross-lingual transfer with scheduled unfreezing. In Kevin Duh, Helena Gómez-Adoroo, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mezico City, Mexico, June 16-21, 2024, pp. 1998-2015. Association for Computational Linguisties, 2024. doi: 2024. naacl-long. 111.
URL wttphttt:////e //ctps://doi.. http:////oooooo/////o/////// 186533/v1//
Yang Liu, Hua Cheng, Russell Klopfer, Matthew R. Gormley, and Thomas Schaaf. Effective convolutional attention network for multi-label clinical document classification. In Marie-Franche Moens, Xuanjing Huang, Lucia Specia, and Scott. Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5941-5953, Association for Computational Linguistics, 2021. doi: 10.18653/ 20111663/V/2222// 1122222.EEMLL-MAAAA
URL https://doli org/10/ 10. 18663/11/22221 http://m.2......mmimmmmm eenlp-main 481
Dan Malkin, Tomasz Limisicwiez, and Gabriel Stanovsky. A balanced data approach for evaluating cross- lingual transfer: Mapping the linguistic blood bank. In Marine Carpuat, Marie-Catherine de Marnefle, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 4903-4915. Association for Computational Linguisties, 2022. doi: 200G33/V1/022NNAAACCC
URL btttps://don http:://ooooo///////
org/ 100. 11....88/6/112.222 naacl maim..6nn.........
Pedro Henrique Martins, João Alves, Patrick Fernandes, Nuno Miguel Guerreiro, Ricardo Rei, M. Amin Farajian, Matcusz Klimaszewski, Duarte M. Alves, José Pombal, Nicolas Boizard, Manuel Faysso, Picrre Colombo, François Yvon, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. Eurollm-9b: Technical report. CoRR, abs/25066 abs/2506.04079) 2025.
Francisco Rogerlândio Martins-Melo, Alberto Novaes Ramos, Jr, Carlos Henrique Alencar, and Jorg Heukel- bach. Prevalence of chagas discase in brazil: a systematic review and meta-analysis. Acta Trop., 130: 167-174, February 2014.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Antonio Escalda-E Aitor Gonzalez-Agirre, Jordi Armengol-Estan Estapé, and Martin Krallinger. Overview of automatic clinical coding: annotations, guidelines, and solutions for non-english clinical cases at codiesp track of clef chealth 2020. In Working Notes of Conference and Labs of the Evaluation (CLEF) Forum CEUR Workshop Proceedings, 2020.
Antonio Miranda- Luis Gascó, Salvador Lima-López, Fulâlia Faduel, Darryl Estrada, Anas- tasios Nentidis, Anastasia Krithara, Georgios Katsimpras, Georgios Paliouras, and Martin Krallinger. Overviow of distemist at bioasq: Automatic dotection and normalization of discases from clinical texts: results, methods, evaluation and multilingual resources. In CLEF (Working Notes), volume 3180 of CEUR Workshop Proceedings, pp. 179-203. CEUR- WS.org 2022.
G M Naylor, T Gotoda, M Dixon, T Shimoda, L Gatta, R. Owen, D Tompkins, and A Axon. Why does japan have a high incidence of gastric cancer? comparison of gastritis between UK and japanese patients. Gut, 55(11):11465 11522, November 2006.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Dinh C. Nguyen, Quoc-Viee Viet Pham, Pubudu N. Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia A. Dobre, and Won-Joo Hwang. Federated learning for smart healthcare: A survey. ACM Comput. Surv., 55(3):60:1 60:37, 2023. doi: 10.1145/350129. URL https://doi org/10..1111000. 11453350111/33
Luiz E. S. E. Oliveira, Ana Carolina Peters, Anderson M. P. da Silva, Caroline P. Gebeluca, Yohan B. Gumiel, Luciana M. M. Cintho, Diego R. Carvalho, Sami Al Hasan, and Claudia M. C. Moro. Semclinbr a multi-instii and multi-specialtyy semantically annotated corpus for portuguese clinical nlp tasks.
Journal of Biomedical Semantics, 13( 1):13, 2022. doi: 10.1166/113326-02--000923 org/10. 1111///11332--22--026--999
URL https://doi.
Jens-Michalis Papaioannou, Paul Grundmann, Betty van Aken, Athanasios Samaras, Tlia Kyparissidis, George Giannakoulas, Felix A. Gers, and Alexander Löser. Cross-lingual knowledge transfer for clinical phenotyping. In LREC, pp. 900-909. European Language Resources Association, 2022.
Jonas Pfeiffer, Ivan Vulie, Iryna Gurevych, and Sebastian Ruder. MAD-X: an adapter- adapter-based framework for multi-task cross-lingual transfer. In EMNLP (1), pp. 7654-7673. Association for Computational Linguis- tics, 2020.
Fred Philippy, Siwen Guo, and Shohreh Haddadan. Towards a common understanding of contributing factors for cross-lingual transfer multilingual language models: A review. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. /10G3///V//223.ATTT..00
5877-5891. Association for Computational Linguisties, 2023. doi:
URL https://doi. 18633/11/0153/1/222 acl-long 323
Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? efficient inter- mediate task selection. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 10585- 10605. Association for Computational Linguisics,, 2021. doi: 10.18653/ C0153/V1/2000 /11//2221.EMM222 https://doi.. www.ooo.//...... 181...1116111110222. emnlp- maapp.mmann......m...mm..mmm...m 827
URL
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulic, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. Adapters: unified library for parameter-officient and modular transfer learning. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 System Demonstrations, Singapore, December 6-10, 2023, pp. 149-160. Association for Computational Linguistics, 2023. doi: 10.18653/V URL https:///oi.org: 10. 10653//11/2033 emnlp-demo 13
W0P.EENLP-DEMOO
Vitaly Protasov, Elisci Stakovskii, Ekaterina Voloshina, Tatiana Shavrina, and Alexander Panchenko. Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource lan- guages. In Atul Kr. Ojha, Chao-hong Liu, Ekaterina Vylomova, Flammie Pirinen, Jade Abbott, Jonathan Washington, Nathaniel Oco, Valentin Malykh, Varvara Logacheva, and Xiaobing Zhao (eds.), Proceedings of the Seventh Workshop Techmologies for Machine Translation of Resourer Languages (LoResMT 2024). pp. 94-108, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: 10.18653/ 2006533/w12024.oremmmtt 100. URL
org/2024. 122224..oreemm1...
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized em- beddings on large-scale structured electronic health records for disease prediction. npj Digit. Medicine, 4, 2021. doi: 00.1038/5A1746-001-006555
URL htt.ttts::/// https:///oi... org/ 10. 1038/ /1//://4111100000555555-011
Tom Roehr, Alexii Figueroa, Jens-Michalis Papaioannou, Conor Fallon, Keno K. Bressem, Wolfgang Nejdl, and Alexander Löser. Revisiting clinical outcome prediction for MIMIC-IV. In ClineauNLPONAACI, pp. 208-217. Association for Computational Linguistics, 2024.
Sebastian Ruder. Neural Transfer Learning for Natural Language Processing. PhD thesis, National University of Treland, Galway, 2019.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Konstantinos Sechidis, Grigorios Tsoumakas, and Ioannis P. Vlahavas. On the stratification of multi-label data. In ECML/PKDD (3), volume 6913 of Lecture Notes in Compuler Science, pp. 145-158. Springer, 2011.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR (Poster). OpenReview.non net, 2017.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, pp- 4077 4087, 2017.
Sai Ashish Somayajula, Youwei Liang, Li Zhang, Abhishek Singh, and Pengtao Xie. Generalizable and stable finetuning of pretrained language models on low-resource texts. In Kevin Duh, Helena Gómez- Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 4936-4953. Association for Computational Linguisties, 2024. doi: 10.18653/V1 /A0GG//111/22ANAAILLLOOCCC 2024. naacl-long.. 277.
URL BtLL:ttt::/ htttps://bo..org/10..o 10tg3/0.1865/v1/
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In NewrIPS, pp. 14193-24200 2021.
Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding, In David S. Touretzky, Michael Mozer, and Michael E. Hasselmo (eds.), Advances in Neural Information Processing Systems 8, NTPS, Denver, CO, USA, November 27-30, 1995, pp. 1048-1044. MIT Press, 1995.
Robert. Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Fine-tuning large neural language models for biomedical natural language processing. Patterns, 4(4):100729, 2023.
Thomas Vakili and Hercules Dalianis. Are clinical BERT models privacy preserving? the difficulty of extracting patient-oondition associations. In Thomas E. Doyle, Aisling Kelliher, Roza Samavi, Barbara Barry, Steven J. Yule, Sarah Parker, Michael D. Noseworthy, and Qian Yang (eds.), Proceedings of the AAAT 2021 Fall Symposium on, Human Partnership with Medical AT: Design, Operationalization, and Ethics (AAAI-HUMAN 2021), Virtual Event, November 4-6, 2021, volume 3068 of CEUR Workshop Proceedings. CEUR-WS.org. 2021. URL https://ceur--e ws. org Sete/Voo-306888/ 3068/ s0ott//oorr..p..
Betty van Aken, Jens-Michalis Papaioannou, Manuel Mayrdorfer, Klemens Budde, Felix A. Gers, and Alexan- der Löser, Clinical outcome prediction from admission notes using self-supervised knowledge integration. Tn FACL, pp. 881-893. Association for Computational Linguisties, 2021.
Betty van Aken, Jens-Michalis Papaioannou, Marcel Ganesh Naik, Georgios Eleftheriadis, Wolfgang Nejdl, Felix A. Gers, and Alexander Löser. This patient looks like that patient: Prototypical networks for interpretable diagnosis prediction from clinical text. In AACL/IJCNI (1), pp. 172 184. Association for Computational Linguistics, 2022.
Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang, Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, Kanmin Xue, Xiaoying Li, and Ying Chen. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concep trial. Nature Medicine, 29(10):2633 2642, Oct 2023. ISSN 1546-170X. doi: 111003//41191--222 241591-023-02552 025522--99
URL https://doi: org/ prg/10.1000 1038/
Benjamin Winter, Alexei Figueroa Rosero, Alexander Löser, Felix Alexander Gers, and Amy Siu. KIMERA: injecting domain knowledge into vacant transformer heads. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thir- teenth Language Resources and Evaluction Conference, LREC 2022, Marseille, France, 20-25 June 2022, pp. 363-373, European Language Resources Association, 2022. URL https: : http:///aaaatthog....oo wwwnthoooog.org/200000 2022. 1rec- 1.38
20

---

Published in Transactions on Machine Learning Research (12/2025)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedinggs of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 Demos, Online, November 16-20, 2020, pp. 38 45. Association for Computational Linguistics, 2020. doi: 10.18653/V1 Ww1.///2..E222-DDEEE.. URL https://doi. org/10. (19653/v1/200 2ww220000.mmpmp.. wwnee-eneeeeeeee demos.6
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Trans. Knowl. Discov. Data, 18(6), April 2024. ISSN 1556-4681, doi: 10.1145/3649506. URL
https://doi... org/ 10. 1145/3649506
Zhichao Yang, Shufan Wang, Bhanu Pratap Singh Rawat, Avijit Mitra, and Hong Yu. Knowledge injected prompt based fine-tuning for multi-label few-shot ICD coding. In Yoar Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.); Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 1767-1781, Association for Computational Lin- URL. https:: htt/:/...////.133333
guistics, 2022. doi: 10. N0.18633/11/220 FWA.O222FFNNNNGGS-AMMMOM 2022. www.:ffinnnnneeml.pp..
Alan L. Yuille and Norberto M. Grzywacz. A winner-take-all mechanism based on presynaptic inhibition feedback. Neural Comput., 1333:334-347, 1989. doi: 10.112//NEC0.1999..3.. URL https://doi.
org/10.110 11/20..11///cccc 1989. 1.3.334
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traun, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4791 4800. Association for Computational Linguistics, 2019. URL https://doi. http:////ooooogo/....o 1ttp://0..1663/11/11191991
doi: 111653/VV//19-/
Shurui Zhang, Bozheng Zhang, Fuxin Zhang, Bo Sang, and Wanchun Yang. Automatic ICD coding exploiting discourse structure and reconciled code emboddings. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Scung-Hoon Na (eds.). Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pp. 2883 2891. International Committee on Computational Linguistics, 2022. URL wttp:/2222.cooonn11.22222 254
A Datasets
We present in Table all data splits created with stratified sampling for the datasets used for training and evaluation in our experiments. DiSTEMIST comprehends the same 439 EHR notes in seven different languages, for a total of 3,073 samples.
B Impact of sequential training on downstream performance
We compare the average performance of models trained soquentially against models trained exclusively on a single dataset. Table t shows the absolute performance difference for every datasel. With very few exceptions, the performance boost is significant, highlighting the positive effect knowledge transfer in soquential training. SPONGE$ outperforms all models on all datascts on the average sequence when finetuned sequentially. S-PROTOXuOR outperforms XLM-R and XLM-R + A when trained and evaluated only MIMIC(single dataset). However, for the other datasets that are trained in isolation S-PROTOxuso is the
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: Dataset splits.
Dataset
Train
Val
Test
(M) MIMIC Johnson et al. 2023
102,199
9,358
7,618
(A)
Achopa Papaioannou
2022)
1,590
407
394
(S) Stockholm University Lamproudis
2023
1,583
256
232
(C) CodieEsp MMranda-Fssallan et. al. 2020
656
158
182
B) SemclinBr Oliveira et al. 2022)
453
107
109
DisTEMMST
AatrrandaEEscaldd et al. 2022
3,073 (439 X 7)
worst-performing model, indicating that it is not suitable for low-resource datasets. Morcover, SPONGE oulperforms all models also for the case of single datasets which shows the data elliciency of our approach.
Generally, all models perform better when sequentially trained, this is confirmed by the positive performance difference for both metrics (bottom of Table
Table 6: Performance difference when comparing sequential training results versus single-dataset training for every downstream dataset. We underline the best results when training on a single dataset (top), and boldface (middle) stands for the best results in sequential-transfer learning. All models benefit signifi- cantly(botton) from multi-step sequential transfer learning.
macro AUROC
macro PRAUC
Model/Dataset Dataset
C
5
PROTOSES
87.51
78.70
56.35
65.46
69.29
28.08
34.41
7.17
13.38
26.73
SPONGESS
89.70)
89.48
67.05
82.92
84.17
34.17
60.65
20.53
36.32
43.33
Single dataset
XLM-R
86.21
87.37
65.18
77.74
87.01
27 27:27
55.80
18:29
28. 18
63.80
XLM-R+A
85.74
81.43
63.68
79.69
87:35
26.38
52.08
16.73
31.54
53.12
S- PROTOOESA SPONGE
81.83
70.75
83.33
84.42
42.22
23.83
34.66
47.33
90.05
82.65
91.03
90.62
59.93
42.16
55.22
59.67
Sequence average
XLM-R
85.78
73.30
85.67
87. .6.3
58.85
27.73
39.20
57.90
XLM-R+A
84.83
75.93
86.38
88.08
55.65
28.54
38.77
57.52
S-PROTORES
+3.13
+ 14.20
+17.87
+ 15.13
+-7.81
16.66
+22128
+-20.60
Performance Difference
SPONGESA
+0.57
+15.6
+8.11
+6.45
-0.72
+21.63
+ +1.9
+16.44
XLM-R
-1.59
+8..12
+7.93
+0.62
+33.05
+9.44
+11.02
-5.90
XLM-R+A
+3.4
+12.25
+6.69
+0.73
+3.57
+11.81
+7.23
+4.4
C Hyperparameters
We use subnetwork exploration for 10 epochs to build distributed knowledge of MIMIC, with a batch size of 8 and learning rate as per Pfeiffer et al. (2020). We disable exploration for the remaining datasets of the training sequences. For S-PROTOxax.
use hyperparameters as in Figueroa et al. (202)). SPONGE and S- S-PROTOgma use the same hidden dimensions for the prototypical layer, XLM-R as Transformer, and are trained on four A100 GPUs.
Adapter choice We evaluate SPONGEL and X across the four low-resource datasets using different adapter types: prefix-tuning (Li & Liang 2021 LoRA (Hu et al.) 2022), prompt-tuning [Lester
et al. 2021) and bottleneck adapters (Pleiffer al., 2020 Consistent with findings from Poth et al. (2023)
bottleneck adapters are the most effective for downstream classification tasks for SPONGE. For XLM-R,
LoRA delivered the best performance. We use the best performing adapters for each model family. All models with adapters have significantly less trainable parameters than both S-PROTOxIxa and XLM-R.
Cross-lingual Transformer choice. Because of the large amount of experiments and the extent of our evaluation, we consider smaller Transformers like XLM-R instcad of significantly larger LLMs. We favor XLM-R over Glot500 [Imani et al. 2023) since the latter performs worse on classification tasks in all but one of the languages of our training datasets (see Table 19 in Tmani et al. 2023
22

---

Published in Transactions on Machine Learning Research (12/2025)
D Best performing sequence checkpoint of S-Proto for DisTEMIST
In Table we report the best performing sequence results of S-PROTOX and compare them BemClinBr(Portugueess
those of HSPONGE... This sequence corresponds to Mimic(English) CodiEsp(Spanish) or M-B-C. The two models resulting from this sequence significantly outperform the average scores over all sequences of all models reported in Table 3
Table 7: Sequence M + + C yields the best S-Proto model for DiSTEMIST, we report results for this sequence for HSPONNGE..
macto AUROC
micro AUROC
macro PRAUC
Model
S-Proto
86,16 92.26
84.73 91.89
25.25
HSPONGE
31.19
E All training sequences
In Figure 5 we list all generated 64 dataset sequences that start with MIMIC. Every node corresponds to the end of a sequence and model checkpoint.
F Trainable parameters
Table demonstrates the number of trainable parameters of the models in our experiments. We decompose these according lo the corresponding modules of the architectures. Although SPONGE features the largest number of competing subnetworks (90) it is the second smallest model w.r.t. the number of parameters that are updated. Note, that Hydra used only for inference and involves no training parameters. In our largest model and for our longest sequence of dalasets, Hydra adds roughly 1.8% paramelers when compared XLM-R, presenting no significant overhead.
Table 8: Number of subnetworks and trainable parameters for each model
Model
Subnetworks
Transformer
Adapters
Projection
Prototypical parameters (W and Total
S-PROTOeaa
1x5
270M 270M
0.19M
5x0.25M
271.5M
XLM-R
270M
SPONGE
6x3x5
6x0.88M
3x0.19M
5x0.25M
7.15M
SPONGE
6x1x5
6x0.88M
1x0.19M
5x0.25M
6.72M
SPONGELS
1x6x5
1x0.88M
6x0.19M
5x0.25M
5.99M
SPONGELI
1x1x5
1x0.88M
1x0.19M
5x0.25M
2.32M
XLM-R+A
0.88M
0.88M
G Experiments in a transfer learning scenario using same language datasets
Although the nature of clinical data is decentralized and often cross-lingual, we argue that sparse represen- tations can also be beneficial for knowledge transfer in standard monolingual settings. To evaluate this, we conduct the same sequential transfer learning experiments, for all the sequences of datasets as in Figure 5 however, with English translations. To translate the datascts we use EuroLLM (Martins et al. 2025) since it is an open source SOTA model that can run locally while preserving the privacy of the clinical records. In addition, we focus on biomedical encoders that are pretrained on English corpora, and are SOTA in multi- label classification for English clinical text (Grundmann al., 2025), namely, BiomedBERT (Gu et al. 2022) and ClinicalModemBERT (Lee al.) 2025). Following the approach described in Section performance metrics in Table g for the monolingual setting.
we report the
23

---

Published in Transactions on Machine Learning Research (12/2025)
B
M
5
B
s
B
Figure 5: All 64 training sequences that we evaluate in our experiments.
24

---

Published in Transactions on Machine Learning Research (12/2025)
From the two baselines BiomedBERT performs better, hence create versions of our most sparse model SPONG B as well as, S-PRROOORO with this Transformer encoder. We add to the results, the best performing model SPONGE from the cross-lingual setting (see Table for reference.
We highlight that all models exhibit better performance for the monolingual experiments when compared to the cross-lingual setting. This could be attributed to the specialized pre-training of BiomedBERT and its domain specific tokenizer. Nevertheless, consistent with the cross-lingual results of Table 1 SPONGE" E outperforms the other methods, suggesting that SPONGE can also boost performance in a standard mono- lingual transfer learning setting. When focusing only on the clinical outcome prediction task (van Aken et al. 2021) on the MIMIC dataset (left of Table we underline how the cross-lingual SPONGE XLM-R is still very competitive method even when compared to domain specialized models for English clinical text.
Table 9: Results of experiments in a monolingual (English) transfer learning scenario. (Left) model per- formance the MIMIC dataset. (Right) average performance the last dataset of each sequence. SPONGE consistently outperforms across both settings.
MIMIC Only
Sequential Training
Model
AUROC
micro AUROC
PRAUC
AUROC
miero AUROC
PRAUC
BERT
87.73
93.84
31.42
91.01
93.15
60.20
MonicalModeraBEERN
86.77
93.88
29.79
86.72
90.67
51.64
S PROTOB
88.73
94.42
35.83
91.21
93.50
58.91
SPONGE BRMMOBERT
90.45
95.08
39.51
92.63
94.64
66.11
SPONGE 6.33
89.70
94.70
34.17
88.58
92.09
54.24
25


Reviews:
[{"review_id": "vDqEU7Aa6t", "reviewer": "Reviewer_j6HD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n[S1] Evaluation Process: Constructing an evaluation procedure that closely resembles realistic conditions seems important. \n\n[S2] Empirical Performance: The empirical performance seems great over a variety of tasks/ datasets.\n\n## Weaknesses\n\n[W1] Structure: The paper has a loose structure only. As a result, judging what is an original contribution and what is taken from prior work is hard for the reader. For example, the method section was not enough for me to understand the proposed method, as details are spread over the main text, cf. the related work section, i.e., p.4 \"we use a parametric inhibition approach\". \n\n[W2] Related Work: Although there is a related work section, this section is very short for a 12-page manuscript, but it uses almost half of this space to introduce parts of the proposed method. Additional related work and concepts are introduced throughout the paper. This contributes to the difficulty of separating the original contribution. I.e., p.8 Experiments, p.5 Methods.\n\n[W3] Reproducability: The paper's main contribution is a clinical data model, which achieves great scores on the newly constructed evaluation procedure. Reproducing those results based on the manuscript seems impossible. While parameter counts and the general architecture are provided, no concrete specifics on the parts are given. Appendix C delegates almost all hyperparameters for the proposed method to prior work, making the original contribution questionable.\n\n[W4] Naming: While SPONGE and Hydra are catchy names, there is no justification for them. As a result, it just introduces additional overhead when reading. I prefer not to put a strong emphasis on a name - putting it in the title also does not help a reader judge whether this paper is relevant to them.\n\n[W5] The little math used in the paper is inconsistent:\n- p.6: The adapter $\\eta^t\\in\\mathbb{R}^{s\\times h}$ is defined to depend in size on the position of the index in the index set of tokens $s\\in E$. What is $\\mathbb{R}^{E}$ meant to represent? Do you mean a one-hot encoding $\\mathbb{R}^{|E|}$? Later, the inner product between $\\eta^t\\in\\mathbb{R}^{s\\times h}$ and $L^p$ (which is never defined) is of size $\\mathbb{R}^{C\\times E}$, which is inconsistent with the notation before.\n\n[W6] Dissemination of experiments: The description of individual experiments is very short, which makes it hard to judge the results. In addition, it remains unclear which algorithmic choice has which impact on performance, as numerous changes are evaluated simultaneously. In addition, the results are not always aligned with the conclusion, e.g., p.11 \"Hydra and label recency\" claims S-Proto and SPONGE \"underperform (wrt. what?) when not modified with Hydra\", but Table 3 shows that S-Proto performs *worse* on unseen data/ languages when modified with Hydra.\n\n[W7] Description of human evaluation: While it is certainly interesting to see how the models align with clinicians' judgments, that description on p.12 did not enable me to understand how these findings were collected, i.e., based on a multiple choice test, an interview, etc. This evaluation would have been particularly interesting if it had been carried out and described more systematically.\n\n[W8] Motivation of choices: Most of the architectural decisions are based on intuition and \"speculation\", p.5. As such, the value of this work to adjacent fields might be limited, as they might not generalize.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a method to train a model to combine knowledge across datasets that can only be accessed in sequence, a procedure they explain to be realistic when sharing data via their model across institutions. They construct a novel evaluation procedure by combining existing datasets to evaluate their work and achieve good results on this benchmark. \n\nDisclaimer: I am no expert in natural language processing for clinical data. As such, my review focuses on clarity and methodology.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "## Required\nWhile these changes are required, the paper will look very different if they are adopted. I cannot guarantee that those will be enough for me to recommend acceptance. Further reviews might be required. \n\n[R1] Overhaul the structure of the paper:\n- Add a structure section at the end of the introduction. \n- Separate introduction, methods, and related work. The introduction should describe the problem at hand and sketch your solution approach. Related work should focus on preliminary work and may list its limitations, but your method should be described in the method section.\n\n[R2] Explain your architecture. While you depict the architecture in Figure 2, it is almost not used to aid the explanations in 5.1.\n\n[R3] Properly introduce and explain the formulas in the method section (see [W5]). \n\n## Minor\n\n[M1] p.12 The citation (Zellers, et al., 2019) confused me, as it is placed in such a way as if it would support your claim, when its purpose is to explain \"annotation artifacts\". \n\n[M2] Change the names/ title, see ([W4]).\n\n[M3] Some acronyms are not introduced, ie. ICD-10, CCSR. The link to CCSR should be spelled out", "broader_impact_concerns": "As the proposed model takes no measures to avoid memorizing the data, the \"privacy constraint\" seems not to be respected without further measures, making the model vulnerable to potential extraction attacks (A). The authors should comment on this \nissue.\n\n(A): Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Raffel, C. (2021). Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21) (pp. 2633-2650).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U8uQkCVtX0", "reviewer": "Reviewer_Xo5B", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Advantages\n\nSPONSE, a novel architectural design, successfully integrates three distinct technical concepts—PEFT adapters, prototype networks, and a generalized WTA mechanism—into a cohesive whole. This innovation effectively addresses the core pain point of inter-task interference in multi-step sequence learning.\n\nThe Hydra strategy is integrated with SPONSE. Hydra is a simple, low-cost module applied only at inference time that accurately addresses the identified problem without increasing training complexity.\n\nCompared to testing on a single, fixed curriculum, this paper employs a large-scale, sequence-independent evaluation approach. This approach provides a more realistic and robust assessment of model performance, making the conclusions more convincing and generalizable.\n\nSPONGE achieves state-of-the-art performance while requiring only approximately 2.4% of the model's total parameters to train, a significant practical advantage. It significantly reduces the cost of training and adapting large models.\n\nThe model not only performs well across metrics, but also generates interpretable outputs recognized by domain experts. This is an important step in promoting the application of AI technology in high-risk fields such as healthcare.\n\n## Disadvantages\n\nWhile the Hydra strategy offers zero training overhead, it introduces significant complexity during inference. It requires storing and loading multiple sets of historical prototype parameters in memory. For very long learning sequences, this increases memory usage and computational latency during inference.\n\nThe WTA mechanism selects one subnetwork as the winner, while all other subnetworks are ignored. If the WTA selection process is affected by noise or makes a suboptimal choice for a particular input, the system becomes completely dependent on this suboptimal subnetwork, lacking the ability to correct errors or integrate signals from other \"near winners.\"\n\nThe evaluation scope is limited to Indo-European languages and a single diagnostic and predictive task. No effort has been made to include more complex languages, such as Chinese.\n\nThe entire framework was designed and evaluated for classification tasks. No effort has been made to apply the entire framework to generative tasks to evaluate model performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses a key challenge in clinical natural language processing (NLP): how to effectively transfer knowledge between models when data is isolated due to privacy constraints. The standard approach is sequence transfer learning, which often fails because new knowledge overwrites previously learned information—a problem known as \"catastrophic forgetting.\" To address this, this paper proposes a two-part solution: the SPONGE architecture, which prevents knowledge overwriting, and the Hydra strategy, which addresses the label recency bias problem. Key experimental results confirm that the SPONGE architecture is highly effective at sequential learning without forgetting, and when combined with the Hydra strategy, achieves state-of-the-art performance in generalizing to completely unseen data while being extremely parameter-efficient.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the disadvantage part.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cAqf3hC1zZ", "reviewer": "Reviewer_XPbz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper is generally well-written and organized clearly. There is a logical flow from one passage to the next, and illustrations help guide the reader through the narrative of the study.\n- The problem setting is explained well and properly motivated: in highly sensitive spaces like healthcare, it is often preferable to share models (weights) than data. This poses unique challenges that the paper then identifies and crafts a method to solve.\n- The release of open-source code to enable reproducibility is appreciated.\n- The experimental framework is unique and well-suited to the domain. Experiments are thorough and appear to be soundly conducted. There are many relevant experiments and ablation studies showing the value of the proposed approach.\n\n**Weaknesses**:\n- I would be interested in comparison to other baseline approaches. I am not intimately familiar with the space but I would imagine there are multiple recent efforts for clinical text understanding that might be competitive in diagnosis prediction other than S-Proto and XLM-R\n- I am left wondering whether the results and takeaways are driven by the cross-lingual nature of this specific experimental setup. In other words, is SPONGE/Hydra still useful (or as useful) in a *same-language* sequential transfer learning framework? This is a common (and still challenging) setting, where differences in patient demographics, labels, and many other factors can differ from dataset to dataset. If the authors could present some auxiliary results in a setting like this, then the practical applicability of this method might be improved – i.e., it is not always necessary to train a model with multi-lingual capabilities.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for diagnosis prediction from electronic health records (EHR) via multi-step sequential transfer learning. The proposed method, SPONGE, incorporates multiple task adapters and a prototypical network to enable diagnosis prediction with a winner-takes-all strategy among subnetworks. The architecture is parameter-efficient and further boosted by Hydra, a method to overcome “label recency”, where consecutively fine-tuned models tend to favor the most recent set of labels encountered. Models are evaluated on multiple fine-tuning sequences across datasets that vary in terms of language, label sets, imbalance, etc. Results show that the sparsest form of SPONGE outperforms other approaches in virtually all scenarios, Hydra improves performance in the zero-shot setting, and an early “exploration” phase in subnetwork selection provides an additional performance boost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Main Feedback**\n- If possible, include additional baseline methods for benchmarking. If not possible or prohibitive, please explain why.\n- Please add an experiment evaluating the proposed methods in a multi-step sequential transfer learning setting where language does *not* change.\n- Please clarify the “Analysis of saliencies with doctors.” Ideally, the methodology for this analysis should be described earlier in the Methods section. My main issue is that this paragraph does not provide any concrete results as far as I can tell. This paragraph does not reference a figure or table and seems to vaguely discuss the results of an experiment that I cannot find elsewhere in the paper. As written, I would simply remove this analysis and its corresponding sentence from the Abstract unless it can be clarified and supported with concrete experimental evidence.\n- Related Work: The first paragraph can be bolstered with more relevant background and related approaches on disease prediction from clinical text [1-5]. This is such a well-studied area that perhaps a summary/review paper would be useful to reference as well.\n\n**Minor Feedback**\n- Top of page 2: I might suggest semicolons to separate the numbered items\n- Section 4: Change “MIMIC – (High-resource)” to “MIMIC (High-resource)”? Unsure why the dash is included.\n- Experiments: I don’t understand what XLM-R + A means without further explanation.\n- Results: Change “This is evident when comparing SPONGE$^{6,1}$, SPONGE$^{1,6}$ against…” -> “This is evident when comparing SPONGE$^{6,1}$ and SPONGE$^{1,6}$ against…”\n- In Tables, I would write “XLM-R” for consistency with the text.\n- For Appendix F, also include the number of trainable parameters for all other SPONGE variants. \n\n**References**\n\n[1] He, Kai, et al. \"A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics.\" Information Fusion (2025): 102963.\n\n[2] Rasmy, Laila, et al. \"Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.\" NPJ digital medicine 4.1 (2021): 86.\n\n[3] Yang, Zhichao, et al. \"Knowledge injected prompt based fine-tuning for multi-label few-shot icd coding.\" Proceedings of the conference on empirical methods in natural language processing. Conference on empirical methods in natural language processing. Vol. 2022. 2022.\n\n[4] Liu, Yang, et al. \"Effective convolutional attention network for multi-label clinical document classification.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[5] Zhang, Shurui, et al. \"Automatic ICD coding exploiting discourse structure and reconciled code embeddings.\" Proceedings of the 29th International Conference on Computational Linguistics. 2022.", "broader_impact_concerns": "I don’t personally believe that a Broader Impacts statement is necessary for this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)