Repurposing Foundation Model for Generalizable Medical Time Series Classification

## Abstract

Abstract Medical time series (MedTS) classification is critical for a wide range of healthcare applications such as Alzheimer’s Disease diagnosis. However, its real-world deployment is severely challenged by poor generalizability due to inter- and intra-dataset heterogeneity in MedTS, including variations in channel configurations, time series lengths, and diagnostic tasks.
Here, we propose FORMED, a foundation classification model that leverages a pre-trained backbone
and tackles these challenges through re-purposing.
FORMED integrates the general representation learning enabled by the backbone foundation model and the medical domain knowledge gained on a curated cohort of MedTS datasets. FORMED can adapt seamlessly to unseen MedTS datasets, regardless of the number of channels, sample lengths, or medical tasks.
Experimental results show that, without any task-specific adaptation, the repurposed FORMED achieves performance that is competitive with, and often superior to, 11 baseline models trained specifically for each dataset. Furthermore, FORMED can effectively adapt to entirely new, unseen datasets, with lightweight parameter updates, consistently outperforming baselines. Our results highlight FORMED as a versatile and scalable model for a wide range of MedTS classification tasks, positioning it as a strong foundation model for future research in MedTS analysis. Code release upon acceptance.

## 1 Introduction

Figure: Figure 1: Paradigms of building models for different MedTS classification tasks. Task-Specific Model (TSM): Traditional classification models are designed for specific input shape and output classes, thus require retraining from scratch for each new dataset. Task-Specific Adaptation (TSA): By using a pre-trained and fixed backbone foundation models, the adaptation to new datasets requires training fewer parameters for each dataset, such as pre- and post-backbone adapters, which makes the combined model no longer applicable to other tasks, lacking generalization across tasks, and more prone to overfitting. Generalizable Adaptation: Generalizable adaptation is a post-backbone adaptation module that is shared across tasks of different datasets, which carries domain knowledge and transferable to unseen datasets with minimal training.
Refer to caption: x1.png

Medical time series (MedTS) classification, such as on electrocardiograms (ECG) and electroencephalograms (EEG),
is critical for a wide range of medical scenarios such as diagnosing Alzheimer’s Disease (AD; Jeong ()), Parkinson’s Disease (PD; Aljalal et al. (; )), and heart Arrhythmia (Jin et al., ).
Despite significant advancements in developing deep learning models for MedTS classification, several challenges still hinder their ability to generalize effectively across different datasets, and even among patients within the same dataset, posing a critical barrier to the successful translation
of predictive algorithms into real-world clinical settings.

As MedTS data can be of various modalities and dimensions, in real-world scenarios, they often differ from the training data in all aspects. Therefore, developing machine learning models that can generalize across these diverse datasets is essential for translating predictive algorithms into clinical settings. However, there are three unique key challenges in MedTS that a generalizable model will have to face:
(1) Inter-dataset Heterogeneity: The explicit characteristics of each dataset are diverse due to factors like the domain of physiological data, the equipment for data collection, etc. They require the model to be able to handle variations in the number of channels, sample duration, sampling rates, diagnostic targets, and so on (Ganapathy et al., ).
(2) Intra-dataset Heterogeneity: Given the intractable nature of the underlying physiological state, even within the same dataset, heterogeneity still exists across the time of recording, experimental session, and most significantly, among patients due to the presence of noise and artifacts (Wang et al., ; Ganapathy et al., ). As they are prevailing in the data, models are prone to overfitting to the training data, and thus show poor generalizability in real-world deployment.
(3) Data Insufficiency: The other challenges could have been solved provided with sufficient data for deep learning methods, while in reality, available MedTS datasets are often small due to costly data collection or simply privacy concerns (Kaushik et al., ). Consequently, it increases the difficulty of training robust models that handle the above challenges effectively (Ganapathy et al., ).

To this end, developing a foundation model for MedTS classification requires capturing and sharing medical domain knowledge across tasks. Previous attempt such as Yang et al. () adopts a Task-Specific Adaptation (TSA; see Figure 1) approach, in hope of capturing such knowledge in the backbone model. Yet the results of negative performance gain indicate that the model may only focus on extracting features that are informative for the training task, not of interest to future datasets, and thus lack generalization from task to task. Meanwhile, recent advances in foundation models bring sunlight for overcoming the above challenges by learning generic representations of time series data (Liang et al., ). However, they have focused predominantly on forecasting tasks (Ye et al., ; Wen et al., ), and simply applying TSA to adapt them for MedTS is not enough for capturing the sophisticated patterns for specific tasks from our preliminary results. Therefore, although they can serve as a great backbone model for extracting patterns in time series, dedicated effort in adaptation design is still required.

In this paper, we propose a novel approach to re-purpose foundation models pre-trained on large-scale, generic time series data for MedTS classification. We introduce FORMED, a Foundation model Repurposed for Medical time series classification, which achieves generalizable adaptation by seamlessly handling datasets with arbitrary channel configurations, dynamic time series lengths, and diverse diagnostic targets across multiple tasks (see Figure 1).
Specifically, FORMED employs a pre-trained foundation model as a backbone, which captures general temporal features from time series data.
We then adapt this backbone to the medical domain by integrating a specialized shell enriched with medical knowledge. This shell is trained on a curated cohort of MedTS data, enabling the model to effectively capture the unique characteristics of medical sequences.
Even with far fewer samples in the MedTS cohort than during pre-training, FORMED retains strong generalization abilities and domain knowledge for MedTS classification tasks.

We curate a repurposing cohort of 5 MedTS datasets, including 2 ECG’s and 3 EEG’s, containing only 340K samples or 90 million time-points in total.
These datasets feature diverse channel configurations (ranging from 12 to 33 channels), sample lengths (from 250 to 300 time-points), and diagnostic tasks (ranging from binary neurological to 5-class cardiovascular classification).
Our evaluation focuses on two aspects: First, for datasets partially included in the repurposing cohort, FORMED achieves superior performance on unseen patients, outperforming 11 state-of-the-art TSA and TSM models across all five datasets. Second, for a completely new dataset not included in the cohort, FORMED can be efficiently adapted by updating only a small number of parameters while still achieving the highest accuracy and AUROC compared to TSM and TSA models, even across different data availability scenarios.

## 2 Related Work

Foundation Models for General Time Series.
To date, no foundation model has been specifically designed for time series classification tasks, let alone MedTS classification; instead, recent advances in time series foundation models mainly concentrate on forecasting tasks (Liang et al., ).
Noticing their success in forecasting, it is as much tempting as theoretically and practically challenging to re-purpose these models for MedTS classification, yet these models all have major limitations such as design for univariate time series and requiring Task-Specific Adaptations (mid-column in Figure 1) that prevent them from being directly applicable to MedTS classification tasks (Cao et al., ; Sun et al., ; Chang et al., ).

For instance, Time-LLM (Jin et al., ), UniTime (Liu et al., ) and GPT4TS (Zhou et al., ) are all backboned on large language models, therefore they naturally handle time series data in a univariate manner, lacking the ability to integrate information across multiple channels for MedTS classification.
Moreover, we empirically observe that time series foundation models that take LLM as backbone don’t work well on time series datasets, in agree with Tan et al. ().
Similarly, although TimeGPT (Garza et al., ) and TimesFM (Das et al., ) are pre-trained on large scale time series data, they treat co-evolving multivariate time series data as independent of each other, thus sharing the same limitation as the previous models.
The one of its kind model, UniTS (Gao et al., ), is able to handle multivariate time series data and trained on multiple task domains including classification, yet due to its scale and design, it often requires fine-tuning of the whole model or performing prompt learning for optimal performance.
This is both computationally expensive for gradients calculation (Figure 1), and more importantly, data-greedy due to the massive parameters to tune, making it not suitable for small-scale MedTS datasets.

Therefore, despite their effort and success, current foundation models require significant adaptation to meet the demands of MedTS classification effectively. This motivates the need for a specialized foundation model tailored to the complexities of MedTS, which can address these limitations with dedicated architectural components.

Adaptation of Foundation Models for MedTS.
As general-purpose models, foundation models usually require various techniques to be effectively adapted for specific downstream tasks, including prompting, fine-tuning, re-programming and proposed re-purposing. Here we focus on re-programming and re-purposing as they can serve our purpose, see rest in Appendix A.

Re-programming: By reusing the pre-trained model’s backbone (usually all the Transformer layers) without altering its internal weights, it leverages the model’s existing capabilities. It is able to handle new domain of data or type of task, by wrapping the backbone model with input adapters and task heads (Figure 1 TSA). Yet on its dark side, the re-programmed model no longer serves as a general-purpose model, as both the input adapters and the task heads are task-specific, making it incapable of generalizing across tasks and datasets (Tan et al., ).

Re-purposing: Proposed in this work, it focuses on adapting the model to a new type of task with minimal modification to the task head only and refrains from being specific to certain task. Therefore, the repurposed model remains a general-purpose model for the field, which can serve as a new foundation model and be further adapted to new datasets efficiently (see Section 3). Its generalizability, data-efficiency and domain-expert nature make it extremely suitable for adapting time series foundation models for MedTS classification tasks.

Forecasting v.s. Classification.
Although both forecasting and classification are key tasks in time series analysis, they are fundamentally different in nature. The primary distinction lies in the relationship between the input and output spaces. In forecasting, the model predicts future values within the same domain as the input, i.e., mapping sequence $\rightarrow$ sequence (Lim & Zohren, ). For example, using past EEG signals to predict future EEG signals (Wang et al., ). In contrast, classification uses the input to predict a categorical label, i.e., sequence $\rightarrow$ category (Ali et al., ), such as diagnosing a neurological disease from EEG data (Wang et al., ).
In essence, forecasting only involves mapping input to output within the same domain, whereas in classification, the mapping is from one domain to a variety of others. Therefore, adapting a forecasting model for general classification tasks requires more than simply modifying the prediction layer; it demands a comprehensive redesign and a deeper understanding of the problem space.

## 3 Problem Statement

Figure: Figure 2: The three-stage process of adapting a time series foundation model for MedTS classification tasks. 1) Pre-training is already done on a cohort of diverse general time series datasets with forecasting tasks. 2) Repurposing the foundation model involves changing the forecasting head to a classification head, while keeping the rest of the model fixed, and the new model is then trained on a cohort of MedTS datasets to capture domain knowledge in MedTS. 3) Adapting the repurposed model to the new MedTS datasets with minimal training, where few parameters are adjusted for the new dataset and task, while the majority of the model remains fixed. The backbone foundation model is frozen in pre-training while trainable in repurposing and adapting. The classifier is trainable in repurposing while frozen in adapting.
Refer to caption: x2.png

Foundation models have showcased their capability in capturing general time series patterns, through pre-training on forecasting tasks, yet modifying them into foundation classification model for MedTS is not as straightforward. Here we define the problem and the key concepts involved in the adaptation process from a forecasting model into a general-purpose classification model.

###### Definition 1

Repurposing: The process of changing the objective of a pre-trained foundation model to a type of tasks that it was not originally trained for, by replacing and training a relative small output network while keeping the majority of the model fixed.

The original pre-trained model contains a backbone model $f$ for representation learning and a forecasting head $g$ that predicts the horizon from the representations. It takes the input ${\bm{x}}\in\mathbb{R}^{L}$ from last $L$ steps of a univariate time series and predicts horizon $\hat{{\bm{x}}}\in\mathbb{R}^{N}$ in the next $N$ steps, which is essentially a dynamic mapping as $L$ and $N$ can vary:

$$ $\displaystyle g\circ f:\mathbb{R}^{L}\rightarrow\mathbb{R}^{N}$ (1) $$

We replace the forecasting task head $f$ with a classification task head $h$, forming a new model that takes extra parameters ${\bm{E}}\in\mathbb{R}^{C\times D}$ and ${\bm{Q}}\in\mathbb{R}^{K\times D}$ for indicating the task-specific channels and classes, respectively, with $D$ as the model dimension, $C$ as the number of channels, and $K$ as the number of classes. The new model is then trained on a curated list of MedTS datasets $\mathcal{D}^{\textrm{Med}}$, where the input ${\bm{X}}\in\mathbb{R}^{C\times T}$ is a multivariate time series data with $C$ channels and $T$ time steps, and the output $\hat{{\bm{y}}}\in\Delta^{K}$ is a label prediction where $\Delta^{K}=\left\{{\bm{d}}\in[0,1]^{K}:\sum_{i=1}^{K}{d_{i}}=1\right\}$ is the probability simplex for $K$ classes. This is also a dynamic mapping as $C$, $T$ and $K$ may vary across datasets:

$$ $\displaystyle h\circ f:\mathbb{R}^{C\times T}\times\mathbb{R}^{C\times D} \times\mathbb{R}^{K\times D}\rightarrow\Delta^{K}$ (2) $$

###### Definition 2

Adapting: The process of adjusting a repurposed foundation model to new datasets and tasks with a few data- or task-related parameters, while keeping the majority of the model fixed.

After repurposing, the new model $h\circ f$ captures all the domain-specific knowledge in MedTS classification tasks, and can handle varying number of channels, length of input and number of classes (Section 4), therefore it is fixed for new coming datasets and tasks. For new datasets $\mathcal{D}^{\textrm{New}}$ with $C^{\prime}$ channels, $T^{\prime}$ time steps, and $K^{\prime}$ classes, the model is adapted to the new dataset by constructing the ${\bm{E}}^{\prime}\in\mathbb{R}^{C^{\prime}\times D}$ and ${\bm{Q}}^{\prime}\in\mathbb{R}^{K^{\prime}\times D}$, which is trained by calculating the loss $\mathcal{L}(\hat{{\bm{y}}},{\bm{y}})$ on the new dataset:

$$ $\displaystyle({\bm{E}}^{\prime},{\bm{Q}}^{\prime})=$ $\displaystyle\operatorname*{arg\,min}_{{\bm{E}}^{\prime},{\bm{Q}}^{\prime}} \sum_{({\bm{X}}^{\prime},{\bm{y}}^{\prime})\in\mathcal{D}^{\textrm{New}}}{ \mathcal{L}\left((h\circ f)({\bm{X}}^{\prime},{\bm{E}}^{\prime},{\bm{Q}}^{ \prime}),{\bm{y}}^{\prime}\right)}$ (3) $$

###### Definition 1

###### Definition 2

## 4 Model Architecture

### 4.1 Feature Extractor for Variable Length Time Series

Figure: Figure 3: The architecture of the proposed model in repurposing and adapting. The backbone foundation model acts as a feature extractor and remains frozen all the time. The Channel Embeddings (CEs) and Label Queries (LQs) are task-specific parameters that are learned during both repurposing and adapting, and new ones will be created and learned if encountering new datasets. The Shared Decoding Attention (SDA) is a shared Transformer decoder layer that captures the interaction between all the features and classes, which once get trained on curated MedTS datasets $\mathcal{D}^{\text{Med}}$ during repurposing, will be fixed and reused when adapting to all future datasets and tasks $\mathcal{D}^{\text{New}}$. The $\oplus$ denotes broadcast addition.
Refer to caption: x3.png

We take TimesFM (Das et al., ) as the backbone for repurposing based on our preliminary comparative analysis of existing time series foundation models.
TimesFM is pre-trained on a largest-scale dataset of diverse time series data for forecasting tasks and is able to capture general time series patterns within dynamic length of historical input. To repurpose it for MedTS classification, we can break down the model’s anatomy into three parts, the input patching network, the stacked Transformer, and the output prediction network.

Input Patching Network. Given a univariate time series input ${\bm{x}}\in\mathbb{R}^{T}$ and binary mask ${\bm{m}}\in\{0,1\}^{T}$ with length $T$, they are first broken up into patches ${\bm{X}}\in\mathbb{R}^{L\times P}$ and ${\bm{M}}\in\{0,1\}^{L\times P}$ in a non-overlapping fashion, where $P$ is the patch size and $L=\left\lceil\frac{T}{P}\right\rceil$ is the number of tokens. Each patch ${\bm{X}}_{i,:}$ is the concatenation of $P$ consecutive elements of the input sequence ${\bm{x}}$ in a non-overlapping fashion and so is the ${\bm{M}}_{i,:}$. The ${\bm{X}}_{i,:}$ and ${\bm{M}}_{i,:}$ denote the $i$-th row of ${\bm{X}}$ and ${\bm{M}}$, respectively. The sequence of patches ${\bm{X}}$ and ${\bm{M}}$ are then projected to a sequence of tokens ${\bm{Z}}\in\mathbb{R}^{L\times D}$ in the model dimension $D$ using an input residual block:

$$ $\displaystyle{\bm{Z}}_{i,:}=$ $\displaystyle\texttt{InputResidualBlock}({\bm{X}}_{i,:};{\bm{M}}_{i,:})$ (4) $$

Stacked Transformer. Before passing into the stacked Transformer, the positional encoding will be added to the tokens to form the input sequence $\tilde{\bm{Z}}\in\mathbb{R}^{L\times D}$. The stacked Transformer is then applied to the input sequence $\tilde{\bm{Z}}$ to capture the temporal dependencies and extract features using casual self-attention, outputting feature rich tokens ${\bm{H}}\in\mathbb{R}^{L\times D}$:

$$ $\displaystyle\tilde{\bm{Z}}_{i,:}=$ $\displaystyle{\bm{Z}}_{i,:}\oplus\texttt{PositionalEncoding}(i)$ (5) $\displaystyle{\bm{H}}_{i,:}=$ $\displaystyle\texttt{StackedTransformer}(\tilde{\bm{Z}}_{1,:},\tilde{\bm{Z}}_{ 2,:},...,\tilde{\bm{Z}}_{i,:};\dot{m}_{1},\dot{m}_{2},...,\dot{m}_{i})$ $$

where $\dot{m}_{i}=\min\{{\bm{M}}_{i,:}\}$ is the mask for the $i$-th patch for masking out completely empty ones.

Output Prediction Network. The output prediction network is a residual block layer that maps the last output ${\bm{H}}_{L,:}$ from the Transformer back to the original input spaces $\hat{\bm{x}}\in\mathbb{R}^{N}$, forming the prediction of the next $N$ time steps:

$$ $\displaystyle\hat{\bm{x}}=\texttt{OutputResidualBlock}({\bm{H}}_{L,:})$ (6) $$

In summary, the duty of prediction lies solely on the last output prediction network, while the input patching network plus the stacked Transformer can be viewed as a feature extractor that maps the input time series ${\bm{x}}$ to a sequence of feature tokens ${\bm{H}}$ (Figure 3). This can be easily extended to process multivariate MedTS by processing each channel of input individually and stack the extracted features as ${\bm{\mathsfit{H}}}\in\mathbb{R}^{C\times L\times D}$ for data of $C$ channels. This will serve as the backbone feature extractor for the downstream classification model.

### 4.2 Attention-based Classifier for Inconstant Channel and Class

Instead of using a simple linear classifier, as other time series classification models (Zerveas et al., ; Yang et al., ), which will require the input, output or both to have fixed number of channels and classes, we propose to use a Transformer decoder layer (Vaswani et al., ) for tackling such variability. Although inspired by techniques commonly employed in object detection (Carion et al., ) and image classification (Meng et al., ), it introduces key modifications tailored to address the unique challenges in MedTS classification. Our design is optimized for practical use, with an aim at handling dynamic shape input and outputing dynamic number of output classes, while reducing the computational overhead for training on new dataset and lowering risk of overfitting. Our attention-based classifier contains three key components: Channel Embeddings (CEs), Label Queries (LQs) and Shared Decoding Attention (SDA).

Channel Embeddings. As MedTS often exist in a multi-variate manner, injecting information about the channel will help the classifier distinguish between channels, thus promoting a more robust correspondence between the task and specific channel features. The Channel Embeddings are lightweight parameters that are grouped into a look-up table that maps the name of dataset to learnable channel embeddings ${\bm{E}}\in\mathbb{R}^{C\times D}$. These embeddings are then added to the feature tokens ${\bm{\mathsfit{H}}}$ to form the prompted feature tokens $\tilde{{\bm{\mathsfit{H}}}}\in\mathbb{R}^{C\times L\times D}$:

$$ $\displaystyle\tilde{{\bm{\mathsfit{H}}}}_{:,i,:}={\bm{\mathsfit{H}}}_{:,i,:} \oplus{\bm{E}}$ (7) $$

Label Queries. Just as CEs, the label queries ${\bm{Q}}\in\mathbb{R}^{K\times D}$ are also task-specific, learnable embeddings, where $K$ is the number of classes for the given task. These task-specific queries are used to guide the attention mechanism to focus on the relevant features for the specific task. The label queries independently attend to the prompted feature tokens ${\bm{\mathsfit{H}}}^{\prime}$ to find evidence for each class.

Shared Decoding Attention. The core evidence-finding process in classification is achieved through a shared decoding attention mechanism. It is a single decoder layer similar in Vaswani et al. (), that performs multi-head attention using ${\bm{Q}}$ as queries and $\tilde{{\bm{H}}}$ as keys and values, where $\tilde{{\bm{H}}}=\texttt{Flatten}(\tilde{{\bm{\mathsfit{H}}}})\in\mathbb{R}^{(
C\cdot L)\times D}$. It is followed by a residual block to obtain the logits $\hat{\bm{y}}\in\mathbb{R}^{K}$ for each class, where the probability prediction can be obtained using softmax or sigmoid functions depending on the type of task:

$$ $\displaystyle\hat{\bm{y}}=$ $\displaystyle\texttt{ResidualBlock}\left(\texttt{MultiHeadAttention}({\bm{Q}}, \tilde{{\bm{H}}},\tilde{{\bm{H}}})\right)$ (8) $$

Note that all the parameters in SDA is independent on either input length, number of channels, or number of classes, therefore it is able to handle the inconstant channel, length and class in MedTS classification tasks. Moreover, as it defines how the task queries will interact with the prompted feature tokens and is shared across datasets and tasks, it is coerced to learn a shared dynamics and form a domain knowledge that is fixed and can be reused in adapting to new classification tasks.

### 4.3 Repurposing and Adapting

During repurposing, the backbone foundation model is frozen, and the weights in SDA is randomly initialized. For each dataset in our MedTS cohort, a pair of ${\bm{E}}$ and ${\bm{Q}}$ is also randomly initialized. These are then trained over the MedTS cohort to update their parameters (see Repurposing in Figure 2). After repurposing, the SDA should already capture the domain knowledge required for MedTS classification, thus it will be fixed and reused when adapting to new datasets and tasks. For unseen datasets that need to be classified, a new pair of ${\bm{E}}$ and ${\bm{Q}}$ will be created and learned during adapting, while the majority of the model remains fixed (see Adapting in Figure 2).

Summary.
Our approach enables generalizability to new datasets, making it particularly suited for MedTS classification, and serves as a strong foundation model for all future MedTS tasks.
In particular, our design brings significant benefits in overcoming the aforementioned challenges:

- •
Generalizability Across Datasets: The backbone foundation model is able to capture general time series patterns and is fixed for all datasets, while the SDA is independent of channel number, input length or class number, so that it can ben shared across datasets, and gains domain knowledge during repurposing. This ensures that the model never overlooks general patterns in the data, and also gains sufficient domain knowledge for MedTS classification. FORMED can be effectively generalized to datasets with different sample length, channels, and classes.
- •
Generalization Across Subjects Within Dataset: The ${\bm{E}}$ and ${\bm{Q}}$ are the only task-specific parameters that are dependent on the dataset and task, and they are used to guide the model to focus on the relevant features for the specific task. As their number of parameters is very limited, it is highly unlikely to memorize the specific pattern of the training data, keeping the model highly performant across diverse patients.
- •
Lowered Data Requirement: As the SDA is shared in all tasks, it can be trained on a joint of diverse small MedTS datasets as a whole, without the need for a single, large and comprehensive dataset which doesn’t exist in practice. On the other hand, as the majority of the model parameters is fixed during adapting, the task-specific ${\bm{E}}$ and ${\bm{Q}}$ can be easily tuned with a little data from the new dataset. This design significantly reduces the data requirement for repurposing and adapting, making it particularly suitable for MedTS classification tasks with limited data.

## 5 Experiments

**Table 1: Results on MedTS Cohort for disease classification. Best results in non-TSM models are highlighted in bold, and the best results across all models are underlined. Our model, FORMED, consistently outperforms the other non-TSM model across all datasets on F1 along with many other metrics, and achieves highly competitive performance with SOTA TSM models. The delta values are shown in parentheses: lower delta values indicate more robustness against intra-dataset variances.**
| Datasets | Adaptation | Models | Accuracy | Precision | Recall | F1 score | AUROC | AUPRC |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PTB<br>(2-Classes) | TSM | Autoformer | 73.35 (17.17) | 72.11 (5.51) | 63.24 (8.45) | 63.69 (7.51) | 78.54 (2.78) | 74.25 (6.62) |
| Crossformer | 80.17 (11.51) | 85.04 (9.66) | 71.25 (7.19) | 72.75 (6.92) | 88.55 (3.64) | 87.31 (7.54) |  |  |
| FEDformer | 76.05 (15.54) | 77.58 (5.72) | 66.10 (8.12) | 67.14 (6.70) | 85.93 (3.01) | 82.59 (7.71) |  |  |
| Informer | 78.69 (13.96) | 82.87 (5.60) | 69.19 (7.54) | 70.84 (6.07) | 92.09 (1.77) | 90.02 (10.05) |  |  |
| iTransformer | 83.89 (6.14) | 88.25 (17.43) | 76.39 (3.05) | 79.06 (5.17) | 91.18 (1.80) | 90.93 (19.78) |  |  |
| MTST | 76.59 (18.40) | 79.88 (6.57) | 66.31 (14.20) | 67.38 (15.61) | 86.86 (4.61) | 83.75 (2.75) |  |  |
| Nonformer | 78.66 (14.59) | 82.77 (3.94) | 69.12 (9.66) | 70.90 (7.89) | 89.37 (1.22) | 86.67 (5.19) |  |  |
| PatchTST | 74.74 (20.40) | 76.94 (10.95) | 63.89 (15.42) | 64.36 (18.50) | 88.79 (5.47) | 83.39 (4.65) |  |  |
| Reformer | 77.96 (14.80) | 81.72 (4.22) | 68.20 (8.55) | 69.65 (7.36) | 91.13 (0.86) | 88.42 (9.28) |  |  |
| Transformer | 77.37 (15.43) | 81.84 (4.38) | 67.14 (10.22) | 68.47 (8.93) | 90.08 (2.08) | 87.22 (7.22) |  |  |
| TSA | PatchTST-TSA | 78.61 (11.68) | 80.32 (7.87) | 68.74 (2.87) | 70.07 (5.97) | 93.28 (1.51) | 97.15 (1.83) |  |
| GA | FORMED (Ours) | 86.24 (3.62) | 89.27 (7.20) | 79.36 (4.18) | 82.11 (4.19) | 95.45 (3.01) | 97.33 (1.08) |  |
| PTB-XL<br>(5-Classes) | TSM | Autoformer | 61.68 (0.87) | 51.60 (2.28) | 49.10 (1.53) | 48.85 (1.75) | 82.04 (0.82) | 51.93 (1.92) |
| Crossformer | 73.30 (1.37) | 65.06 (1.60) | 61.23 (1.83) | 62.59 (1.80) | 90.02 (0.66) | 67.43 (1.84) |  |  |
| FEDformer | 57.20 (0.46) | 52.38 (1.35) | 49.04 (1.27) | 47.89 (1.41) | 82.13 (0.52) | 52.31 (1.44) |  |  |
| Informer | 71.43 (1.36) | 62.64 (1.76) | 59.12 (2.20) | 60.44 (2.08) | 88.65 (0.81) | 64.76 (2.20) |  |  |
| iTransformer | 69.28 (0.83) | 59.59 (1.28) | 54.62 (1.58) | 56.20 (1.62) | 86.71 (0.73) | 60.27 (1.79) |  |  |
| MTST | 72.14 (1.00) | 63.84 (1.40) | 60.01 (1.64) | 61.43 (1.61) | 88.97 (0.64) | 65.83 (2.02) |  |  |
| Nonformer | 70.56 (1.36) | 61.57 (2.10) | 57.75 (2.33) | 59.10 (2.26) | 88.32 (0.94) | 63.40 (2.52) |  |  |
| PatchTST | 73.23 (1.07) | 65.70 (1.53) | 60.82 (1.90) | 62.61 (1.86) | 89.74 (0.60) | 67.32 (2.28) |  |  |
| Reformer | 71.72 (1.09) | 63.12 (1.34) | 59.20 (1.74) | 60.69 (1.60) | 88.80 (0.73) | 64.72 (1.98) |  |  |
| Transformer | 70.59 (1.25) | 61.57 (1.82) | 57.62 (2.04) | 59.05 (1.96) | 88.21 (0.81) | 63.36 (2.17) |  |  |
| TSA | PatchTST-TSA | 61.45 (0.69) | 53.38 (2.13) | 43.78 (1.43) | 44.41 (1.63) | 82.40 (0.66) | 51.36 (1.62) |  |
| GA | FORMED (Ours) | 71.31 (0.79) | 63.94 (1.87) | 56.40 (1.47) | 57.58 (1.77) | 88.44 (0.92) | 63.67 (2.65) |  |
| TDBrain<br>(2-Classes) | TSM | Autoformer | 87.33 (7.23) | 88.06 (6.72) | 87.33 (7.23) | 87.26 (7.29) | 93.81 (4.96) | 93.32 (5.42) |
| Crossformer | 81.56 (12.81) | 81.97 (12.47) | 81.56 (12.81) | 81.50 (12.87) | 91.20 (7.38) | 91.51 (7.08) |  |  |
| FEDformer | 78.13 (16.85) | 78.52 (16.56) | 78.13 (16.85) | 78.04 (16.93) | 86.56 (12.43) | 86.48 (12.51) |  |  |
| Informer | 89.02 (5.79) | 89.42 (5.66) | 89.02 (5.79) | 88.98 (5.82) | 96.64 (2.66) | 96.75 (2.57) |  |  |
| iTransformer | 74.67 (12.00) | 74.71 (12.07) | 74.67 (12.00) | 74.65 (12.00) | 83.37 (10.02) | 83.73 (9.60) |  |  |
| MTST | 76.96 (13.65) | 77.24 (14.51) | 76.96 (13.65) | 76.88 (13.65) | 85.27 (12.28) | 82.81 (13.93) |  |  |
| Nonformer | 87.88 (8.02) | 88.86 (7.16) | 87.88 (8.02) | 87.78 (8.11) | 97.05 (2.31) | 96.99 (2.35) |  |  |
| PatchTST | 79.25 (11.04) | 79.60 (11.82) | 79.25 (11.04) | 79.20 (11.01) | 87.95 (9.92) | 86.36 (11.10) |  |  |
| Reformer | 87.92 (7.02) | 88.64 (6.46) | 87.92 (7.02) | 87.85 (7.08) | 96.30 (2.92) | 96.40 (2.84) |  |  |
| Transformer | 87.17 (7.85) | 87.99 (7.19) | 87.17 (7.85) | 87.10 (7.92) | 96.28 (2.82) | 96.34 (2.74) |  |  |
| TSA | PatchTST-TSA | 75.50 (13.50) | 77.23 (12.45) | 75.50 (13.50) | 75.09 (13.86) | 82.28 (14.48) | 84.73 (12.19) |  |
| GA | FORMED (Ours) | 89.56 (3.42) | 89.94 (3.66) | 89.56 (3.42) | 89.53 (3.44) | 96.25 (2.84) | 96.89 (2.06) |  |
| APAVA<br>(2-Classes) | TSM | Autoformer | 68.64 (7.87) | 68.48 (8.33) | 68.77 (8.69) | 68.06 (8.20) | 75.94 (11.64) | 74.38 (11.74) |
| Crossformer | 73.77 (8.12) | 79.29 (6.07) | 68.86 (10.40) | 68.93 (11.18) | 72.39 (20.13) | 72.05 (19.55) |  |  |
| FEDformer | 74.94 (10.26) | 74.59 (8.07) | 73.56 (7.11) | 73.51 (8.89) | 83.72 (15.66) | 82.94 (17.12) |  |  |
| Informer | 73.11 (5.18) | 75.17 (5.93) | 69.17 (5.99) | 69.47 (6.49) | 70.46 (14.33) | 70.75 (14.59) |  |  |
| iTransformer | 74.55 (9.03) | 74.78 (8.49) | 71.76 (11.37) | 72.30 (10.78) | 85.59 (6.15) | 84.39 (6.90) |  |  |
| MTST | 71.14 (15.51) | 79.30 (8.65) | 65.27 (19.58) | 64.01 (21.71) | 68.87 (25.53) | 71.06 (22.50) |  |  |
| Nonformer | 71.89 (5.29) | 71.80 (5.85) | 69.44 (5.86) | 69.74 (5.96) | 70.55 (15.03) | 70.78 (14.44) |  |  |
| PatchTST | 67.03 (15.93) | 78.76 (6.74) | 59.91 (20.38) | 55.97 (25.34) | 65.65 (27.19) | 67.99 (24.14) |  |  |
| Reformer | 78.70 (2.33) | 82.50 (2.82) | 75.00 (3.19) | 75.93 (3.16) | 73.94 (14.21) | 76.04 (12.49) |  |  |
| Transformer | 76.30 (3.03) | 77.64 (3.67) | 73.09 (3.31) | 73.75 (3.55) | 72.50 (13.18) | 73.23 (12.77) |  |  |
| TSA | PatchTST-TSA | 69.80 (4.71) | 79.62 (13.96) | 63.49 (6.55) | 61.25 (7.41) | 74.78 (8.71) | 74.36 (12.24) |  |
| GA | FORMED (Ours) | 76.46 (8.84) | 77.11 (8.08) | 74.42 (11.68) | 74.65 (10.50) | 82.13 (11.86) | 83.69 (12.40) |  |
| ADFTD<br>(3-Classes) | TSM | Autoformer | 45.25 (4.53) | 43.66 (5.28) | 42.96 (6.02) | 42.59 (4.96) | 61.02 (4.80) | 43.10 (5.42) |
| Crossformer | 50.45 (7.53) | 45.57 (11.71) | 45.88 (11.27) | 45.50 (11.51) | 66.45 (4.77) | 48.33 (6.26) |  |  |
| FEDformer | 46.30 (4.79) | 46.05 (4.52) | 44.22 (5.82) | 43.91 (4.52) | 62.62 (4.98) | 46.11 (5.41) |  |  |
| Informer | 48.45 (5.12) | 46.54 (6.95) | 46.06 (6.15) | 45.74 (5.94) | 65.87 (2.26) | 47.60 (4.22) |  |  |
| iTransformer | 52.60 (2.79) | 46.79 (6.02) | 47.28 (6.30) | 46.80 (5.83) | 67.26 (3.29) | 49.53 (3.93) |  |  |
| MTST | 45.60 (3.30) | 44.70 (2.73) | 45.05 (2.65) | 44.31 (2.60) | 62.50 (2.36) | 45.16 (2.27) |  |  |
| Nonformer | 49.95 (2.87) | 47.71 (5.54) | 47.46 (4.39) | 46.96 (4.66) | 66.23 (2.22) | 47.33 (5.89) |  |  |
| PatchTST | 44.37 (7.57) | 42.40 (7.97) | 42.06 (8.24) | 41.97 (7.12) | 60.08 (8.03) | 42.49 (8.78) |  |  |
| Reformer | 50.78 (2.18) | 49.64 (4.08) | 49.89 (2.30) | 47.94 (2.62) | 69.17 (2.03) | 51.73 (3.93) |  |  |
| Transformer | 50.47 (3.49) | 49.13 (4.48) | 48.01 (3.84) | 48.09 (3.83) | 67.93 (2.40) | 48.93 (3.92) |  |  |
| TSA | PatchTST-TSA | 50.95 (5.90) | 53.34 (7.91) | 43.50 (1.47) | 40.61 (3.93) | 62.77 (3.56) | 46.89 (5.80) |  |
| GA | FORMED (Ours) | 47.76 (2.54) | 46.58 (2.27) | 43.26 (3.92) | 43.05 (2.46) | 61.70 (2.82) | 44.31 (2.46) |  |

Datasets.
We select 5 MedTS datasets to formulate a MedTS cohort and use it for repurposing (Figure 2).
These datasets provide a broad range of physiological signals, capturing both cardiac and neurological activity, which are among the most commonly analyzed modalities in MedTS.
See Table 3 for details on the datasets. Moreover, we also include an unseen, out-of-domain dataset (Liu et al., ) to assess our model’s ability to generalize to new tasks. All datasets are split into train-test-valid sets following the patient-independent setting as in Wang et al. ().
The datasets span a wide range of channels, sampling rates, sample durations, and disease labels, allowing for the evaluation of inter-dataset heterogeneity. We use inter-subject variation, a key contributor to intra-dataset heterogeneity (Wang et al., ), as a proxy to assess the generalization capability of FORMED.

Baselines. We compare FORMED with 11 SOTA baselines including 10 TSM and 1 TSA models.
The TSM models, including Autoformer (Wu et al., ), Crossformer (Zhang & Yan, ), FEDformer (Zhou et al., ), Informer (Zhou et al., ), iTransformer (Liu et al., ), MTST (Zhang et al., ), Nonformer (Liu et al., ), PatchTST (Nie et al., ), Reformer (Kitaev et al., ) and Transformer (Vaswani et al., ), are included for comparing our model’s performance on seen tasks to verify the applicability of repurposing.
The additional TSA model, PatchTST-TSA, is modified from PatchTST by adding task-specific classification heads on top of the backbone model and trained on all datasets jointly from scratch. Due to its architectural similarity to TimesFM (Das et al., ), we use it to evaluate both the quality of repurposing and adapting.

Evaluations. The effectiveness of our method is demonstrated through the performance in terms of accuracy, precision, recall, F1 score, AUROC, and AUPRC, evaluated on the test sets. Additionally, the robustness of the models against intra-dataset distribution discrepancies is assessed by comparing delta values, i.e., the absolute difference between the performance on validation and test sets. The generalization ability of the models to unseen tasks is evaluated by conducting few-shot adapting experiments on a small, unseen, out-of-domain dataset. These experiments are conducted on five random seeds for all models, and the results are averaged across the seeds.

### 5.1 Evaluation on Repurposing: Generalize to Unseen Subjects

Figure: Figure 4: Evaluation of model consistency and robustness across six metrics: accuracy, precision, recall, F1, AUROC, and AUPRC. X-axis: delta values, calculated as the absolute difference between validation and test sets, lower is better; Y-axis: models for comparison, ordered by average delta values. The delta values are collected from 5 datasets for each model and each metric. The range of delta values (minimum and maximum) are indicated by the horizontal lines, and the average delta values are shown with vertical marks. Joint training of multiple datasets helps to reduce the delta values (compare PatchTST-TSA with PatchTST), yet it still falls far behind many other models including ours. Our model consistently exhibits smaller delta values across all metrics, indicating superior robustness and consistency against distributional discrepancies among subjects.
Refer to caption: x4.png

Setup.
For repurposing datasets in MedTS cohort, we trained 50 TSM models (10 models for each), and 1 TSA model but with 5 task-specific heads. our one FORMED model is trained on all 5 datasets with no change to it during repurposing.

Effectiveness of Repurposing.
We find that repurposing with a generalizable adaptation layer is more effective than TSM and TSA methods in classification tasks.
As shown in Table 1, our model surpasses the TSA model in F1 across all datasets, as well as many other metrics. On top of that, it achieves competitive performance compared to the TSM models, if not better, on most datasets.
These findings demonstrate the overall effectiveness of our proposed repurposing framework.

Quality of Repurposing.
The repurposing also grants the model more robustness towards intra-dataset discrepancies across subjects.
The delta values of our repurposed model across six key metrics Figure 4 outperform all 11 baselines, showcasing its consistency and robustness against such variations in data.
This implies the applicability of our methods towards real-world healthcare usage, where the subject population at the time of testing is often not fully represented in the training data.

### 5.2 Evaluation on Adapting: Generalize to Unseen Task

Figure: Figure 5: Performance on few-shot adapting to small, unseen, out-of-domain dataset. Numbers of trainable parameters are included in parenthesis. The performance is plotted against the ratio of available training data. FORMED dominates other models across all data ratios in both metrics.
Refer to caption: x5.png

Setup.
In few-shot adapting evaluation, we use a small out-of-domain dataset with a limited amount of training data and a binary classification task. The data is recordings of phonocardiogram (PCG), and is pre-processed into spectrogram time series, where 61 channels each represent a different frequency band. The PatchTST-TSA previously trained on MedTS cohort is modified with a new head, and the PatchTST with reduced parameters is also included for a fair comparison of both the PatchTST-TSA and our model. Our model is frozen, and only the newly added channel embedding and task query are learnable.

Results.
We find that adapting to even drastically different dataset and different task is easily achievable with our model.
Despite the such inter-dataset heterogeneity, our model outperforms all baselines across all data ratios in both accuracy and AUROC Figure 5. Interestingly, the PatchTST’s performance drops with more available data, and a potential explanation to it is that it quickly memorizes the few training data and comes to an early stop. Nonetheless, our method demonstrates the superior generalization ability to unseen tasks, a significant advantage for use in real-world healthcare applications, where new tasks may arise frequently, and the expert-labeled data is often limited in such field.

## 6 Conclusion and Discussion

In this paper, we present FORMED, a foundation model for MedTS classification, that leverages a pre-trained backbone that can capture general time series patterns and a generalizable adaptation head to repurpose the model and capture domain-specific knowledge.
We demonstrate that FORMED can effectively generalize both within and across datasets, providing superior performance with more robustness against distribution discrepancies compared to state-of-the-art models, and can be seamlessly adapted to unseen MedTS datasets with lightweight training. Next, we discuss the potential impact of our work, the limitations, and future directions.

Potential Impact.
Our work has mainly focused on field of MedTS classification, where leakage of patient information and bias in the model are critical concerns. Regarding the former, we only use datasets that are publicly available and have been de-identified, and the details and sources of them are provided in Table 3. As for the latter, we have taken steps to ensure that our model is fair, such as using a backbone model that has been pre-trained on the largest dataset to capture more general time series patterns, and no covariate information is used other than dataset-level embeddings. Yet, we acknowledge that there may still be biases in the data that we have not accounted for, and we are to release the weights of our model along with a detailed model card (Mitchell et al., ) for our community to assess the potential bias and privacy concerns in a joint effort.

Relation to TimesFM.
The TimesFM (Das et al., ) is a foundation model whose sole purpose is time series forecasting, and by repurposing it, we create FORMED which is now a foundation model for medical time series classification, fundamentally different from TimesFM.

Backbone and Repurposing Domain.
The proposed repurposing-and-adapting framework is not limited to specific backbone model, and can be applied to other time series foundation models with similar anatomy.
Our framework can repurpose to other domains like weather forecasting, financial tasks, etc., by simply using a domain-specific repurposing data cohort.

Computation Efficiency.
Computational cost is of great concern for large foundation models, especially so when the model needs to be frequently adapted to new downstream tasks (Hu et al., ). We have recognized such need and already incorporated several strategies to make our model more computationally efficient. By freezing the backbone model and omitting pre-backbone adapters, the gradients do not need to be back-propagated through the backbone model during repurposing (Figure 1), which significantly reduces the computational cost. Moreover, we take what is categorized as an external memorization approach (Wang et al., ), where new knowledge of specific tasks is stored in the task-specific embeddings and queries, rather than tuning the model parameters, which further reduces the computational cost at the adaptation stage. On the whole, our model is designed to be computationally efficient and scalable to larger datasets and more complex tasks.

Interpretability and Explainability.
When it comes to medical applications, interpretability and explainability are crucial for the model to be trusted and adopted by healthcare professionals. As our model is fully transformer-based, it can harness the power of tools that dissect the attention mechanism like Chefer et al. (); Hao et al. (). Moreover, the task-specific knowledge is explicitly stored in channel embeddings and label queries, which can be used to compare and explain the model’s behavior across different tasks. However, all these are beyond the scope of this paper and deserves to be explored in future work.

## Appendix A Comparison of Adaptation Techniques in Foundation Models

As discussed in Section 2, adaptation techniques for foundation models mainly includes Prompting, Fine-tuning, Re-programming, and Re-purposing. We have introduced re-programming and re-purposing, and here we provide a brief overview of prompting and fine-tuning, and compare these techniques based on three aspects: Data Efficiency, New Task Type, and Generalizability.

Prompting & Fine-tuning: Both are common adaptation techniques for foundation models, where prompting involves conditioning the model with specific instructions or cues, either handcrafted (Zhou et al., ; Reynolds & McDonell, ) or learned through data (Zhou et al., ), and fine-tuning involves updating the model’s internal parameters on dedicated dataset (Howard & Ruder, ; Ding et al., ). While they focus on different aspects of adaptation, they share the commonality of not altering the model’s core architecture, therefore the functionality of the model remains unchanged, e.g., model for forecasting remains a forecasting model. Moreover, fine-tuning is often more data-greedy, as it requires updating the whole model’s parameters, while prompting only requires learning a few task-specific embeddings or prompts.

In general, these techniques can be categorized based on three aspects: Data efficiency, as the scale of dataset used for adaptation, typically measured by the number of parameters updated; New Task Type, as the ability to adapt to new tasks that are different from the original task, such as from forecasting to classification; and Generalizability, as the ability for the adapted model to be used on unseen datasets and share knowledge across tasks. Table 2 provides a comparison of these techniques based on these aspects.

**Table 2: Comparison of adaptation techniques of time series foundation models.**
| Adaptation | Data Efficiency | New Task Type | Generalizability |
| --- | --- | --- | --- |
| Prompting | ✓ |  | ✓(^1^11Although the model structure is fixed and still applicable to other datasets and tasks, the engineered or learned prompts can be task-specific.) |
| Fine-tuning |  |  | ✓ |
| Re-programming |  | ✓ |  |
| Re-purposing | ✓ | ✓ | ✓ |

## Appendix B Datasets

Here we provide the details of the datasets Table 3 used as the MedTS cohort for repurposing in Section 5. The datasets are publicly available, and we follow the pre-processing and splitting procedures as in Wang et al. ().

**Table 3: MedTS Cohort Datasets.**
| Dataset | Type | # Subject | # Sample | Sampling<br>Rate | Sampling<br>Length | # Channel | # Classes |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PTB<br>(Goldberger et al., ) | ECG | $198$ | $64\,356$ | \qty250 | $300$ | $15$ | $2$ |
| PTB-XL<br>(Wagner et al., ) | ECG | $17\,596$ | $191\,400$ | \qty250 | $250$ | $12$ | $5$ |
| TDBrain<br>(van Dijk et al., ) | EEG | $72$ | $6240$ | \qty256 | $256$ | $33$ | $2$ |
| APAVA<br>(Escudero et al., ) | EEG | $23$ | $5967$ | \qty256 | $256$ | $16$ | $2$ |
| ADFTD<br>(Miltiadous et al., ; ) | EEG | $88$ | $69\,762$ | \qty256 | $256$ | $19$ | $3$ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: I have a few questions, and I’m unsure whether they represent limitations of the paper or simply my own confusion. Could you please help clarify?

1. Task-Specific Design: Is the proposed method in this paper specific to classification tasks? If I wanted to adapt it to a different type of task, such as regression or anomaly detection, would it require retraining a new layer to replace the very large classifier?


2. Use of Transformer Decoder Layer for Classification: Instead of a simple linear classifier, the paper employs a Transformer decoder layer called Shared Decoding Attention (SDA). Since the SDA mechanism only generates an output embedding, I’m unclear on how a decoder layer or attention module can perform classification directly. Could you clarify the role of this module in classification?

    I assume it applied label query of **Q \in K \times D** and generates output embedding with a shape of ** K \times D**. And how will it make classification?

3. How does this SDA module adapt to different classification tasks? The paper claims that the attention module gains domain knowledge during the re-purposing stage, but I am uncertain how an attention layer trained on a limited biosignal dataset can generalize to unseen data.

    For example, if the model is re-purposed by training this attention layer on ECG and EEG signals, it might learn domain knowledge specific to heart or brain functions. It is reasonably adapted to related signals like heart rate during the adapting stage. However, how would this model handle entirely unrelated signals in unseen datasets (i.e., inter-dataset Heterogeneity), such as body temperature, pulse rate, respiration rate, or blood pressure?
soundness: 3
strengths: I believe the main strength of this study lies in its clear motivation and the reasonable solutions it proposes.

1. This study clearly addresses the existing challenges, such as inter-dataset heterogeneity, intra-dataset heterogeneity, and data insufficiency. For instance, a biosignal dataset may contain recordings from multiple participants, each with a unique health status, leading to notable intra-dataset heterogeneity. Additionally, across different datasets, biosignals often exhibit distinct patterns; for example, ECG and heart rate data both describe heart function but follow completely different temporal patterns.

2. It clearly identifies the challenges in adapting foundation models for the time-series domain, particularly the need for a dataset-specific alignment module and output layer, which limits the model’s generalization across different datasets.

3. This study proposes a reasonable two-stage solution. In the stage of re-purporsing, it introduces a generalization classifier designed to capture the domain knowledge, through the transformer's attention module. In the adaptation stage, it learns the channel embedding and label query specifically tailored to each dataset and task.

4. In the adaptation stage, it requires learning only 30k parameters, making minimal modifications to the task head and avoiding specificity to a single task.
summary: The paper introduces FORMED, a new generalization method for foundation models. It introduces a novel mechnanism called re-purposing, designed for generalizable medical time series (MedTS) classification tasks. Generalization in medical time series data is challenging due to inter- and intra-dataset heterogeneity and data insufficiency, which often hinder model adaptability across various datasets. To overcome these issues, FORMED uses a pre-trained foundation model, TimesFM, as a backbone model and employs a two-stage procedure of re-purposing and adapting. In the re-purposing stage,  it learns the weights of the channel embedding, label query, and classifier. During the stage of adapting, it learns only the weights of the channel embedding and label query, while keeping the classifier frozen. This setup allows the channel embedding and label query to be tailored to specific datasets and tasks,  while the classifier retains essential domain knowledge. As a result, FORMED can adapt effectively to new datasets with varying channel configurations, lengths, and diagnostic tasks. Through evaluations on a curated cohort of five MedTS datasets, the model consistently outperforms traditional task-specific and task-adaptive models, maintaining high performance. 

I believe this paper aims to introduce a novel transfer learning paradigm specifically for medical time series dataasets. However, I think some modules and configurations are unclear and require more details. Additionally, based on my experience with biosignal data, the performance appears suboptimal compared to existing methods. Furthermore, the model still requires training a very large classifier (8 million parameters), and given its relatively poor performance, I am not seeing clear advantages. If these points can be addressed, I would consider revisiting my assessment.
weaknesses: The primary weakness of this paper is its suboptimal performance on medical time series datasets compared to existing methods. Additionally, some modules and configurations are not clearly explained. Given that experimental performance is the main concern, I will begin by discussing limitations in the experiments and model comparisons.

1. In the two-stage solution, its re-purposing stage involves a training of large classifier with 8 million parameters. This parameter size is actually larger than that of some pre-trained models on the medical time series data, which raises questions about the benefits of this adaption approach.


    For instance, Cross Reconstruction Transformer (CRT) introduces a dropping-and-reconstruction pre-training paradigm. Its default setup is 6 encoder layers, 2 decoder layers, and an embedding size of 128, resulting in a model with 3.9 million parameters. Both this study and the CRT paper use PTB-XL biosignal data, which is the largest biosignal dataset used in this study. Consequently, a 4-million-parameter Transformer is sufficient for pretraining, while this adaption method needs to train a classifier with 8 million parameters. It is problematic if domain adaptation requires an even larger adapter, as this undermines the idea of efficient adaptation.

    [1] Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer. IEEE NNLS, 2023. 

2. The classification performance of this paper actually is not impressive. In terms of PTB dataset, it is a very simple binary classification task. As it is simple, we did not really see recent works about pre-training or adaption make it as benchmark. For example, a simple convolution-based model with thousands of parameters can achieve an accuracy of 95%, while the baseline of RNN and SVM can also achieve over 90% [2]. However, in this paper, the proposed method and Transformer baselines are ranged from 73% to 86%, signficantly lower than exisitng methods. 

    [2] ECG Heartbeat Classification: A Deep Transferable Representation. ICHI, 2018.

3. The experimental results on the PTB-XL dataset are also lower than those of existing methods. For instance, this paper references the Biosignal Transformer, which achieves a balanced accuracy of 84.21%, an AUPRC of 92.21%, and an AUROC of 76.59%. Additionally, the CRT model reports an accuracy of 87.81% and an AUROC of 89.22%. In contrast, FORMED achieves a balanced accuracy of 71.31%, an AUPRC of 63.67%, and an AUROC of 88.44%. Other related works on this dataset also generally achieve accuracy scores over 80%.

    [3] BIOT: Biosignal Transformer for Cross-data Learning in the Wild. NeurIPS, 2023.

4. The experiments also do not show the advantages of FORMED compared to Task-Specific Model (TSM). For example, the baseline among TSMs are 2% higher than FORMED. Morever, most of baselines are designed for forecasting, like Informer, Autoformer, Fedformer, which may not be suitable baseline. This paper may include Transformer-based baselines for classification tasks. 

5. The datasets used in this study are relatively easy for classification tasks due to their shorter sampling lengths, ranging from 250 to 300. Specifically, this study uses PTB-XL with a sequence length of 250. In contrast, CRT employs a sequence length of 5000, and BIOT uses a sequence length of 2500, making those tasks more challenging.

6. I also have questions about the rationale behind using the foundation model TimesFM. This paper states, "Repurposing the foundation model involves changing the forecasting head to a classification head", and "Foundation models have showcased their capability in capturing general time series patterns, through pre-training on forecasting tasks". Here, the forecasting pre-training task implies using a sequence to predict another sequence, whereas next-token prediction refers to using a historical sequence to predict the next token.


    However, as far as I know, most foundation models rely on next-token prediction rather than forecasting tasks. It should be noted that TimesFM, with its forecasting-based pre-training, is an unusual case. In comparison, next-token prediction is generally better suited for learning generalizable knowledge, often enabling zero-shot or few-shot learning capabilities [4,5].


    As a result, I am uncertain about the rationale for re-purposing by changing the head, as **the primary purpose** of a foundation model is to capture inherent generalizable knowledge.


    [4] Language Models are Unsupervised Multitask Learners.


    [5] Language Models are Few-Shot Learners.


7. Although this study highlights the challenges of domain adaptation in medical time series data, such as inter-dataset heterogeneity, intra-dataset heterogeneity, and data insufficiency, no experiments demonstrate these aspects. 

    Aside from the benchmarking results, there is a lack of informative ablation studies, and no experiments are provided to support various claims made in the paper.


8. Few typos, such as line 202 in page 4 (f->g)

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: - At line 160/161, you state: "adapting a forecasting model for general classification tasks requires more than simply modifying the prediction layer; it demands a comprehensive redesign and a deeper understanding of the problem space". Would you happen to have a reference on the matter? My understanding was that fine-tuning a task head was in fact quite successful at adapting models to new tasks (from forecasting to classification, from classification to regression, etc), but I would like more information on this subject.
- How do you explain the large delta values that can be observed for some models (for example, 10 points of F1 Score for FORMED on APAVA)? Is there a meaningful difference between your validation and test sets, in size for example?
- For clarification, when adapting FORMED on the PCG dataset, was the SDA trained using all datasets in the MedTS cohort for repurposing or only one of them? If so, which one?
soundness: 2
strengths: **Originality :** As remarked upon by the authors, research efforts on medical TS often involve task-specific models, or in the best cases task heads that need to be trained for each task. Moreover, when the input dimensions change (apart from the sequence length, whose variation is well handled by the Transformer architecture), input adapters also need to be trained. The authors' approach is original in the sense that they do away with both of those requirements by placing the training-required modules after the weight-heavy backbone.

**Quality :** The authors have tested their repurposing method extensively, providing comparisons with 11 different models on 5 different datasets.

**Clarity :** The provided figures are clear and convey information in a very efficient and readable way.

**Significance :** The presented technique has the potential to greatly reduce the cost (time- and data-wise) of training a model for a new dataset. This is significant, as medical institutions may lack computational resources, large enough datasets, and/or the time to train a new model from scratch.
summary: In this paper, the authors present the FORMED architecture, which allows:
- The repurposing of a large time series foundational model from a forecasting task to a classification task;
- The adaptation of that repurposed model on new datasets, with potentially different numbers of channels and target classes.

This is achieved through the use of trainable channel and label embeddings, while the backbone remains frozen. A combination of a Transformer decoder layer and a residual network is also employed to form the desired output: it is trained during repurposing but frozen during adaptation.

FORMED is thus a lightweight model: training it after repurposing only requires training of the channel and task embeddings.

The authors use a pretrained TimesFM backbone and demonstrate the high performance of their repurposing approach on five medical datasets, and of their adapting approach on fractions of a sixth unseen medical dataset.
weaknesses: This paper suffers from one critical weakness: as the performance of FORMED is in most cases very close or below that of a task-specific model, any motivation behind its use lands entirely on its alleged lightness. Unfortunately, the authors do not provide any quantitative results demonstrating the time and computational efficiencies of their method, especially when compared to task-specific and task-adapted models; they merely provide arbitrary parameter counts. 

Moreover, the most common scenario for medical institutions is supposed to be adaptation of a repurposed model, as it requires the least amount of data and computational resources. Yet, the proof provided by the authors that adaptation works is very limited (fractions of a single dataset, only two competing models, and again no efficiency results).

Finally, the claims at the end of section 4.3 are scarcely substantiated: for example, the "domain knowledge" gained by the SDA during repurposing is never demonstrated.

In the current state of the paper, it is impossible to assess how significant the authors' contribution is.

Additionally:
- At line 284, the authors state that they are "processing each channel of input individually", which means that before the SDA block and thus within the backbone model where most of the weights are, channels are processed independently. This has been repeatedly reported to hinder performance on EEG and ECG data, and is even mentioned as a drawback of other models by the authors at lines 120-127.
- The bold highlight in Table 1 is unnecessary as it only denotes the best of two models. One highlighting method would be enough.
- The paper should be checked carefully for spelling and grammar errors (see below).

___

Miscellaneous non-exhaustive paper improvement remarks:
- Line 47, "using a pre-trained and fixed backbone foundation models", 'models' should be singular.
- Line 123/124, "in agree with Tan et al. (2024)" should read "as mentioned by Tan et al." or any correct equivalent wording.
- Line 127, "time series data and trained on multiple": a verb such as 'was' is missing before 'trained'.
- Line 131, "not suitable" can be replaced with "unsuitable".
- Line 142, "able to handle new domain of data": 'a' is missing between 'handle' and 'new'.
- Line 143/144, "on its dark side" should be more formal: 'However' is a valid alternative in this context for example.
- Line 147/148, "modification" should most likely be plural; "specific to certain task" is missing an 'a' between 'to' and 'certain'.
- Line 185/186, "The backbone foundation model is frozen in pre-training while trainable in repurposing and adapting": shouldn't this be the opposite?
- Line 284, a 'the' should be added between 'of' and 'input'.
- Line 295/296, There should be an 's' at the end of 'input', a 'a' before 'dynamic number of output classes', a 'a' before 'new dataset' and a 'the' before 'risk of overfitting'.
- Line 320, "all the parameters in SDA is independent on" should read "all the parameters in SDA are independent of".
- Line 326, "the weights in SDA is randomly", 'is' should be 'are'.
- Line 490, "despite the such inter-dataset", either 'the' or 'such' should be removed.
- Line 505/506, "on field of MedTS", a 'the' should be added between 'on' and 'field'.
- Line 537, "deserves", no final 's' is needed here.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Given that several datasets used are quite similar, have you considered using datasets from distinctly different tasks to better evaluate FORMED’s adaptability?

2. Why did you choose primarily Transformer-based TSM models as baselines, and would you consider comparing FORMED to another architecture such ResNet-based models?

3. Have you conducted more experiments?  how FORMED performs when trained on a single dataset alone, and if not, would you consider this for a better understanding of dataset influence?

4. A minor typo: The terms "label queries" (LQs) and "task query" are used inconsistently in the paper, creating some confusion. Ensuring consistency in terminology would improve clarity.
soundness: 3
strengths: This is a well-written and well-structured paper that proposes a robust foundation model, FORMED, for Medical Time Series (MedTS) classification. FORMED introduces a new approach to repurposing a foundation model for MedTS, tackling challenges of dataset heterogeneity and adaptability in a novel way. By freezing the backbone model and only adapting task-specific layers, it achieves high computational efficiency, allowing it to be fine-tuned on new datasets with minimal data. The model is thoroughly tested on five MedTS datasets and compared to multiple baselines across key metrics,  demonstrating its robustness and consistency across diverse domains.  Overall, this paper offers a meaningful contribution, establishing a strong foundation for future work in MedTS classification.
summary: This paper introduces FORMED, a foundation model designed for medical time series (MedTS) classification. FORMED repurposes a pre-trained backbone model, originally created for general time series forecasting, to address key challenges in MedTS, including dataset heterogeneity and limited data availability. By leveraging a generalizable adaptation layer, FORMED adapts effectively to various MedTS datasets, handling differences in channels, sample lengths, and diagnostic tasks. The model demonstrates strong generalization across five datasets, and proves versatile in adapting to new datasets with minimal parameter updates.
weaknesses: 1. Although five datasets are used to test generalizability, some datasets are quite similar, such as PTB and PTB-XL or datasets with Alzheimer’s data differing only by channel configurations. Testing on a wider range of datasets with different tasks could better showcase FORMED’s adaptability and robustness.

2. Most baseline models are Transformer-based; including comparisons with ResNet-based models, especially those tailored for MedTS tasks (like 12-lead ECG classifiers achieving strong results on PTB-XL), could provide a more comprehensive evaluation of FORMED’s performance.

3. The paper could benefit from additional experiments to deepen the analysis. For example, evaluating how FORMED performs when trained on a single dataset rather than a diverse set would provide insight into how training on multiple datasets impacts generalization. Such experiments could reveal how the model leverages knowledge across datasets and whether it shows improved performance or robustness compared to training on a single dataset alone.

4. Although Medformer is mentioned in the paper, it is not included as a baseline model. Adding Medformer to the baseline comparisons would provide a more comprehensive evaluation and better highlight FORMED's performance relative to existing MedTS models.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Seen in the weakness.   
My primary questions are about the unclarified novelty and lack of experiment
soundness: 2
strengths: 1. Generalizability: The particular "repurposing" enables the FORMED to seamlessly different data characteristics and tasks, which is important  for the MedTS  
2. Efficiency: Fine-tuning under the "repurposing" requires minimal parameter updates,  when adapting to new datasets.  
3. Performance: Experimental results show that FORMED outperforms or matches the specific-trained baseline models in MedTS classification tasks.
summary: This paper introduces FORMED, a foundation model repurposed for MedTS classification. FORMED takes "re-purposing" to integrate a pre-trained backbone model with the medical domain knowledge learned from a curated cohort of MedTS datasets . And "re-purposing" allows the FORMED to adapt to diverse datasets with varying channels, lengths, and tasks. The results show that  FORMED achieves competitive or superior performance, compared with 11 specifically trained baselines.
weaknesses: 1. Novelty: The technique novelty has not been clarified in this paper: (1) The keyword "repurposing" is not explained in the Abstract section, and is only merely mentioned with "a specialized shell enriched with medical knowledge. " in the Introduction section. However, How specialized the shell is, which makes it integrate domain knowledge from diverse datasets and diagnosis tasks, has not been emphasized. (2) "curated cohort of MedTS data" has been mentioned in many times in both the Abstract and Introduction. It seems that the collection of the curated cohorts is more important than the design in "repurposing".  

2. Experiment: More experiments are required to prove the effectiveness of the proposed method (1) The authors emphasize that "repurposing" enables minimal modification & lightweight parameter update for a specific task in the adaptation. However, no experiment results prove this point by comparing it with existing task-specific adaption methods.  Meanwhile, the extra "repurposing" phase may introduce more parameter updates, even if the update of the "adapting" phase can be lightweight. (2) This paper takes the TimesFM as the backbone. However, it lacks a comparison with the TSA-TimesFM. (3) Lack of ablation study to indicate the effectiveness of the delicate design in the “repurposing”.