Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification

Authors: yihe  wang
nan  huang
taida  li
yujun  yan
xiang  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:18:33

Abstract: Medical time series (MedTS) data, such as Electroencephalography (EEG) and Electrocardiography (ECG), play a crucial role in healthcare, such as diagnosing brain and heart diseases. Existing methods for MedTS classification primarily rely on handcrafted biomarkers extraction and CNN-based models, with limited exploration of transformer-based models. In this paper, we introduce Medformer, a multi-granularity patching transformer tailored specifically for MedTS classification. Our method incorporates three novel mechanisms to leverage the unique characteristics of MedTS: cross-channel patching to leverage inter-channel correlations, multi-granularity embedding for capturing features at different scales, and two-stage (intra- and inter-granularity) multi-granularity self-attention for learning features and correlations within and among granularities. We conduct extensive experiments on five public datasets under both subject-dependent and challenging subject-independent setups. Results demonstrate Medformer's superiority over 10 baselines, achieving top averaged ranking across five datasets on all six evaluation metrics. These findings underscore the significant impact of our method on healthcare applications, such as diagnosing Myocardial Infarction, Alzheimer's, and Parkinson's disease. We release the source code at https://github.com/DL4mHealth/Medformer.

Main paper: 
 introduction: 
 Medical time series refers to sequences of health-related data points recorded at successive times, tracking various physiological signals over time [1, 2] . Effective classification of MedTS data enables continuous monitoring and real-time analysis of a subject's physiological state, supporting early abnormality detection, accurate diagnosis, timely intervention, and personalized treatment-ultimately enhancing patient outcomes and healthcare efficiency [3, 4] . For instance, Electroencephalography (EEG) provides insights into a subject's neurological status [5, 6] , while Electrocardiography (ECG) aids in diagnosing heart conditions [7, 8, 9] . Most current MedTS classification approaches rely on handcrafted biomarker extraction [10, 11, 12] , convolutional neural networks (CNN)-based models [13, 14, 15, 16] , graph convolutional networks (GNNs)-based models [17, 18] , or combinations of CNNs and self-attention modules [19, 20] . Notably, effective transformer-based models for MedTS classification remain underexplored.Transformers have demonstrated strong performance in time series representation learning across tasks such as forecasting [21, 22, 23] , classification [24, 25] , and anomaly detection [26, 27] , with [30, 28, 29 ] employ a single crosschannel timestamp as a token; iTransformer [31] utilizes an entire channel as a token; and PatchTST and Crossformer [32, 33] adopt a patch of timestamps from one channel as a token. For MedTS classification, we propose Medformer considering inter-channel dependencies (cross-channel), temporal properties (multi-timestamp), and multifaceted scale of temporal patterns (multi-granularity).To bridge this gap, we propose Medformer * , a multi-granularity patching transformer designed explicitly for MedTS classification. Our approach introduces three mechanisms to enhance learning capacity. First, we propose a novel token embedding method using cross-channel patching, effectively capturing both multitimestamp and cross-channel features. To the best of our knowledge, this is the first application of cross-channel patching for transformer embedding in time series analysis. Second, rather than using fixed-length patches, we employ multigranularity patching with a list of patch lengths, enabling the model to capture features in different scales. This multigranularity approach could simulate different frequency bands, capturing bandspecific features without relying on handcrafted up/downsampling and band filters. Third, We introduce a two-stage (intra-and inter-granularity) self-attention mechanism to capture features within individual granularities and correlations across granularities, enabling complementary information integration across scales.We conduct extensive experiments using ten baselines across five public datasets, including three EEG datasets and two ECG datasets, focused on detecting diseases such as Alzheimer's and cardiovascular conditions under both subject-dependent and subject-independent setups (Figure 2 ). Results show that Medformer achieves the highest average ranking across all six evaluation metrics and five datasets (Figure 4 ), highlighting its superior effectiveness, stability, and potential for real-world applications.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The claims made in abstract and introduction are supported by the results in Section 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Medical Time Series. Medical time series refers to specialized time series data collected from the human body, commonly used for disease diagnosis [3, 7] , health monitoring [6, 1] , and brain-computer interfaces (BCIs) [39, 2] . Different MedTS modalities include EEG [40, 41, 42] , ECG [7, 8, 9] , EMG [43, 44] , and EOG [45, 46] , each offering distinct capabilities for various medical applications.For example, EEG and ECG data are instrumental in assessing brain and heart health [40, 7] . Recent BCI research explores using EEG to control objects, providing functional support to individuals with disabilities [2, 47] . Unlike general time series research, which predominantly focuses on forecasting tasks [48, 49] , MedTS research is centered around signal decoding, which involves classifying hidden information within MedTS sequences. Current approaches often rely on biomarker identification and deep-learning models utilizing CNNs, GNNs, or hybrid models combining CNNs with self-attention modules. For instance, band features such as relative band power and inter-band correlations [11, 50] have proven effective in EEG-based Alzheimer's disease diagnosis. Deep-learning models like EEGNet [14] , EEG-Conformer [20] , and TCN [51, 13] have also shown strong performance across various MedTS classification tasks.Table 1 : Existing methods do not fully utilize all potential characteristics in MedTS.Multi-TimestampAutoformer [28] ✓ Crossformer [33] ✓ ✓ FEDformer [52] ✓ Informer [29] ✓ iTransformer [31] ✓ ✓ MTST [53] ✓ ✓ Nonformer [54] ✓ PatchTST [32] ✓ Pathformer [55] ✓ ✓ Reformer [56] ✓ Transformer [30] ✓ Medformer(Ours)✓ ✓ ✓ ✓Transformers for Time Series. Transformer has demonstrated its strong learning and scaling-up ability in many domains, including natural language processing [30, 57] and computer vision [58, 59] . Existing transformerbased methods for time series analysis can be categorized into two main directions: modifying token embedding methods and selfattention mechanisms, or both. For example, PatchTST [32] uses a sequence of singlechannel timestamps as a patch for token embedding. Methods like Autoformer [28] , Informer [29] , Nonformer [54] , and FEDformer [52] develop new self-attention mechanisms or replace the self-attention module to improve learning ability and reduce complexity. Crossformer [33] and iTransformer [31] modify both token embedding methods and self-attention mechanisms. Patching. Patch embedding has been widely used in time series transformers since the proposal of PatchTST [32] . Existing methods of patching, such as Crossformer [33] , CARD [23] ,and MTST [53] , inherit from PatchTST [32] and utilize a sequence of single-channel timestamps for patching. This channel-independent patching might benefit learning ability in time series forecasting but may not be as effective in MedTS classification. Multi-Granularity. Existing methods such as Pyraformer [21] , MTST [53] , Pathformer [55] , and Scaleformer [60] , utilize multi-granularity embedding to capture features at different scales, allowing models to learn both fine-grained and coarse-grained patterns. We discuss the differences between our method and existing multi-granularity approaches in Appendix G.1.Medformer includes both novel token embedding and self-attention mechanisms. Figure 1 and Table 1 present a comparison of token embedding methods and feature utilization between our method and existing methods. The components of our method can be easily incorporated into existing methods to improve classification learning ability. For example, cross-channel multi-granularity patching can be integrated with methods that modify self-attention mechanisms, such as Autoformer [28] and Informer [29] , for token embedding. Similarly, the two-stage multi-granularity self-attention can be combined with existing multi-granularity methods, like MTST [53] and Pathformer [55] , to enhance the learning of inter-granularity features.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Appendix G.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries and problem formulation: 
 Disease Diagnosis with MedTS. Medical time series data typically exhibit multiple hierarchical levels, including subject, session, trial, and sample levels [13] . In disease diagnosis tasks using MedTS, each subject is usually assigned a single label, such as indicating the presence or absence of Alzheimer's disease. However, multi-labeling may be necessary when a subject has co-occurring conditions [61] . Notably, a subject's medical or physiological state remains relatively stable over time (or within short periods without significant change). For instance, a subject diagnosed with Alzheimer's disease is expected to retain that diagnosis for many years. If the subject also has Parkinson's disease, a multi-label learning approach is required, which essentially conducts classification tasks for each disease independently if they are not mutually exclusive.Since long sequences of time series data (e.g., trials or sessions) are often segmented into shorter samples for deep learning training, all samples from a single subject should ideally retain the same medical condition label. Thus, each MedTS sample typically includes a class label indicating a specific disease type and a subject ID indicating its original subject. Given the ultimate goal of diagnosing whether a subject has a particular disease, experimental setups must be carefully designed to reflect real-world clinical applications. Diverse setups can yield markedly different outcomes, potentially leading to misleading conclusions. Here, we introduce two widely used setups in MedTS classification and clarify their distinctions. Figure 2 provides a simple illustration of these two setups.Subject-Dependent. In this setup, the division into training, validation, and test sets is based on time series samples. All samples from various subjects are randomly shuffled and then allocated into the respective sets. Consequently, samples with identical subject IDs may be present in the training, validation, and test sets. This scenario potentially introduces "information leakage," wherein the model could inadvertently learn the distribution specific to certain subjects during the training phase. This setup is typically employed to assess whether a dataset exhibits cross-subject features and has limited applications under real-world MedTS-based disease diagnosis scenarios. The reason is simple: we cannot know the label of unseen subjects and their corresponding samples during training. Generally, the results of the subject-dependent setup tend to be notably higher than those from the subject-independent setup, often showing the upper limit of a dataset's learning capability.Figure 2 : Subject-dependent/independent setups (figure adopted from our previous work [13] ). In the subject-dependent setup, samples from the same subject can appear in both the training and test sets, causing information leakage. In a subject-independent setup, samples from the same subject are exclusively in either the training or test set, which is more challenging and practically meaningful but less studied.Subject-Independent. In this setup, the division into training, validation, and test sets is based on subjects. Each subject and their corresponding samples are exclusively distributed into one of the training, validation, or test sets. Consequently, samples with identical subject IDs can only be present in one of these sets. This setup holds significant importance in disease diagnosis tasks as it closely simulates real-world scenarios. It enables us to train a model on subjects with known labels and subsequently evaluate its performance on unseen subjects; in other words, evaluate if a subject has a specific disease. However, this setup poses significant challenges in MedTS classification tasks. Due to the variability in data distribution and the potential presence of unknown noise within each subject's data, capturing general task-related features across subjects becomes challenging [62, 13, 63, 64] . Even if subjects share the same label, the personal noise inherent in each subject's data may obscure these common features. Developing a method that effectively captures common features among subjects while disregarding individual noise remains an unsolved problem.In this work, we evaluate our model mainly in the subject-independent setup to better align with real-world applications, aiming to draw attention within the time series research community to the substantial challenges posed by this setup. While our model is not specifically tailored to address subject-independent problems, it integrates multi-timestamp, cross-channel, and multi-granularity features in MedTS, enhancing its capacity to capture subject-invariant representations. Consequently, our model is well-equipped to tackle the subject-independent challenge to a certain extent, and our results (Section 5) confirm such capability of Medformer.We next present the problem formulation for multivariate medical time series classification in the context of disease diagnosis.Problem (MedTS Classification). Consider an input MedTS sample x in ∈ R T ×C , where T denotes the number of timestamps and C represents the number of channels. Our objective is to learn an encoder that can generate a representation h, which can be used to predict the corresponding label y ∈ R K of the input sample. Here, K denotes the number of medically relevant classes, such as various disease types or different stages of one disease. gr . Additionally, a granularity-specific router u (i) is designed to capture integrated information for each respective granularity. We then perform intra-granularity self-attention, focusing on individual granularities, and inter-granularity self-attention, using the routers to facilitate communication across different granularities.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We include an anonymous link in the Abstract providing source codes with full implementation details for our methods and all baselines. All five datasets used for evaluation are publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 In this section, we first describe the cross-channel multi-granularity patching mechanism for learning spatial-temporal features in different scales. Next, we analyze the two-stage multi-granularity selfattention mechanism, which leverages features within the same granularity and correlations among different granularities. The architecture of the proposed Medformer is illustrated in Figure 3 .Cross-Channel Multi-Granularity Patch Embedding. From the medical perspective, the brain or heart functions as a cohesive unit, suggesting a naive assumption that there are inherent correlations among different channels in MedTS [35, 36, 37] , as each channel represents the activities of distinct brain or heart regions. Motivated by the above assumption, we reasonably propose multi-channel patching for token embedding, which is different from existing patch embedding methods that embed patches in a channel-independent manner and fail to capture inter-channel correlations [32, 33, 23] . Figure 1 provides an overview comparison of existing token embedding methods and ours. Additionally, existing research on EEG biomarker extraction has shown that certain features are linked to different frequency bands, such as α, β, and γ bands [10, 12] . This motivates us to embed patch tokens in a multi-granularity way. Instead of using traditional methods like up/downsampling or handcrafted band filtering, multi-granularity patching automatically corresponds to different sampling frequencies, which can simulate different frequency bands and capture band-related features.Given the above rationales, we propose a novel token embedding approach: cross-channel multigranularity patching. Given an input multivariate MedTS sample x in ∈ R T ×C , and a list of different patch lengths {L 1 , L 2 , . . . , L n }. For the i-th patch length L i denoting granularity i, we segment the input sample into N i cross-channel non-overlapping patches x(i) p ∈ R Ni×(Li•C) . Zero padding is applied to ensure that the number of timestamps T is divisible by L i , making N i = ⌈T /L i ⌉.The patches are mapped into latent embeddings space using a linear projection:x (i) e = x (i) p W (i) , where x (i) e ∈ R Ni×D and W (i) ∈ R (Li•C)×D .Inspired by the augmented views contrasting in the contrastive learning framework [65, 13, 66] , we further apply data augmentations such as masking and jittering on x e ∈ R Ni×D . We assume the augmentation can improve the learning ability in the following inter-granularity self-attention stage by forcing different granularities to learn and complement information from each other.A fixed positional embedding W pos ∈ R G×D is generated for positional encoding [30] , where G is a very large number. We add W pos [1 : N i ] ∈ R Ni×D , the first N i rows of the positional embedding W pos , and a learnable granularity embedding W (i) gr ∈ R 1×D , to obtain the final patch embedding for the i-th granularity with patch length L i :EQUATIONwhere x (i) ∈ R Ni×D . Note that the granularity embeddingW (i)gr aims to distinguish among granularities and is broadcasted to all N i embedding rows with D dimension during addition.To reduce time and space complexity, we initialize a router to be used in the multi-granularity self-attention (as described later) for each granularity:EQUATIONwhereEQUATIONgr ∈ R 1×D . Here, W pos [N i + 1] is not used for positional embedding but to inform the router about the number of patches with the current L i granularity, and W (i) gr contains the granularity information. Both components help distinguish the routers from one another.Finally, we obtain a list of final patch embeddings x (1) , x (2) , . . . , x (n) and router embeddings u (1) , u (2) , . . . , u (n) for different granularities with patch lengths {L 1 , L 2 , . . . , L n }. We feed the embeddings to the two-stage multi-granularity self-attention.Multi-Granularity Self-Attention. Our goal is to learn multi-granularity features and granularity interactions during self-attention. A naive approach to achieve this goal is to concatenate all the patch embeddings x (1) , x (2) , . . . , x (n) into a large patch embedding X ∈ R (N1+N2+...+Nn)×D and perform self-attention on this new embedding, where n denotes the number of different granularities. However, this results in a time complexity of O (n i=1 N i )2 , which is impractical for a large n.To reduce the time complexity, we propose a router mechanism and split the self-attention module into two stages: a) intra-granularity self-attention and b) inter-granularity self-attention. The intragranularity stage performs self-attention within the same granularity to capture the distinctive features of each granularity. The inter-granularity stage performs self-attention across different granularities to capture their correlations.a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention a) Intra-Granularity Self-Attention. For the i-th patch length L i denoting granularity i, we vertically concatenate the patch embedding x (i) ∈ R Ni×D and router embedding u (i) ∈ R 1×D to form an intermediate sequence of embeddings z (i) ∈ R (Ni+1)×D :EQUATIONwhere [•∥•] denotes concatenation. We perform self-attention on the new z (i) for both the patch embedding x (i) and the router embedding u (i) :EQUATIONwhere Attn (Q, K, V ) denotes the scaled dot-product self-attention mechanism in [30] . Note that the router embedding u (i) is updated concurrently with the patch embedding x (i) to maintain consistency, ensuring that the router effectively summarizes each granularity's features in the current training step and is ready for the subsequent inter-granularity self-attention. The intra-granularity self-attention mechanism allows the model to capture temporal features within a single granularity, facilitating the extraction of local features and correlations among timestamps at the same scale. Inter-Granularity Self-Attention. We concatenate all router embeddings u (1) , u (2) , . . . , u (n) to form a sequence of routers U ∈ R n×D :EQUATION)where n is the number of different granularities. For granularity i with patch length L i , we apply self-attention to the router embedding u (i) ∈ R 1×D with all the routers U :EQUATIONEach router contains global information specific to one granularity by doing intra-granularity selfattention. By performing self-attention among routers, information can be exchanged and learned across different granularities, effectively capturing features across various scales. Additionally, the use of the router mechanism successfully reduces the time complexity of the naive approach from O (n i=1 N i ) 2 to O n i=1 N 2 i + n 2 .Given that N i ≤ T , the worst-case time complexity for our self-attention mechanism is O nT 2 + n 2 . However, a reasonable choice of patch lengths as a power series, i.e., L i = 2 i , leads to a time complexity of O(T 2 ). To further reduce complexity and memory consumption, we apply shared layer normalization and feed-forward layers across all granularities. See appendix F for more details about complexity analysis. Summary. Our method utilizes the standard transformer architecture shown in Figure 3 . For given sample x in , after M layers of self-attention learning, we obtain a list of updated patch embeddings x (1) , x (2) , . . . , x (n) , which we concatenate them to form a final representation h that can be used to predict label y ∈ R K in a downstream classification task. Note that although we discuss multigranularity here, our method is flexible and can be easily adapted to variants such as single-granularity or even repetitive same granularities. See Appendix D.2 for more details. 
experiments: 
 We compare our Medformer with 10 baselines across 5 datasets, including 3 EEG datasets and 2 ECG datasets. Our method is evaluated under two setups (Section 3): subject-dependent and subject-independent. In the subject-dependent setup, training, validation, and test sets are split based on samples, while in the subject-independent setup, they are split based on subjects. is labeled with a binary indicator of Myocardial Infarction. ( 5) PTB-XL [71] is an ECG dataset with a five-class label for each sample, representing various heart conditions. Table 2 provides information on the processed datasets. For additional details on data characteristics, train-validation-test splits under different setups, and data preprocessing, please see Appendix B.Baselines. We compare with 10 state-of-the-art time series transformer methods: Autoformer [28] , Crossformer [33] , FEDformer [52] , Informer [29] , iTransformer [31] , MTST [53] , Nonformer [54] , PatchTST [32] , Reformer [56] , and vanilla Transformer [30] .Implementation. We employ six evaluation metrics: accuracy, precision (macro-averaged), recall (macro-averaged), F1 score (macro-averaged), AUROC (macro-averaged), and AUPRC (macroaveraged). The training process is conducted with five random seeds (41) (42) (43) (44) (45) on fixed training, validation, and test sets to compute the mean and standard deviation of the models. All experiments are run on an NVIDIA RTX 4090 GPU and a server with 4 RTX A5000 GPUs.For data augmentation methods, we provide six widely used methods in time series augmentation [72, 66, 73, 61] . For more details about these six methods, see Appendix A. For the parameter tuning in our method and all baselines, we employ 6 layers for the encoder, set the dimension D to 128, and the hidden dimension of feed-forward networks to 256. We utilize the Adam optimizer with a learning rate of 1e-4. The batch size is set to {32,32,128,128,128} for datasets APAVA, TDBrain, ADFTD, PTB, and PTB-XL, respectively. The training epoch is set to 100, with early stopping triggered after 10 epochs without improvement in the F1 score on the validation set. We save the model with the best F1 score on the validation set and evaluate it on the test set. See Appendix C for any additional implementation details of our method and all baselines.Setup. In this setup, the training, validation, and test sets are split based on samples. All samples from all subjects are randomly shuffled and distributed into the training, validation, and test sets according to a predetermined ratio, allowing samples from the same subject to appear in three sets simultaneously. As discussed in the Preliminaries section 3, this setup has limited applicability for MedTS-based disease diagnosis in real-world scenarios. It is usually used to evaluate whether the dataset exhibits cross-subject features quickly. The results obtained from this setup are typically much higher than those from the subject-independent setup, showing a dataset's upper limit of learnability.Results. We evaluate the EEG dataset ADFTD using this setup to provide a direct comparison of results with the subject-independent setup. The results are presented in Table 3 . Our method outperforms all the baselines, achieving the top-1 results in all six evaluations, with an impressive F1 score of 97.50%. Notably, baseline methods like Informer, Nonformer, Reformer, and Transformer also demonstrate strong performance, achieving F1 scores exceeding 90%. The overall results indicate the presence of discernible and learnable features related to Alzheimer's Disease within this dataset.Setup. In this setup, the training, validation, and test sets are split based on subjects. All subjects and their corresponding samples are distributed into the training, validation, and test sets according to a predetermined ratio or subject IDs. Samples from the same subjects should exclusively appear in one of these three sets. This setup simulates real-world MedTS-based disease diagnosis, aiming to train a model on subjects with known labels and then test it on unseen subjects to determine if they have a specific disease. The challenges associated with this setup are discussed in section 3. All five datasets are evaluated using this setup. The heatmap table shows the average rank of Medformer and 10 baselines across 5 datasets using the subjectindependent setup. A lower number indicates better results. The average rank is calculated across the 5 datasets to obtain the overall average rank.Results. Table 4 presents the results of the subject-independent setup. Our method achieves the top-1 F1 scores on 4 out of 5 datasets. Overall, our method achieves 15 top-1 and 30 top-3 rankings out of 30 evaluations conducted across 5 datasets and 10 baselines, considering 6 different metrics. Figure 4 provides an overview heatmap table of average rank across 5 datasets on 6 metrics for all methods. Lower rank numbers indicate better results, with rank 1 representing the best performance among all methods. Our method demonstrates the best average rank among all methods across the 6 metrics. Additionally, it is notable that the result for ADFTD is a 50.65% F1 score under the subject-independent setup, which is significantly lower than the 97.50% F1 score achieved under the subject-dependent setup. This comparison highlights the challenge of the subject-independent setup, which better simulates real-world scenarios.Ablation Study. 1) Module Study: We conduct a module study to evaluate the effectiveness of each proposed mechanism in our method (Appendix D.1). 2) Patch Length Study: We perform parameter tuning on the list of patch lengths to evaluate the effectiveness of multi-granularities (Appendix D.2).Additional Experiments. We also perform experiments on two human activities recognition datasets [74, 75] to demonstrate the learning ability of our model on general time series with potential channel correlations (Appendix E). 
conclusion and limitations: 
 Conclusion This paper presents Medformer, a multi-granularity patching transformer tailored for MedTS classification. We introduce three novel mechanisms that leverage the distinctive features of MedTS, including channel correlations and frequency band biomarkers. These mechanisms include cross-channel patching to capture multi-timestamp and cross-channel features, multi-granularity embedding to learn features at various scales, and a two-stage multi-granularity self-attention mechanism to extract features both within and across granularities. Experimental results across five datasets, evaluated against ten baselines under the subject-independent setup, demonstrate the effectiveness and robustness of our approach, underscoring its potential for real-world applications.Limitations and Future Work The design of Medformer allows for inputting various patch lengths, offering both opportunities and challenges. While variable patch lengths provide flexibility and have been shown to outperform uniform lengths in some cases (see Appendix D.2 and Appendix C), not all patch length configurations yield optimal results. Some combinations may perform worse than uniform patch lengths, necessitating careful tuning of patch lengths as hyperparameters. Future work could explore mechanisms for automatically selecting the most effective patch lengths and optimizing the model's ability to capture relevant granularities. Additionally, while our method outperforms baselines and performs well on certain datasets under the subject-independent setup, it does not include mechanisms specifically designed for this setup. Developing modules that decompose subject-specific features from task-related features could further enhance learning under the subject-independent setup, presenting an intriguing direction for future research.For all methods, we employ 6 layers for the encoder, with the self-attention dimension D set to 128 and the hidden dimension of the feed-forward networks set to 256. The optimizer used is Adam, with a learning rate of 1e-4. The batch size is set to {32,32,128,128,128} for the datasets APAVA, TDBrain, ADFD, PTB, and PTB-XL, respectively. Training is conducted for 100 epochs, with early stopping triggered after 10 epochs without improvement in the F1 score on the validation set. We save the model with the best F1 score on the validation set and evaluate it on the test set. We employ six evaluation metrics: accuracy, precision (macro-averaged), recall (macro-averaged), F1 score (macroaveraged), AUROC (macro-averaged), and AUPRC (macro-averaged). Both subject-dependent and subject-independent setups are implemented for different datasets. Each experiment is run with 5 random seeds (41) (42) (43) (44) (45) 4, 8, 8, 16, 16, 16, 16, 32, 32, 32, 32, 32, 32, 32 , 32}, respectively. The data augmentations are randomly chosen from a list of four possible options: none, jitter, scale, and mask. The number following each augmentation method indicates the degree of augmentation. Detailed descriptions of these methods can be found in Appendix A. The augmentation methods used for the datasets APAVA, TDBrain, ADFD, PTB, and PTB-XL are {none, drop0.35}, {none, drop0.25}, {drop0.5}, {drop0.5}, and {jitter0.2, scale0.2, drop0.5}, respectively.Autoformer Autoformer [28] employs an auto-correlation mechanism to replace self-attention for time series forecasting. Additionally, they use a time series decomposition block to separate the time series into trend-cyclical and seasonal components for improved learning. The raw source code is available at https://github.com/thuml/Autoformer.Crossformer Crossformer [33] designs a single-channel patching approach for token embedding. They utilize two-stage self-attention to leverage both temporal features and channel correlations. A router mechanism is proposed to reduce time and space complexity during the cross-dimension stage. The raw code is available at https://github.com/Thinklab-SJTU/Crossformer.FEDformer FEDformer [52] leverages frequency domain information using the Fourier transform. They introduce frequency-enhanced blocks and frequency-enhanced attention, which are computed in the frequency domain. A novel time series decomposition method replaces the layer norm module in the transformer architecture to improve learning. The raw code is available at https: //github.com/MAZiqing/FEDformer.Informer Informer [29] is the first paper to employ a one-forward procedure instead of an autoregressive method in time series forecasting tasks. They introduce ProbSparse self-attention to reduce complexity and memory usage. The raw code is available at https://github.com/zhouhaoyi/Informer2020.iTransformer iTransformer [31] questions the conventional approach of embedding attention tokens in time series forecasting tasks and proposes an inverted approach by embedding the whole series of channels into a token. They also invert the dimension of other transformer modules, such as the layer norm and feed-forward networks. The raw code is available at https://github.com/thuml/iTransformer.MTST MTST [53] uses the same token embedding method as Crossformer and PatchTST. It highlights the importance of different patching lengths in forecasting tasks and designs a method that can take different sizes of patch tokens as input simultaneously. The raw code is available at https://github.com/networkslab/MTST.Nonformer Nonformer [54] analyzes the impact of non-stationarity in time series forecasting tasks and its significant effect on results. They design a de-stationary attention module and incorporate normalization and denormalization steps before and after training to alleviate the over-stationarization problem. The raw code is available at https://github.com/thuml/Nonstationary_Transformers. i . One potentially useful patch list is the power series {2 1 , 2 2 , . . . 2 n }, where 2 n < T . In this case, the complexity of intra-granularity attention reduces as follows:O n i=1 N 2 i = O n i=1 T 2 i 2 ≤ O n i=1 T 2 i + 1 2 = O n i=1 T 2 2 2i + 2 T 2 i + 1 = O T 2 n i=1 1 2 2i + 2T n i=1 1 2 i + n ≤ O 1 3 T 2 + 2T + log T = O(T 2 )The complexity of inter-granularity attention is O(n 2 ) ≤ O(log 2 T ). Therefore, the total time complexity of the two-stage multi-granularity self-attention module is O(T 2 ), which is the same complexity as the vanilla transformer. This analysis demonstrates our model's ability to incorporate different granularities without significantly increasing computational overhead.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 5 and Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We run experiments over five random seeds and report the average value with the standard deviation. See Table 2-6.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix H.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not pose a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We properly cite baseline models and the code library used for implementation. See Appendix C.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not publish new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not include crowdsourcing experiments nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not include crowdsourcing experiments nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The paper presents a new transformer model for medical time series. The reviewers are unanimously positive about the paper, appreciating the combination of ideas that went into the models. Questions were raised about comparisons to state of the art models, which the authors successfully addressed during the rebuttal phase. The authors also added several datasets to demonstrate the generalizability of the method, and also explained how their method could be used to derive explanations. Overall, this is a strong paper and ready to be presented.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduced a multi-granularity patching transformer tailored specifically for medical time series classification. To leverage the characteristics of medical time series data, the model incorporated three unique features : cross-channel patching to leverage inter-channel correlations, multi-granularity embedding for capturing features at different scales, and two-stage (intra- and inter-granularity) multi-granularity self-attention for learning features and correlations within and among granularities. The model was well tested with several datasets and showed mixed results against other models.
strength_weakness: strength_weakness: Because this model has been evaluated on very specific time series data, EEG and ECG, it is unclear whether this approach can be extended to many other time series data outside the medical domain.
questions: questions: As mentioned in Appendix G, what domains or data sets should be tested next to test the ability of this approach?
limitations: limitations: Currently the proposed model was tested only in EEG and ECG domains, thus the scope is too narrow for the general audience of the NeurIPS.
paper_summary: paper_summary: This paper proposes a multi-granularity patching Transformer for medical time series classification. The proposed model, Medformer leverages cross channel patching to learn inter-channel correlations. It utilizes multi-granularity embedding for learning temporal patterns. Finally, it takes 2 stage self-attention to aggregate information from multiple granularities. The experiments on 5 datasets show the effectiveness of Medformer.
strength_weakness: strength_weakness: 1.The novelty of this paper is not enough. The idea of leveraging multi-granularity and multi-channel features has been studied in time series field.

2.Time series classification has also been well studied. The paper should clarify and focus the specific challenges for medical time series classification compared to general time series.

3.For general time series classification, several new SOTA methods have been released, such as Time-LLM, ModernTCN, and Lag-llama, etc. Such recent studies should be compared in the experiments.

4.The results of Medformer are not superior in several metrics.
questions: questions: 1.How about the performance of the medformer when it is applied to general time series classification? This experiments could demonstrate the advantages of the model.

2.What are the new challenges of time series classification in medical datasets? How does Medformer resolve it?
limitations: limitations: Yes.
paper_summary: paper_summary: The authors have created a model that seems to perform especially well for high-frequency waveform classification. They have benchmarked their model over several datasets, and its overall rank is higher than that of other models.
strength_weakness: strength_weakness: The limitations and discussion should at least be briefly discussed in the paper itself, not just in the appendix. The compared models, while plentiful, seem only transformer-based; it would be interesting to see a comparison to different architectures, especially since TCN is also mentioned in the related work (e.g., try the Mamba model https://arxiv.org/pdf/2312.00752). The improvements over existing transformer models seem marginal. 
The efficiency of the models should be benchmarked (e.g., training time). 
Table 3: some results are tied as they lie in the range of the standard deviation of the best model. It would be good to highlight these results as well.
Line 724: Comparison misspelled.
questions: questions: The authors use two types of vital signs to benchmark their model. Is there enough reason that we could assume the superior performance also translates to other vital signs and, more broadly, medical time-series prediction tasks?
limitations: limitations: Tasks could be extended to (cheap to collect) PPG signals. Additionally, one could think of a multi-model setup as we often have more (static or lower frequency time-series) data on specific test subjects.
paper_summary: paper_summary: This paper introduces Medformer, a transformer variant designed to learn complex dynamics from medical signals. This is achieved in three key ways : 
1. Cross-channel patching for token embedding;
2. Multi-length patching for coarse and fine feature processing;
3. Multi-granularity self-attention for information granularities integration;

The proposed model is then extensively validated through comparison to ten baselines on five datasets.
strength_weakness: strength_weakness: This paper suffers from one main weakness : processing medical time series without providing explainability studies undermines its usability and transparency. This could be addressed in the future by adapting one of the techniques developed to interpret predictions made by transformer models. 

The provided code could also benefit from more comments, for example in the `exp/*` files.
questions: questions: - For clarification, does the random seed only affect the initialization of the model? Are the training/validation/test splits the same for all five runs?
- What was your reasoning behind the different augmentation techniques used depending on the dataset?
limitations: limitations: The authors have addressed the technical limitations of their work, albeit not in the main body of the article. However, they have not addressed the societal impact of their model, were it to be adopted in a healthcare setting : steps should be taken to either prevent or disclose clearly whether demographic biases (for example) found in the data are also found in the model's predictions.

Comments: title: Thanks for the rebuttal
Please respond to the rebuttal of NeurIPS Submission7556! The response period ends in one day!
Main paper revision
comment: Thank you for your reply and endorsement!
Thanks for the rebuttal. The authors clarify the ideas and motivations of MedFormer and add more baselines. I have increased my score.
Reviewer vj5L,

Thank you for your service in reviewing for NeurIPS!
Your work is not done, however, as part of a reviewer's responsibility is to engage with the authors during the discussion period.

The authors of Submission 7556 (Medformer) have provided an extensive response to your review.
Please read it and indicate the extend to which it addresses your concerns. Please indicate clearly which of the issues you raised are addressed and which are not, such that the authors have a final chance to reply.

There is only one day left in the discussion period, so please do this ASAP!


The AC for Submission 7556
Dear reviewer vj5L:

We would like to kindly remind you that the author-reviewer discussion will end in 1 day. Could you take a look at our rebuttal and let us know if our rebuttal has addressed your concerns? As our paper is currently on the borderline, your input will be greatly appreciated.

Best regards
Thank you so much for your endorsement! We will integrate everything into the final version of paper.
I would like to thank the authors for their rebuttal. The additional experiments clearly addressed my concerns and strongly demonstrated the advantage of the proposed method. Thus I updated my score. I look forward to reading the integrated final version.
Dear Authors,

Thank you for your answer. This has apparently changed this year. You have addressed my points. However, I think the current score is appropriate.
Dear Reviewer,

Thank you for your feedback on our additional material. While we would be happy to provide a revised version of the entire paper, according to the rebuttal rules of Neurips 2024, we are not allowed to upload the revision of our paper until the camera-ready stage. We are only allowed to upload one page of PDF in the global response, including tables and figures.

For your convenience, we quote some FAQs from **NeurIPS 2024 FAQ for Authors** in the **Reviewing/Discussion process**:

* **Can we upload a revision of our paper during the rebuttal/discussion period?** No revisions are allowed until the camera-ready stage.

* **Can we upload a revision of the supplementary materials during the rebuttal/discussion period?** No. You may revise it for the camera-ready stage.

* **What is the length limit in the rebuttal phase?** You can submit a rebuttal of up to 6000 characters per review, and one global rebuttal of up to 6000 reviews. These are posted by clicking the "Rebuttal" and "Author Rebuttal" buttons. You can additionally add a one-page PDF with Figures and tables. You can upload this PDF after you click the "Author Rebuttal" button.

* **Where can I see my submitted PDF for the author-rebuttal?** There is a link to the PDF at the end of the global rebuttal.

Could you use our current rebuttal for your assessment? We are happy to answer your additional questions. Thank you again!
Dear Authors,

Thank you for addressing my concerns with additional material. However, a revised version of the paper must be provided to fully address the points in my and the other authors' reviews. Thank you.
Thank you again for your endorsement! We will include the explainable results and new experiments in our final paper.
Thank you for your response. I appreciate your demonstration that Medformer can be explainable, and the clarifications about augmentation strategies and demographic information. I hope you will find a way to include this and your additional benchmarks in your final paper.

My rating remains unchanged and positive (Accept); I do not think that a method being primarily applicable to clinical data and not general time series is grounds for rejection, since the "Machine learning for healthcare" area of NeurIPS exists for this purpose.