Frequency-Enhanced Hybrid Multimodal CNN-Transformer Network for Electrocardiogram Classification

Authors: yufeng  wei
cheng  lian
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-06-29 11:13:27

Abstract: Recently, deep learning-based models have been widely adopted for electrocardiogram (ECG) classification tasks, demonstrating greater accuracy and efficiency than manual diagnosis. Most existing methods use raw ECG or its time-frequency domain representation as input. These methods are constrained by their reliance on a single input modality, thereby limiting the network's ability to capture discriminative information effectively. In our study, we treat the frequency spectrum of ECG as an independent modality and input it into a multimodal classification model along with ECG. Our method combines depthwise separable convolution and Transformer architectures for unimodal feature extraction. A linear layer aligns the features from both modalities, and a Transformer layer facilitates multimodal feature fusion. We evaluate the performance of our model in both the multi-label classification task using the Ningbo dataset and the multi-class classification task using the Real World dataset. Our model demonstrates superior classification performance compared to the competitive baseline models.

Main paper: 
 i. introduction: 
 Cardiovascular disease stands as one of the primary causes of mortality worldwide [1] . Early detection of cardiovascular disease is crucial to initiate effective treatment. ECG reflects the electrical activity of heart and is widely used to monitor heart health. Initially, the analysis of ECG relied on human experts. However, this manual diagnosis proves not only timeintensive and laborious but also susceptible to subjectivity and variability in professionalism [2] . To address the limitations inherent in manual diagnosis, several studies have explored the application of traditional machine learning techniques, such as support vector machines (SVM) and random forests (RF), in classifying manually extracted features from ECG [3, 4] . While these methods offer good interpretability, they fail to accurately capture the diverse variations in ECG across different environments and populations [5] .In recent years, deep learning models have been widely used in ECG classification tasks. These models leverage data-driven techniques to dynamically extract discriminative features, resulting in excellent classification performance. For instance, Xu et al. [6] introduce a model based on a coupled convolutional layer structure for heartbeat classification, achieving an overall accuracy of 99.43%. Similarly, Sun et al. [7] propose a recurrent neural network (RNN) comprising stacked long short-term memory (LSTM) units for atrial fibrillation (AF) prediction, yielding a F1 score of 92%. Moreover, in [8] , an 8layer convolutional neural network (CNN) with shortcut connections is combined with a 1-layer long short-term memory (LSTM) network to enhance the model's capacity to capture long-term dependencies. However, these methods focus on leveraging time domain information and often overlook the wealth of knowledge embedded in the frequency domain of ECG.Prior research has underscored the efficacy of incorporating frequency domain information into ECG classification tasks [9] [10] [11] . These studies either utilize time-frequency spectrogram as input or employ encoders to represent information in both time and frequency domains. While the former method remains confined to modeling a single modality, the latter method, by separately modeling the frequency domain, offers a more comprehensive exploration of valuable information inherent in frequency domain.In this paper, we treat the frequency spectrum of ECG as an independent modality and propose a multimodal feature extraction and fusion model for ECG classification. Specifically, the frequency spectrum is generated via fast Fourier transform (FFT) and subsequently fed into an encoder separately from the ECG. We devise a CNN layer mainly comprising depthwise separable convolution to initially extract unimodal features, followed by the utilization of a Transformer layer to capture long-term dependencies. To facilitate multimodal fusion more effectively, a linear layer is employed to align the features of the two modalities. Subsequently, another Transformer layer is utilized for the final feature fusion. Our main contributions can be summarized as follows:• Proposal of a multimodal model encompassing feature extraction, alignment, and fusion processes tailored for ECG classification. The subsequent sections of this paper are organized as follows: Section II describes the related work concerning deep learning models for ECG classification. Section III describes the proposed method in detail. Section IV describes the experimental design and presents the analysis of the experimental results. Section V describes the conclusions drawn from this study and outlines potential directions for future research.Most ECG classification models based on deep learning leverage CNNs due to their robust feature extraction capabilities. For example, Niu et al. [12] develop a multi-view convolutional neural network (MPCNN) that employs symbolic representations of heartbeats to automatically learn features and classify heartbeats. Additionally, Wang [13] propose an automatic AF detection method using an 11-layer neural network, primarily composed of a CNN and an improved Elman neural network (MENN). Some researchers also explore the use of LSTM or bidirectional LSTM for ECG classification, which mainly considers the temporal properties of ECG [14, 15] . To construct more powerful models, LSTM and attention mechanism are introduced and combined with CNN, further enhancing classification performance [8, 16] . Furthermore, the vision transformer, a popular model in the field of computer vision, is introduced into ECG classification tasks, typically involving the transformation of ECG into a two-dimensional image [10] .This section provides a detailed description of our method. As illustrated in Fig. 1 (a), frequency spectrum is generated from ECG via the FFT and then fed into our model along with ECG. The extraction of multimodal features involves two stages: unimodal feature extraction and multimodal feature fusion. In the first stage, a carefully designed CNN layer and a Transformer layer are employed for unimodal feature extraction. In the second stage, a linear layer is used for feature alignment, and a Transformer layer facilitates multimodal feature fusion. The final multimodal features are then subjected to global average pooling before being fed into a linear layer for classification. The specifics of the network architecture are detailed in Table I , which represents the Base version. We also provide a Small version of our model, in which the number of CNN blocks is reduced to 2, the number of downsampling layers is reduced to 1, and the dimension of feature channels is halved. The following sections will introduce the different components in detail.Given an ECG s t ∈ R N ×L , where N is the number of leads and L is the length of each lead, the frequency spectrum s f ∈ R N ×L is obtained using the FFT. The transformation formula for i th lead is as follows:s f i (k) = ABS(FFT[s t i ]) = ABS( L-1 l=0 s t i (l)) • e -j 2π L lk , l = 0, 1, . . . , L -1 (1)where k represents the different frequencies in ECG, and ABS denotes the operation of taking the absolute value, which ensures that only the amplitude of frequency spectrum is retained. The transformed frequency spectrum has the same length as the corresponding ECG but does not belong to time series. Instead, it reflects the essential characteristics of ECG from a different perspective.This part of the network comprises three components: the Value Embedding layer, the CNN layer, and the Transformer layer.1) The Value Embedding layer: This layer consists of a 1D convolutional layer followed by a batch normalization (BN) layer to adjust the channel dimensions and initially capture the local relationships within the data. The encoding process for modality m is described by the following formula:EQUATIONwhere m represents ECG or frequency spectrum.2) The CNN layer: Our design of the CNN layer refers to ConvNeXt [17] . It comprises stacked CNN blocks with downsampling layers interspersed between different blocks. The structure of a single CNN block is depicted in Fig. 1 (b) . Depthwise convolution (DC) is employed to model the feature of each channel independently, followed by point convolution (PC) to capture inter-channel interactions. The PC operation is structured to expand and then contract the channel dimension, utilizing two different point convolutions. A BN layer or a GELU activation function is inserted between different convolution layers, and a residual connection is applied to the final output. The downsampling layer consists of a BN layer and a 1D convolutional layer, which doubles the channel dimension of the features while reducing its length by half. The output of the b th CNN Block can be described by the following formula:EQUATIONwhere C represents the channel dimension of the features.3) The Transformer layer: To capture long-term dependencies within the features, we utilize a Transformer block as depicted in Fig. 1 (c ). The Transformer block consists of a multi-head self-attention layer (MSA) and a multi-layer perceptron (MLP) layer. Layer normalization (LN) layers and residual connections are applied before and after each component. The output of the CNN layer is represented as x m B , and the output of the Transformer layer can be described by the following formula:EQUATIONwhere x m T represents the output of the Transformer layer.After obtaining the features from the ECG and frequency spectrum x t T and x f T , we use a linear layer to align the features of the two modalities. This method has been effectively applied to image-text alignment [18, 19] . Specifically, the linear layer is applied only to the features of the frequency spectrum to align it with the features of the ECG. Subsequently, the features from both modalities are concatenated and fed into a Transformer block for final fusion. The process is described by the following formula:EQUATIONEQUATIONwhere x out represents the output of the Transformer layer, [; ] represents concatenation operator.To verify the effectiveness of our proposed model, we conduct experiments on two ECG datasets: the Ningbo dataset [20] from the Computing in Cardiology Challenge (CinC) 2021 database [21] and the Real World dataset which is collected by ourselves. The Ningbo dataset comprises 34,905 samples with a sampling frequency of 500 Hz and includes 25 classes representing different heart rhythm types. Each sample is 10 seconds long and contains multiple labels. The Real World dataset consists of 1,091 samples, with a sampling frequency of 1,000 Hz and 3 classes (normal, AF, and others). The duration of each sample in the Real World dataset is variable, and each sample has a single label. For consistency in processing, we extract a 10-second segment from each sample for classification. We utilize stratified sampling to partition 80% of each dataset into a training set and 20% into a test set.The two datasets represent multi-label and multi-class classification tasks, respectively. For the multi-label classification task, we use the following evaluation metrics: accuracy (Acc), sample F1 score (Sample-F1), area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve (AUPRC). For the multi-class classification task, we use accuracy (Acc), macro F1 score (Macro-F1), precision (Pre), and recall (Rec) as evaluation indicators.We compare our method with several popular baselines. 1) LSTM [22] : Long short-term memory networks (a special type of RNN).2) BiLSTM [23] : Bidirectional LSTM networks can better capture long-term dependencies in both directions. Considering the size of the two datasets, we use the Base version of the model for the Ningbo dataset and the Small version for the Real World dataset. For a fair comparison, the training parameters for all networks are kept consistent in the experiments: the batch size is set to 128, the learning rate is set to 0.001, the Adam optimizer is used, and the number of training epochs is set to 100. All experiments are performed on an NVIDIA 2080Ti GPU.Table II presents the experimental results on the two datasets. All indicators are expressed as percentages (%), with the best results highlighted in bold. Overall, our model demonstrates superior performance on both datasets. On the Ningbo dataset, the AUPRC is improved by 9% compared to the suboptimal MobileNetV3. On the Real World dataset, the macro F1 score is improved by 7% compared to the suboptimal Xresnet1d101. We attribute this phenomenon to the effective combination of CNN and Transformer architectures and the introduction of multimodal technology. Additionally, we observe that CNN-based models (MobileNetV3 and Xresnet1d101) often outperform LSTM-based and Transformer-based models, indicating that CNNs can extract more discriminative features for ECG classification tasks. The combined model, which leverages both CNN and Transformer architectures, harnesses the strengths of both architectures and achieves superior classification performance through their effective integration.We conduct ablation experiments on the model and explored it from three perspectives: 1) using only a single modality as input (ECG or Frequency Spectrum), 2) omitting the use of the Transformer layer to capture long-term dependencies when extracting unimodal features (w/o long dependency), and 3) not using the linear layer for modality feature alignment (w/o alignment). Table III shows the performance of the model in these three cases. The results indicate that when only ECG or frequency spectrum is used as input, the classification performance of the model is lower than that of the full model, demonstrating that the introduction of multimodal technology enhances the model's discriminative ability. In the scenarios of w/o long dependency and w/o alignment, the classification performance of the model decreases, further proving the effectiveness of these components.In this paper, we propose a multimodal model combined with frequency spectrum for ECG classification. Depthwise separable convolution is employed to initially extract unimodal features, and a Transformer layer is used to capture longterm dependencies. Our method incorporates a linear layer for modality feature alignment and a Transformer layer for multimodal feature fusion. Our model outperforms competitive baselines in both multi-label and multi-class classification tasks, demonstrating its effectiveness. In the future, we aim to design classification models using more modalities for heart disease diagnosis and integrate exogenous data to further enhance the model's performance. 
 
 Appendix:  



Reviews: review: title: title: This paper can be considered for publication.
main_review: main_review: The authors in this paper investigate the frequency-enhanced hybrid multimodal CNN-transformer network for electrocardiogram classification. The reviewer has the following comments. 
1. Would the experiment results be changed with the change of testing datasets?
2. Some typos and grammar errors are found in the paper. The authors are suggested to carefully double-check the manuscript.
title: title: Frequency-Enhanced Hybrid Multimodal CNN-Transformer Network for Electrocardiogram Classification
main_review: main_review: The paper is well-structured and easy to follow. The introduction clearly outlines the motivation and limitations of existing ECG classification methods. However, some minor improvements in clarity could further enhance readability. For example, the details of the proposed method in Section III could benefit from additional subheadings to differentiate between the different components (e.g., Frequency Spectrum Generation, Unimodal Feature Extraction, and Multimodal Feature Fusion). This would make it easier for readers to navigate and understand the technical aspects of the method.
title: title: A frequency-enhanced hybrid multimodal CNN-transformer network is proposed for the electrocardiogram classification. It is interesting and meaningful.
main_review: main_review: 1. The contributions of this paper are listed in the blow of Section I. However, the descriptions of the achieved effect is insufficient.

2. Some annotations of variables in formulas are missing. Please replenish them.