H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Authors: Rushuang Zhou
Yuanting Zhang
Yining Dong
Venue: ICML 2025 poster (2025)
Keywords: Electrocardiograph
Pre-trained models
Fine-tuning
Cardiovascular diseases.
Primary area: applications->health_medicine
TL;DR: This study allows for the utilization of pre-trained models with high computation efficiency and robust performance, exploring a path toward low-cost and efficient CVDs detection.

Abstract: Fine-tuning large-scale pre-trained models provides an effective solution to alleviate the label scarcity problem in cardiovascular diseases (CVDs) detection using electrocardiogram (ECG). However,  as the pre-trained models scale up, the computational costs for fine-tuning and inference become unaffordable on low-level devices deployed for clinical applications. Additionally, maintaining the model performance under low budgets in computational resources remains a significant challenge. However, a comprehensive study that can address them in a joint framework is still lacking. Here, we propose a holistic method (H-Tuning) for low-cost and efficient fine-tuning of pre-trained models on downstream datasets. Then, the inference costs of the models fine-tuned by H-Tuning are further reduced significantly using a knowledge distillation technique. Experiments on four ECG datasets demonstrate that H-Tuning reduces the GPU memory consumption during fine-tuning by 6.34 times while achieving comparable CVDs detection performance to standard fine-tuning. With the knowledge distillation technique, the model inference latency and the memory consumption are reduced by 4.52 times and 19.83 times. As such, the proposed joint framework allows for the utilization of pre-trained models with high computation efficiency and robust performance, exploring a path toward low-cost and efficient CVDs detection. Code is available at https://github.com/KAZABANA/H-Tuning

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Rushuang Zhou 12
Yuan-Ting Zhang 345 Yining Dong 62
Abstract
1. Introduction
Fine-tuning large-scale pre-trained models pro- vides an effective solution to alleviate the la- bel scarcity problem cardiovascular diseases (CVDs) detection using electrocardiogram (ECG). However, as the pre-trained models scale up, the computational costs for fine-tuning and inference become unaffordable on low-level devices de- ployed for clinical applications. Additionally, maintaining the model performance under low budgets in computational resources remains a sig- nificant challenge. However, a comprehensive study that can address them in joint framework still lacking. Here, propose a holistic method (H-Tuning) for low-cost and efficient fine-tuning of pre-trained models on downstream datasets. Then, the inference costs of the models fine-tuned by H-Tuning are further reduced significantly us- ing a knowledge distillation technique. Exper- iments on four ECG datasets demonstrate that H-Tuning reduces the GPU memory consumption during fine-tuning by 6.34 times while achieving comparable CVDs detection performance to stan- dard fine-tuning. With the knowledge distillation technique, the model inference latency and the memory consumption are reduced by 4.52 times and 19.83 times. As such, the proposed joint framework allows for the utilization of pre-trained models with high computation efficiency and ro- bust performance, exploring a path toward low- cost and efficient CVDs detection. Code is avail- -Tuming
Acting as the number one cause of death, cardiovascular diseases threaten the lives of millions of people worldwide (Kelly al., 2010; Mc Namara et al., 2019). In recent years, deep-learning models have been successful in diagnosing cardiovascular diseases (CVDs) using electrocardiograpyy (ECGG)Hannnun et al., 2019; Ribeiro et al., 2020; Strodthoff et al., 2020). However, gathering sufficient labeled data and computational resources required to train and implement deep learning models from scratch still very expensive and time-consuming, especially for developers from resource- limited communities. Fortunately, fine-tuning large-scale pre-trained models provides an effective solution to reduce the requirement of labeled data in downstream datasets. For example, Vaid et al. (2023) pre-trained large-scale vision transformer on a huge dataset with eight million ECG recordings, which demonstrated better transferability and performance than traditional network architectures. Sub- sequently, many studies have tried transfer the success of large pre-trained foundation models from the computer vision and natural language processing domains to medical intelligence domains. For example, foundation models in retinal imaging (Zhou et al., 2023), cancer imaging (Pai et al., 2024), and digital pathology (Wang et al., 2024) were developed for enhancing disease diagnosis performance on downstream datasets. In the field of ECG-based CVDs detection, researchers also validated the effectiveness of fine-tuning foundation models in reducing the requirement on labeled data (Han & Ding, 2024; Mathew et al., 2024; McKeen et al., 2024; Pham et al., 2024).
However, fine-tuning pre- rreitrained models becomes computa- tionally expensive the pre-trained models scale up. Specif- ically, standard fine-tuning optimizes all the parameters of pre-trained models using gradient backpropagation (first- order optimization), requiring a large amount of GPU mem- ory. Unlike industrial servers, such computational costs are unaffordable for low-level devices. Hu et al. (2022) pro- posed the low-rank adaptation (LoRA) alleviate this prob- lem and maintain the model performance on downstream datasets. Specifically, it freezes the pre-trained weights and injects trainable low-rank matrices to fine-tune the pre- trained models, reducing the number of trainable parame-
able at http://ghhuub..ooAAAAAAAAAAA
'Department of Biomedical Engineering, City University of Hong Kong, Hong Kong, China 'Hong Kong Center for Hcalth Engineering, Hong Kong, China 'Department of Electronic Engincering, Chinese University of Hong Kong, Hong Kong, China 'Hong Kong Institutes of Medical Engineering, Hong Kong, China 'The AICARE Bay Lab, Guang- dong Medical University, Dong Guan, China 'Department of Data Science, City University of Hong Kong, Hong Kong, China. Correspondence to: Yining Dong <yinidong@eni @d..k..dd..kk>.
Serebro-Cardiovascule
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Discase Detection with Pre-Trained Models
ters and the GPU memory consumption in saving parameter gradients. However, LoRA cannot avoid caching network activations required during the backpropagation process, greatly limiting its efficiency in reducing GPU memory con- sumption. Malladi et al. (2023) addressed this problem by developing a backpropagation-fret method (MeZO), which can fine-tune pre-trained models with just forward pass. McZO calculates parameter gradients using a zeroth-order optimization method and waives the requirement for activa- tions caching. It can achieve comparable performance with fine-tuning on natural language processing tasks. However, McZO's 's performance would collapse without the assistance of prompt engineering (Malladi et al., 2023). Consequently, its effectiveness in medical image and physiological sig- nal processing is questionable as prompt engincering is not feasible in these domains. In summary, the high and often prohibitive computational costs associated with fine-tuning pre-trained models have greatly impeded their widespread application in cardiac healthcare within resource-limited communities. This phenomenon highlights the need for computationally efficient fine-tuning methods with robust performance the field of ECG-based CVDs detection. Apart from fine-tuning costs, pre-trained models also intro- duce high inference costs, which is also a pressing issue when deploying them on low-level devices. Specifically, de- ploying large-scale models is feasible on personal comput- ers and laptops but is very difficult on mobile devices, such as phones and smartwatches. It limits the development of mobile cardiac health in the era of pre-trained models. For- tunately, the knowledge distillation technique (Hinton et al., 2015) transfer the knowledge of large-scale teacher mod- els to small-scale student models, providing solution to alleviate this problem. In recent years, many studies have validated its effectiveness in ECG-based CVDs detection. For example, Sepahvand & Dodali-Mohammadi (2022) vali- dated that student models can achieve comparable CVDs de- tection performance with large-scale teacher models while demonstrating low computational costs. Despite their effec- tiveness in reducing inference costs, the computational costs of training or fine-tuning teacher models for knowledge distillation are not considered.
ically, mix-order optimization method is first designed to accurately estimate gradient information with low GPU memory footprints during the fine-tuning process. A low- rank adaptation technique is then integrated to reduce the number of trainable parameters. Subsequently, the shallow and deep layers of pre-trained models are updated with dif- ferent schemes to ensure low fine-tuning costs while avoid- ing significant drops in detection performance. Experiment results on four downstream datasets demonstrate that H- Tuning achieves comparable performance with standard fine-tuning while reducing the GPU memory consumption by 6.34 times. To reduce the inference costs of the fine- tuned models for mobile cardiac healthcare, a knowledge distillation technique utilized to transfer the knowledge from large-scale fine-tuned models tiny student models. In comparison the teacher models, the inference latency, the GPU memory consumption, and the number of parame- ters of the student models are reduced by 4.52, 19.83, and 194.23 times, respectively. Notably, student models also achieve similar CVDs detection performance when com- pared to the teacher models. Additionally, the knowledge transfer process improves model performance in recogniz- ing CVDs from 1-lead and 3-lead ECG, facilitating efficient mobile healtheare using wearable ECG devices. With robust CVDs detection performance and low computational costs, the proposed workflow can be deployed for accurate and mobile cardiac healthcare, providing an effective solution for low-cost and efficient ECG-based CVDs detection. The major contributions of the proposed paradigm are listed below:
First, mix-order optimization method is proposed to accurately estimate the gradient information using a coarse-to-fin estimation mechanism. Compared with traditional optimization methods, it greatly reduces the computation costs for fine-tuning pre-trained models while maintaining the CVDs detection performance of the fine-tuned models.
Second, a holistic framework (H- (H-Tuning) is developed to integrate the mix-order optimization with low-rank adaptation and a novel layer-dependent model update scheme, enhancing both computational efficiency and robustness. Then, a knowledge distillation technique introduced to reduce the computational costs for model inference.
In summary, the high fine-tuning and inference costs of pre-trained models hinder their application in ECG-based CVD detection, especially in resource-limited environments. However, comprehensive study that addresses them in joint workflow is still lacking. Additionally, how to main- tain the fine-tuning performance under low computational budgets still needs to be explored. In this study, a holistic framework (H- Tuming) is proposed to fine-tune pre-trained models with low computational costs while maintaining good model performance. By integrating knowledge distil- lation technique, a joint workflow for low-cost and efficient ECG-based CVDs detection is developed (Fi.. Specif-
Third, a comprehensive workflow is proposed for reducing the fine-tuning and inference costs of pre- trained models while maintaining diagnostic perfor- mance, exploring a new path toward low-cost and effi- cient CVDs detection using ECG.
2

---

H- H-Tunnig: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Workflow for E Efficient Cardiac Healthcare
ECG Datasets
Not Feasible Due High Computational Costs
Mobile Cardiac Healthcare
Phones
Eemcey-Efficient
Fine-tuned Model
Knowledge
Tiny Student Models
H-Tuning
Distillaion
Smartwatches
Pre-trained
Model
Accurate Cardiac Healthcare
Computers
b. Mix-order Optimization
c. Low-rank Adaptation
d. Model Update Scheme
Mihi-batch ECG Samples
ECG Samples
Frozen
Trainable
Model
Sample
Tiny batch ECG Samples
Pre-trained Model
Low-rank Matrices
Perturbation
Perturbation
Shallow Layers
Deep Layers
Model
Model
Model
Prediction
Training P Phase
Backpropagation
SPSA Method
ECG Samples
Updated Shallow Layers
Updated Deep Layers
First-order Gradient
Zeroth-order Gradient
Merge
Gradient Refinement
Pre-trained Model
Low-rank Matrices
Updatad
Updated Model
Mix-order Gradient
Model
Descend
Prediction
Testing Phase
Mik-order optimiatton
F First-onder optimiztioon
Figure 1. The workflow of efficient CVDs detection and cardiac healtheare with ECG data. Using the proposed H- H-Tuning framework, large-scale pre-trained model first fine-tuned the downstream ECG datasets in memory-efficient way. Then, to further reduce the computational cost, the knowledge of the large-scale fine-tuned model is transferred into tiny student models using a knowledge distillation method.
2. Method
increases the complexity of forward computation. Based on the estimated gradients at the t-th iteration, the parameters can be updated by the gradient descent process as
2.1. Preliminaries about Zeroth-order Optimization
Fine-tuning pre-trained models using gradient backpropa- gation provides robust method for solving downstream CVDs detection tasks. However, the gradient backpropaga- tion process requires saving the network activations during the forward process, which consumes a prohibitive amount of GPU memory. On the contrary, zeroth-order optimiza- tion estimates the gradients using the loss differences, waiv- ing the need store the network activations (Spall, 1992; Flaxman et al., 2005; Duchi et al., 2015). Specifically, the gradients of a model with parameters 0 can be estimated by the simultaneous perturbation stochastic approximation (SPSA) method (Flaxman et al., 2005),
0 = (((())
(2)
Assuming that the loss function C(0) is L-Lipschitz smooth, the differences between E, [())) and the true gradients
V (() be bounded (Nesterov & Spokoiny, 2017; Gao et al., 2018),
Ex [C -
4
VC(()=
-
1 n
C(e p2,) L(B 122z)
When -> O, Eq. (1) provides us an unbiased estimation of the true gradients of 0 E, [C(()] VL(0), which indicates the feasibility of the zeroth-order methods in opti- mization. In the implementation, is usually set to a small constant (le-5, 1e-3), and n set to for efficient model training, which makes (Z()) become a biased estimation of the true gradients (Malladi al., 2023).
Zi.
24
where 2, E 2d is random vector sampled from the standard Gaussian distribution N(0, Ia), y is the perturbation scale and n is the number of function queries. Note that a large value reduces the variance of the estimated gradients but
3

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Discase Detection with Pre-Trained Models
2.2. Mix-order Optimization
hyperparameter balancing the first-order and the zeroth- order methods. The direction of VL 0) depends on the random vector 2, which might have a large discrepancy with the true gradients. Consequently, we introduce V (B;; 0) to refine the direction of the estimated gradients with an im- portant factor of x. Compared with the gradients V (B 0) estimated by the first-order methods, (B 0) has a much larger expected Frobenius norm when n = 1, which can be formulated as (Malladi et al., 2023),
Although the zeroth-order methods demonstrate low GPU memory footprints, the models fine-tuned with them cannot achieve comparable performance to the models fine-tuned with the first-order methods, which calculate the gradient information using backpropagation (Li et al., 2024). In natural language processing tasks, prompt engineering is adopted to make the optimization trajectory well-behaved, which enables the zeroth-order methods to optimize the whole network (Malladi et al., 2023; Zhang et al., 2024). However, prompt engineering is only feasible when text data is available for model training, which greatly limits the effectiveness of the zeroth-order methods in ECG-based CVDs detection. In response to the aforementioned limita- tions, we propose a mix-order optimization method, which enables memory-efficien and robust model fine-tuning with- prompt engineering.
E [ = dE [IV2(B
(6)
dN E [IV 0)1]]
The proof is provided in Appendix A.1. When 1 S N1 < N < d, we have,
E [ > E [[
(7)
Given random batch of ECG signals B = (xi, 3 and a model with parameters 9, the zeroth-order method (SPSA, Eq. (1)) is first utilized to provide an initial estimation of the parameter gradients, defined as (L 9). Here, () can be regarded as an approximation of A (()) Consider- ing that multiple CVDs can be diagnosed from each ECG segment, the CVDs detection process should be summarized as a multi-labe classification task. Consequently, the loss function utilized in our study can be defined as a multi-label binary cross-entropy loss,
()(: is weighted sum of the gradients estimated by the first-order and the zeroth-order methods. Without norm refinement, its direction and Frobenius norm would be dominated by CLB; f) unless À z 1. A naive solution
is simply dividing the BL(B;) 0) by d.1.
However, as shown in Eq. the random vector z is sampled from the standard Gaussian distribution. This suggests that the dis- crepancy between || [B; 0)/y v
N
dNr and VVL:B ; 0) is N
not stable and will affect the refinement process. To address this problem, we normalize the magnitude of C I) using || C(B 0)ll and (5)
L(B:0)
N
C
1 NC N
(1 log(1 Pt.e) + Vs,e log Pi.e:
Compared with the standard zeroth-order optimization, the proposed mix-order optimization provides more stable and accurate estimation of the parameter gradients in both direc- tion and Frobenius norm. Without prompt engineering, this advantage enables the mix-order optimization to demon- strate a better convergence behavior than the zero-order methods. Additionally, the extra computational burdens introduced by the gradient refinement process are ignorable because the backpropagation process is only conducted on the tiny subset B,.
11
(4)
where Pi.e P(c 1)0;,;)) is the model prediction on class e and C is the number of categories. According to (3), the variance of VL(;; 6) increases rapidly the model size scales up, which results in slow convergence and poor model performance (Gautam et al., 2024). To address this problem, we propose to utilize the first-order method (gradient backpropagation) to refine the norm and the direction of the estimated gradients L 0). In order to avoid heavy computational burdens, backpropagation is only conducted a tiny subset B1 (x., Ui M sampled from B, with N1 & N. Specifically, the gradient refinement process can be formulated as,
2.3. Hybrid Tuning for Lightweight and Robust Model Training
While the proposed mix-order optimization method demon- strates robust performance, its computational efficiency and robustness can be further improved by two techniques. First, we integrated our mix-order optimization with low-rank adaptation, which approximates the incremental update of the pre-trained weights by two low-rank matrices (Hu et al., 2022). With inputs X, the output of linear layer with parameters 0 (Wo, b) can be formulated as,
()::8)x
(5) V VL(B;;0)] (L):0), [VL(B: 8]
= AVL(B::0) + +(1 (1-X) -))
where VC(B;; 8) is the parameter gradients estimated by the gradient backpropagation on the tiny subset B1, X is
h = (Wo + BA)X + b,
(8)

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Discase Detection with Pre-Trained Models
where B E RRdxx and A E Rrxdu, and the rank < min (dnn: dew)) During model training, pre-trained weight Wo is frozen while the bias term b and the low-rank matrices A, B are trainable. Compared with tuning the whole linear layer, the low-rank adaptation greatly reduces the number of trainable parameters, which decreases the variance in the SPSA process according to Eq.(3). Besides, it can signif- icantly reduce the storage consumption of the fine-tuned models, we only need to save the optimized low-rank matrices.
is formulated as,
Lx(B;0s)
N C
1
ΣΣ
(1 - v.,) log(1 - -v(c)= + + log p..
NC
1=1 c=1
(10)
Ls(B:0s) = L(B:0s) + Lx(B:
(11)
The second technique is applying different optimization schemes to different layers distributed at different locations of the model. Specifically, we define the last M = 2 linear layers as the deep layers and the remaining layers as the shal- low layers. It is well known that the deep layers contain the task-variant knowledge while the shallow layers contain the domain-invariant knowledge (Sharif Razavian et al., 2014; Tajbakhsh et al., 2016). More importantly, fine-tuning the deep layers using first-order optimization is computation- ally efficient, whereas fine-tuning the shallow layers using the same method introduces heavy computational burdens. Consequently, we propose tuning the deep layers using first- order optimization to provide good adaptability to the model on downstream tasks. Additionally, we tune the shallow lay- ers using the proposed mix-orde optimization method to improve computational efficiency during the model training. In summary, the proposed pipeline can be formulated as,
where p.c = P(c = 110s: pi.c = = P(c==10r;; x.). B = ( is random batch of ECG signals and the corresponding ground truth for knowledge distillation. To allow the student models to handle ECG recordings with var- ious numbers of leads and meet the requirements for mobile cardiac healtheare, the input x can be ECG signals with different numbers of leads. If only 1-lead ECG is accessible for the student model, we use lead I of x, to generate 2;. If only 3-lead ECG is accessible for the student model, we use leads II, VI, and V5 of I;; to generate 2.. Considering the number of trainable parameters in the student model is very small, we use the gradient backpropagation to optimize its parameters As.
3. Results
3.1. Datasets and Signal Pre-processing
8TTh 17 L(B; m P K M 0.41 = 0Thh - LBB )(T)),, m < K - M²
In this study, the Chapman-Shaoxing database (Zheng et al., 2020b), the Georgia 12-lead ECG Challenge (GI2EC) database (Alday et al., 2020), the Physikalisch-Techniie Technische Bundesanstalt (PTB-XL) database (Wagner et al., 2020), and the Ningbo database (Zheng et al., 2020a) are used for the performance evaluation of our H-Tuning frame- work. The four datasets were also included in the Physionet 2020/2021 challenge (Alday et al., 2020; Reyna et al., 2022). To be specific, there are 10,646 12-lead ECG recordings in the
where K is the total number of layers within the model, ((B; 0mh) is computed using the gradient backpropagation and (B )x
x is estimated using (5). The complete algorithm of H-Tuning is presented in Appendix A.2.
2.4. Knowledge Distillation for Fast Model Inference
database. In addition, the G12EC database, the PTB-XL database, and the Ningbo database contain 10344, 21837, and 34,905 12-lead ECG recordings, respectively. The recordings from the above databases last around 10 seconds with sampling rate of 500 Hz and are annotated with multi-label CVDs ground truths. Note that multiple CVDs can be recognized from each recording si- multaneously. Consequently, automatic CVDs detection can be formulated as a multi-label classification task. To en- sure sufficient representation across the training, validation, and test sets, only CVD categories containing more than 200 ECG recordings are included during model training and evaluation. For signal pre- tre-processing. band-pass (1- 47Hz) is applied to remove the potential noises from the raw ECG recordings, such as the power-line interference and the motion artifacts. Then, z-score normalization is utilized to normalize the filtered signals.
In application scenarios with very limited computation re- sources, the model inference efficiency should be high enough satisfy the hardware requirements. Although the model fine-tuning process can be carried out on the cloud servers, the inference process should be performed on the local devices. When mobile and continuous moni- toring is required, the inference speed should be high, and the corresponding memory footprints should be low. Conse- quently, we first conduct the proposed H- H-Tuning on a large pre-trained model on downstream tasks and then transfer its knowledge to tiny model using classic knowledge distillation method (Hinton et al., 2015). The optimized parameters of the large fine-tuned model are defined as 0r, and the parameters of the tiny student model are defined as 0g. During the knowledge distillation process, the large model frozen, and the training loss of the student model
5

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Discase Detection with Pre-Trained Models
Table 1. Performance of H-Tuning and the compared models on four public datasets. The average performance across four seeds is presented. The dashed lines separate the memory- efffcceet fine-tuning methods and the fine- muninu methods with first-order optimization. The unit of memory is Gigabyte (GB).
G12EC Dataset
PTB-XL XL Dataset
Ningbo Dataset
Chapman Dataset
Methods
Memory
Macro AUC Maero P2-2 Memory Macro AUC Macro F3- Memory Macro AUC Maero P Memory Macro AUC Maero F3- -22
Full FT
9.214
0.869
0.588
9.212
0.919
0.618
9.211
0.933
0.591
9.211
0.930
0.623
LORA
K,754
0.870
0.592
8.754
0.919
0.618
8.754
0.934
0,580
8.754
0.932
0.636
1.166
0.852
0,545
1.416
0.897
0.582
1416
0.906
0,530
1.416
0.898
0.579
McZO
18815
0.532
0.327
1.814
0.515
0.241
18814
0.504
0.231
1.814
0.499
0.277
MeZO + LoRA
1.437
0.507
0.316
1.444
0.483
0.227
1.44
0,530
0.234
1.444
0.481
0.278
Addax
2,000
0.863
0,573
2.003
0.898
0.578
2.003
0,909
0,509
2.004
0,929
0.596
Addax + LoRA
1,453
0.857
0,568
1,451
0.899
0,582
1,451
0.901
0.495
1,451
0,907
0,588
LoHO
2.002
0.851
0,554
2,000
0,902
0,588
2.001
0.907
0.535
1.998
0,905
0.574
LoHO+ LoRA
L.453
0.851
0,557
1.453
0,900
0,581
L.453
0.909
0,535
1.453
0.904
0,580
H-Tuning
1.453
0.870
0.586
1.453
0,923
0.628
1.453
0.931
0.550
1.453
0.929
0.634
Table 2. Ablation study of H-Tuning. The average performance across four seeds is presented. The unit of memory is Gigabyte (GB).
G12RC Dataset
PTR-XI. Dataset
Ningho Dataset
Chapman Daraser
Methods
Memory
Macro AUC Macro F Memory Macro AUC Macro F Memory Macro AUC Macro Fr
Memory Macro AUC Macro F
Without SPSA gradient cstimation
1.451
0.866
0.575
1.451
0.919
0.616
1.451
0.921
0.525
1.451
0.927
0.612
Without gradient refinement
1.453
0.851
0.557
0.900
0
1,453
0.909
0.535
1,453
0.901
0.580
Without gradicnt nommaization
1,453
0,870
0,588
1.453
0,917
0.609
1,453
0.924
0.536
1,453
0,928
0.596
Without low-rank adaptation
2,002
0.564
2,000
0.907
0.588
2,000
0.924
0.527
0.922
0.603
H. Tuning
1.53
0.870
0.586
1.453
0,923
0.628
1,453
0.931
0.550
1,453
0.929
0.634
3.2. Experiment Protocols and Evaluation Metrics
(MeZO) (Malladi et al., 2023), Low-order Hybrid Optimizer (LoHO) (Chen et al., 2025), and Addition of Gradient Es- timates through Memory-Efficient Execution (Addax) (Li et al., 2024). Full FT, LoRA, and LP are three popular adap- tation approaches that use first-order optimization to update the trainable parameters. McZO is the bascline model in zeroth-order optimization. Addax and LoHO combine the order and zeroth-order optimization methods, serving as state-of-the-ar models in memory-efficien fine-tuning. At the same time, we integrate them with LoRA to formulate new comparison methods (MeZO + LoRA, Addax LoRA, LoHO + LoRA), which have less trainable parameters than them. Note that the low-rank matrices are implemented in every layer of the pre-trained backbone. Details of the compared methods are provided in Appendix A.3.
In our experiments, a pre-trained model provided by Zhou et al. (2024) acts as the backbone for all the compared fine- tuning methods. Specifically, the backbone has 50.494 mil- lion parameters and is pre-trained on the Clinical Outcomes in Digital Electrocardiology (CODE) dataset (Ribeiro al., 2019; 2020). Subsequently, the backbone is fine-tuned and evaluated on the four public datasets with a limited number of labeled samples. For each dataset, training set and a held-out test set are randomly sampled in ratio of I: 9. Then, a validation set is collected from the training set and accounts for 20% of it. In our implementation, the vali- dation set is used select the best checkpoint during the fine-tuning process. During the model evaluation process, we utilize two metrics to quantify the performance of dif- ferent methods in multi-label CVDs detection: macro F3=2 and macro AUC. Following the settings in Strodthoff et al. (2020), we set the 3 value to be 2. Additionally, we record the peak GPU memory footprints of different methods dur- ing fine-tuning. In our experiments, we compare H-Tuning with several baseline models: Full Fine-Tuning (Full FT), Low-Rank Adaptation (LoRA) (Hu et al., 2022), Linear Probing (LP), Memory-Efficien Zeroth-Ordes Optimization
3.3. Comparison of H-Tuning with Existing Methods
As presented in Table 1, we report the CVDs detection performance and GPU memory footprints of the proposed H-Tuning and the compared methods. It can be observed that H-Tuning is the only memory-efficien method that achieves comparable performance with Full FT and LoRA. Specifically, H-Tuning achieves an average macro F9=2
6

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Discase Detection with Pre-Trained Models
of 0.600 across four datasets and only has performance losses of 0.5% and 0.7% compared with Full FT and LoRA, respectively. More importantly, H-Tuning achieves a re- markable reduction in GPU memory footprints by 6.02 to 6.34 times compared with LoRA and Full FT, indicating its flexibility and compatibility in clinical devices with lim- ited GPU memory. Specifically, LoRA and Full FT use first-order optimization methods to calculate the parameter gradients, which require large GPU memory to save the activation outputs for backpropagation on the mini-batch samples. In contrast, H-Tunnng estimates the gradients using the proposed mix-order optimization, significantly reduc- ing the need to store the activation outputs and the overall GPU memory footprints. Compared with other memory- efficient methods, H-Tuning demonstrates the best CVDs detection performance without significantly increasing the GPU memory footprints. Compared with the method with the lowest GPU memory footprints (L.P), the extra GPU memory consumption introduced by H-Tuning is ignorable in model training (0.037 GB). However, the improvements in CVDs detection performance are remarkable. For exam- ple, H-Tuning achieves a marco F3=2 score of 0.634 on the Chapman dataset, outperforming LP by 5.5%. In Appendix
method and the first-order gradient backpropagation by hy- perparameter À (5). Here, we adjust its value from 0.85 to 0.99 and present the corresponding model performance across four datasets. As shown in Fig.2a, the experiment results reveal that the model performance is not sensitive to the varying X, suggesting the stability of the proposed module.
(2) The effect of the backpropagation batch size. In the gradient refinement process, conduct gradient backprop- agation on a tiny subset Bi = (I Mi j==1 to refine the esti- mated gradients in (5). As shown in Fig.2b, we adjust N1 from to 8 and report the corresponding model performance across four datasets. It can be observed that the model per- formance generally improves as N₁ increases on the Ningho and the G12EC datasets. However, increasing Ni has a lim- ited impact the PTB-XL and the Chapman datasets. This phenomenon indicates that the difficulty of recovering the true gradient information varies across datasets.
Ni
(3) The effect of the rank parameter for low-rank adap- tation. The proposed H- H-Tuning framework uses low-rank adaptation (Hu al., 2022) to reduce the number of train- able parameters during the mix-order optimization process. The rank of all the low-rank matrices is controlled by the pa- rameter r, and the number of trainable parameters decreases as r decreases. As presented in Fig.2c, we adjust from 4 to 16 and provide the corresponding model performance across four datasets. The experiment results demonstrate that the model performance improves as r increases. This phenomenon can be explained by the various representa- tion capacities of the fine-tuned models with different r. Specifically, increasing the number of trainable parameters enables the low-rank matrices to store more task-specific information in CVDs detection, thus improving the model performance. On the other hand, it is important to note that it will also increase the storage costs of the matrices. Con- sequently, the value rank parameter should be carefully selected according to the maximum storage spaces and the expected CVDs detection performance.
A.4, provide detailed comparisons of different methods using more evaluation metrics.
3.4. Ablation Studies
In this section, ablation studies are performed to demon- strate the contribution of the modules implemented in the proposed H-Tuning As shown in Table 2, we successively remove the components from the H-Tuning and report the corresponding CVDs detection performance across four datasets. (1) Utilizing SPSA methods to estimate the pa-
rameter gradients improves model performance. When the SPSA gradient estimation module is removed from the H- Tuing (A- 1), backpropagation cannot calculate the gra- dient information with enough batch sizes due to limited GPU memory spaces. Therefore, the CVDs detection per- formance decreases in all datasets. (2) A remarkable degen- eration in model performance is observed when the gradient refinement module removed (A = 0). It indicates that refining the initial gradients estimated by the SPSA method benefits the optimization process. (3) Applying gradient normalization (5). during the refinement process has a positive effect on the model performance. Specifically, an obvious performance enhancement is attained when the module is implemented. Ablation studies of H-Tuning on more evaluation metries are provided in Appendix A.4
3.6. Inference Costs Reduction through Knowledge Distillation
Apart from the training process, the computational costs of the fine-tuned models during the inference process are important for their implementations in clinical practices. Therefore, through a knowledge distillation technique (Hin- ton et al., 2015), we compress the fine-tuned large-scale model into a tiny student model, which is able to be de- ployed low-level devices for inference. Considering that most mobile ECG devices only have 1-3 leads, we also investigate the performance of the student model under var- ious lead configurations. As shown in Table 3, we report the CVDs detection performance of the student models
3.5. Sensitivity Analyses
(1) The effect of the control parameter for mix-order optimization. In the proposed mix-order optimization mod- ule, control the importance of the zeroth-order SPSA
7

---

H-Tuning: Toward Low-Cost and Efficient BCG-based Cardiovascular Disease Detection with Pre-Trained Models
Matro Fz=2
Macro Fg
Micrro Fa.
G12EC Datssot
0.674
0.578
0.579
0.956
Dataset
Chapmen Detaner
0
PTD-XL Detaset
0.800
0:611
0.80
Ningho Dateaar
Miggoo
0.58
PTB-AL Dataset
PTB-XL
Ningho
0.522
0.00
G12EC Datrset
Chapman
0928
0.58
0:50
0.52
664
0.54
0.35
0.55
0.80
0.64
A
0
0.95
0.99
N2-2
N2
Effect of the parameter
(b) Effect of the parameter N
(c) Effeec of the parameter
Figure The sensitivity analysis on the H-Tuning.
Table 3. Overall CVDs detection performance of the student models with various numbers of ECG leads. The average performance across four seeds is presented. 'None' represents that the student models are optimized using the ground truths without teachers' assistance.
G12FC Dataset
VTB-XI.
Dataset
Ningho Dataset
Chapman Dataset
Teacher
Student
Macro AUC
Macro F==2
Macro AUC
Macro Fp=2
Macro AUC
Macro F
Macro AUC
Macro F F
None
12-L
0.847
0.558
0.903
0.592
0.921
0.530
0.927
0.617
12-Lead
12-Lead
0.868
0.582
0.921
0.628
0.937
0.568
0.933
0.632
None
12-Lead
0.834
0.537
0.886
0.559
0.903
0.500
0,911
0.577
3-Lead
3-Lead
0.574
0.928
0.549
0,919
0.614
None
1-Lead
0.772
0,457
0.830
0.462
0.844
0.420
0.847
0.498
12-Lcad
I-Lead
0.795
0.495
0.843
0.509
0.866
0.469
0,862
0.538
Teacher's 's performance
0.870
0.586
0,923
0.628
0.931
0.550
0.929
0.634
across four datasets. Note that we use leads II, V1, and V5 simulate 3-lead ECG signals and use lead sim- ulate 1-lead ECG signals. The performance of the teacher models fine-tuned by H-Tuning is presented for compari- son. The results demonstrate that the student models with 12-lead ECG achieve similar and even better diagnostic per- formance than the teacher models. Additionally, it can be observed that the participation of teacher models introduces remarkable improvements in student models' performance, highlighting the effectiveness of the knowledge transfer process. Specifically, the average improvements macro F3=2 score across four datasets 2.81% % 3.98%, and 4.36% for 12-lead, 3-lead, and 1-lead ECG signals. We also present the computational efficiency of the teacher and stu- dent models with different numbers of ECG leads in Fig.3. Specifically, the GPU memory footprints, the inference time, and the number parameters of different models are uti- lized to quantify their computational efficiency. can be observed that the student models have only 0.26 million parameters. They achieve inference time of 1.28 ms and only consume 18.05 MB of GPU memory during the ECG screening process. Compared with the large-scale teacher model, the student models speed up the inference speed by 4.52 times and decrease the GPU memory footprints and the number of parameters by 19.83 times and 194.23 times,
respectively. Consequently, the results demonstrate the high inference efficiency of the student models, highlighting their
great potential in cardiac healthcare using low-level mobile devices. The architecture of the teacher and student models is provided in Appendix A.3.
3.7. External Validation on Wearable 12-lead ECG Dataset
External validation is an important approach to evaluate the generalization performance of the proposed method on unseen datasets. Specifically, We combine the G GI2EC, PTB-XL, Ningbo, and Chapman datasets to fine-tune the pre-trained model and generate three teacher models using three methods (Full FT, LoRA (Hu al., 2022) and the proposed H-Tuning). Following the dataset splitting method defined in Section.3.1, only 10% of the labeled data within the training set are used for fine-tuning. Subsequently, the knowledge distillation technique defined in Eq. is uti- lized to create three corresponding student models. Then, an external validation set consisting of 7000 wearable 12- lead ECG signals is used to evaluate the CVDs detection performance of our classifiers in mobile cardiac healthcare. The external dataset is provided by (Lai et al., 2023), which contains 60 types of CVDs. Here, only the CVDs that
8

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
60
400
6
5.79
Inference GPU Usages (MB)
357.87
The number parameters (M)
50-
50.50
Inference (ms)
6
300
40
200
3
2-
1.28
1.28
100
18.05
17.19
17.02
Teacher Model Twelve-Lead ECG
Student Model with Three-Lead ECG Student Model Single-Lead ECG
Student Model Twelve-Lead ECG
Figure 3. Inference efficiency of the teacher model and the student models with different numbers of ECG leads. The batch size for the four models is set to
Table External validation on wearable 12-lead ECG dataset. The average performance across four seeds presented.
Methods
Ranking Loss
Coverage +
Macro AUC
MAP
Macro G f
Macro Fa 2
Teacher Models
Full
0.137
5,595
0.870
0.600
0.314
0.570
LoRA
0.134
5,440
0.879
0.598
0.319
0.579
H-Tuning
0.141
5,484
0,866
0,575
0.312
0.567
Student Models
Full FT
0.135
5,462
0.867
0.566
0.287
0.534
LoRA
0.129
5.384
0.874
0.582
0.302
0.543
H H-Tuning 0.127
5.297
0.880
0.598
0.311
0.551
co-exist in the external dataset and the training set are se- lected for model evaluation, including NSR, QAb, TAb, SB, SA, PAC, AF, AFL, PVC, IAVB, BBB, CRBBB, IRBBB, CLBBB, and PR. The CVD detection performance of the six classifiers on the external dataset is shown in Table The results demonstrate that the teacher model generated by H- H-Tuming achieves performance similar Full FT and LoRA. As shown in Table 1, the GPU memory consump- tions of H-Tuning are 6.34 times smaller than Full FT. The above two observations reveal that H- H-Tuning can provide low-cost and effective solution fine-tuning pre-trained models. Additionally, our student model performs better than the compared methods. This phenomenon could be explained by the fact that strong teachers might not have better teaching ability than weaker teachers (Huang et al., 2022).
demonstrate that the proposed H-Tuning outperforms other memory-efficient methods in ECG-based CVDs detection with remarkable superiority. Additionally, H- H-Tuming sig- nificantly reduces GPU memory usage and achieves com- parable performance with first-order fine-tuning methods. Subsequently, the fine-tuned models are compressed into small-scale student models through a knowledge distillation technique. Compared with large-scale models, small-scale student models have significantly lower inference costs and are more suitable for mobile cardiac healthcare. In con- clusion, this paper designs joint workflow reduce the fine-tuning and inference costs large-scale pre-trained models in ECG-based CVDs detection. We hope the pro- posed workflow could pave the way toward low-cost and efficient CVDs detention with pre-trained models. In the future, we plan to explore the applications of H-Tuning on other physiological signals, such as electroencephalogrrnss
4. Conclusion
In this paper, we propose the H-Tuning framework for fine-tuning pre-trained models in low GPU memory sumption. Experiment results on four downstream datasets

---

CAA.ERRCUUAATTONAA.............. H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Impact Statement
Osterspey, A., Tamargo. J. L., and Zamorano, J. L. Ac- c/aha/ese 2006 guidelines for the management of patients with atrial fibrillation. Circulation, 1 11477):257-<354, 2006. doi:
Our H-Tuning provides a pathway toward low-cost and effi- cient ECG-based cardiovascular disease detection by design- ing a joint workflow to reduce the fine-tuning and inference costs of large-scale pre-trained models. It can accelerate and popularize the clinical applications of pre-trained ECG anal- ysis models in resource-limite communities. Additionally, it has the potential to be generalized to many other fields in machine learning domains.
Gao, X., Jiang, B., and Zhang, S. On the information- adaptive variants of the admm: iteration complexity perspective. Jounaal of Scientific Compuring, 76:327-363, 2018.
Gautam, T., Park, Y., Zhou, H., Raman, P., and Ha, W. Variance-reduced zeroth-order methods for fine-tuning language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.). Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15180-152000 PMLR, 21-27 Jul 2024.
Acknowledgements
This work was supported in part by InnoHK Project at Hong Health Engineer- ing (COCHE), in part by the the National Natural Science Foundation of China (22322816) and in part by the City University of Hong Kong Project (9610640).
Kong Centre for Cerebro-cardiovan
References
Han, Y. and Ding, C. Foundation models in electrocar- diogram: A review. arXiv preprint arXiv:2410.19877, 2024.
Alday, E. A. P., Gu, A., Shah, A. J., Robichaux, C., Wong, A.-K. I., Liu, C., Liu, F., Rad, A. B., Elola, A., Seyedi, S., al. Classification of 12-lead ECGs: the physionet/- computing in cardiology challenge 2020. Physiological measurement, 41(12):124003 2020.
Hanuun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H.,
Bourn, C., Turakhia, M. P., and Ng. A. Y. Cardiologist- level arrhythmia detection and classification in ambu- latory electrocardiograms using deep neural network. Nature medicine, 25(1):65- 69, 2019.
Chen, M., Huang, Y.-L., and Wen, Z. Towards efficient low-order hybrid optimizer for language model fine- tuning. Proceedings of the AAAI Conference on Arti- ficial Intelligence, 39(22);3605--23666 Apr. 2025. doi: 10.
Hinton, G. Vinyals, O., and Dean, J. Distilling the knowl- edge in a neural network, 2015.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Duchi, J. C., Jordan, M. I., Wainwright, M. J., and Wibisono, A. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788-280) 2015.
Huang. T., You, S., Wang, F., Qian, C., and Xu, C. Knowl- edge distillation from stronger teacher. Advances in Neu- ral Information Processing Systems, 35:33716-33721 2022.
Flaxman, A. D., Kalai, A. T., and McMahan, H. B. Online convex optimization in the bandit setting: gradient de- scent without a gradient. Proceedings of the Sixteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '05, pp. 385-394, USA, 2005. Society for Indus- trial and Applied Mathematics. ISBN 0898715857.
Kashou, A. H.,, Basit, H., and Chhabra, L. Electrical Right and Left Axis Deviation. StatPearls Publishing, Treasure Island (FL), 2023.
Fuster, V., Rydén, L. E., Cannom, S., Crijns, H. J., Curtis, A. B., Ellenbogen, K. A., Halperin, L., Heuzey, J.-Y. L., Kay, G. N., Lowe, E., Olsson, S. B., Prystowsky. E. N., Tamargo, J. L., Wann, S., MEMBERS, A. T. F., Smith, S. C., Jacobs, A. K., Adams, C. D., Anderson, J. L., Antman, F. M., Halperin, J. I.., Hunt, S. A., Nishimura, R., Ornato, J. P., Page, R. L., Riegel, B., GUIDELINES, E. F. P., Priori, S. G., Blanc, J.JJ., Budaj, A., Camm, A. J.. Dean, V., Deckers, J. W., Despres, C., Dickstein,
Kelly, B. B., Fuster, V., et al. Promoting cardiovascular health in the developing world: a critical challenge to achieve global health. National Academies Press, 2010.
Lai, J., Tan, H., Wang, J., Ji, L.., Guo, J., Han, B., Shi, Y., Feng, Q., and Yang, W. Practical intelligent diagnostic algorithm for wearable 12-lead ECG via self-supe learning on turge-scale dataset. Nature Communications, 14(1):3741, 2023.
K., Lekakis, J., McGregor, K., Metra, M., Morais, J.,
10

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Li, Z., Zhang, X., and Razaviyayn, M. Addax: Memory- efficient fine-tuning of language models with combina- tion of 5th Workshop on practical ML limited/low resource settings, 2024.
Sepahvand, M. and Abdali-Mohammadi, F. novel method for reducing arrhythmia classification from 12-lead ECG signals to single-lead ECG with minimal loss of accuracy through teacher-student knowledge distillation. Informa- tion Sciences, 593:64-77, 2022.
and forward-only passes. In
Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes, Advances in Neural Information Processing Systems, 36:53038 53075, 2023.
Sharif Razavian, A., Azizpour, H., Sullivan, J., and Carlsson, S. CNN features off- the--hh--hhlf an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813, 2014.
Mathew, G., Barbosa, D., Prince, J., and Venkatraman, S. Foundation models for cardiovascular disease detection via biosignals from digital stethoscopes. upj Cardiovas- cular Health, 1(1):25, 2024.
Spall, J. C. Multivariate stochastic approximation using simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332-3411 1992.
Mc Namara, K., Alzubaidi, H., and Jackson, J. K. Car- diovascular disease as a leading cause of death: how are pharmacists getting involved? Integrated pharmacy research and practice, pp. 1-11, 2019.
Strodthoff, N., Wagner, Schaeffter, T., and Samek, W. Deep learning for ECG analysis: Benchmarks and in- sights from PTB-XI. IEEE Journal of Biomedical and Health Informatics, 25(5): 2020.
McKeen, K., Oliva, L., Masood, S., Toma, A., Rubin, B., and Wang, B. ECG-FM: An open electrocardiggram foun- dation model. arXiv preprint arXiv:2408.0511 051888 2024.
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B.,, and Liang, J. Convolu- tional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions medical imaging, 3595):1299-1312 2016.
Nesterov, Y. and Spokoiny, V. Random gradient-free mini- mization of convex functions. Foundations of Computa- tional Mathematics, 17(2):527-566, 2017.
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokaë, M., Chaunzwa, T. I.,, Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. Foundation model for cancer imaging biomarkers. Nature machine intelligence, 6(3):354-367, 2024.
Vaid, A., Jiang, J., Sawant, A., Lerakis, S., Argulian, E., Ahuja, Y., Lampert, J.,, Charney, A., Greenspan, H., Narula, J., et al. A foundational vision transformer im- proves diagnostic performance for electrocardiograms. NPJ Digital Medicine, 6(1):108, 2023.
Pham, M., Saeed, A., and Ma, D. C-MELT: Contrastive enhanced masked auto-encoders for ECG-language pre- training. arXiv preprint a:Xiv22410002131, 2024.
Wagner, P., Strodthoff, N., Bousseljot, R.-D., Kreiseler, D., Lunze, F. I., Samek, W., and Schaeffter, T. PTB-XL, a large publicly available electrocardiography dataset. Scientific data, 7(1):544, 2020.
Reyna, M., Sadr, N., Gu, A., Perez Alday, E. A., Liu, C., Seyedi, S., Shah, A., and Clifford, G. Will two do? varying dimensions in electrocardiography: The phys- ionet/computing in cardiology challenge 2021. PhysioNet, 2022. doi: 10. 10.13026/34va-700
Wang, X., Zhao, J., Marostica, E., Yuan, W., Jin, J., Zhang, J., Li, R., Tang, H., Wang, K., Li, Y., et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, pp. 1-9, 2024.
Ribeiro, A. H., Ribeiro, M. H., Paixão, G. M., Oliveira, D. M., Gomes, P. R., Canazart, A., Ferreira, M. P., Andersson, C. R., Macfarlane, P. W., Meira Jr, W., al. Automatic diagnosis of the 12-lead ECG using deep neural network. Nature communications, 11(1):1760, 2020.
Zhang, Y., Li, P., Hong, J., Li, J., Zhang, Y., Zheng, W., Chen, P.-Y., Lee, J. D., Yin, W., Hong, M.. Wang, Z., Liu, S., and Chen, T. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Forty-first International Conference on Machine Learn- ing, 2024.
Ribeiro, A. L. P., Paixao, G. M., Gomes, P. R., Ribeiro, M. H., Ribeiro, A. H., Canazart, J. A., Oliveira, D. M., Ferreira, M. P., Lima, E. M., de Moraes, J. L., et al. Tele-electrocardiograapl. and bigdata: the code (clinical outcomes in digital electrocardiography) study. Journal of electrocardiology. 57:575-S78 2019.
Zheng, J., Chu, H., Struppa, D., Zhang, J., Yacoub, S. M., El- Askary, H., Chang, A., Ehwerhemuepha, L., Abudayyeh, I., Barrett, A., et al. Optimal multi-stage arrhythmia classification approach. Scientific reports, 10(1):2898, 2020a.

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Zheng, J., Zhang, J., Danioko, S., Yao, H., Guo, H., and Rakovski, C. A 12-lead electrocardiogram database for arrhythmia research covering more than 10,000 patients. Scientific data, 7(1):88, 2020b.
Zhou, R., Clifton, L., Liu, Z., Chan, K. W. Y., Clifton,
D. A., Zhang, Y.-T., and Dong, Y. CE-SSL: Computation- efficient semi-supervised learning for ECG-based cardio- vascular diseases detection. 2024.
Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S.,
Williamson, D. J., Struyven, R. R., Liu, T., Xu, M., Lozano, M. G.. Woodward-Court, P., et al. A foundation model for generalizable disease detection from retinal 2023.
images. Nature,
12

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
A. Appendix.
A.1. Proof of Eq.(6)
Consider model with trainable parameter and a batch size of N, we can calculate its gradients using gradient backpropa- gation and loss function as,
N
VL((;0) Σ
(12)
where 24, Yr are ECG sample and its label within the mini-batch B. The computational costs of gradient backpropagation expensive as the model scales up. To address this, we can use the zeroth-order method to estimate the gradient as
1 N
N n Σ
L(z,, V6: 0 + p22z) L(rr,, y65 j1zz) 2j-
C(
(13)
2/1
where Id is a random vector sampled from the standard Gaussian distribution (0,, Ia), μ the perturbation scale and is the number of function queries. As shown in Malladi al. (2023),
E [
d n - E VLL(:
(14)
where d is the number of trainable parameters in the model. In study, set ensure low latency in calculating the zeroth-order gradients, which yields that,
d(B E [ 0)] dE | -
+ [B
d
(E [[V 81: (CC
N
N1 (E
[[V (VV 01: -V]00
Ni N
dNi 1
E [ 0) [V
(15)
N N1
dN1
N
dN
[IVVC(B:;
EV
|V
N
d.,,
[IV
N
where [(0) is the true gradient of 0 and B, (a,, ys M is the tiny mini-batch for the proposed mix-order optimization (N1 < N).
A.2. Algorithm of H-Tuning
The algorithm of the proposed H-Tuning is presented in Algorithm
A.3. Implementation Details
This section introduces the details of the H-Tuning algorithm and the knowledge distillation process. We use the pre-trained model provided by Zhou et al. (2024) for ECG-based CVDs detection on the downstream dataset. It has 50.494 million parameters and is pre-trained the Clinical Outcomes in Digital Electrocardiology (CODE) dataset (Ribeiro et al., 2019; 2020). The backbone consists of three convolution blocks, twelve self-attention blocks, and one classification block. The number of convolution channels within the convolution blocks and the hidden layer dimension control the number of trainable parameters. We fine-tuned the pre- traitrd model on downstream datasets using the proposed H-Tuning method. The fine-tuned model acts as the teacher model during the knowledge distillation process. this study, the student model shares similar architecture with the teacher model but only has 0.26 million trainable parameters. Specifically, the
13

---

H-Tuning:
Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Algorithm 1 The H-Tuning algorithm
Require:
Learning rate 17, Batch sizes N; Batch sizes of the tiny subset NN. The trainable parameters of the large pre-trained model #; À for the gradient refinmment process.
Labeled dataset D= {X, Y);
Ensure: Fine-tuned large model with the updated parameters 0r;
1: 01 =0
2: for T do
3: sample a random batch B {z,, Me J from D;
4: apply data augmentation to B:
5: sample tiny subset B1 { y | from B;
6: Simultaneous Perturbation Stochastic Approximation (SPSA)
7: Based on (13( generate an initial estimation of the parameter gradients (L)(B)))
8: Gradient Refinement
9:
Conduct the gradient backpropagation on the tiny subset B, to calculate VL((;,;
10:
Calculate A (B: x using CBB; 0z), V ( and the (5.(
11: Parameter Optimization
12: Conduct the gradient descent process to update the parameters O, using (q.(
apply an early-stop strategy to avoid overfitting,
14:
end for
convolution channels and the hidden layer dimension of the teacher model are 512, while the number is 64 for the student model. Specifically, Adam optimizer is utilized to conduct the gradient descent process defined in (q,, with learning rate of 1 = 0.002. The batch size N and N, defined in the proposed mix-order optimization are set to 128 and 2, respectively. Additionally, the controlling weight À is searched within a set of (0.85, 0.90, 0.95, 0.99). The perturbation scale μ for the SPSA process is searched within a set of {0.001, 0.0001 and the number of queries n is set to The rank r of the low-rank adaptation process set to 16, and the number of deep layers M is set 2.
For the comparison methods, the same pre-trained model is used for fine-tuning. Adam optimizer is adopted update the model with learning rate 77 0.002. To implement LP, we freeze all the layers within the model except the last two linear layers. The perturbation scales for MeZO, LoHO, and Addax are searched within set of (0.001, 0. 0.0001}, }, and the number of queries n is set to 1. The rank r of the compared methods with low-rank adaptation process (LoRA, Me70 + LoRA, Addax + LoRA, LoHO + LoRA) is set to 16. The training batch size of the proposed H-Tuning and the compared methods set to 128. All the experiments are conducted a single NVIDIA A6000 graphics processing unit using the Pytorch library.
A.4. Extended Experimental Results on More Evaluation Metries.
(1) Detailed comparisons of H-Tuning and the baseline models on more evaluation metrics (Table 5). Apart from macro AUC and macro FØ=2, four metrics on multi-label classification are used evaluate the performance of different fine-tuning methods: ranking loss, coverage, mean average precision (MAP), macro G3=2- For the ranking loss and coverage, a lower value indicates a better detection performance. However, for the MAP, macro AUC, macro Gy=2, and macro F the greater the better. We also report the number of trainable parameters (Params) of different methods to measure their storage consumption during fine-tuning. The higher the number of trainable parameters, the higher the parameter storage consumption. The average time for each iteration during fine-tuning is also included for comparison (Time/iter). Experimental results on six evaluation metrics demonstrate the superiority of H-Tuning against other memory-efficient fine-tuning methods in CVDs detection. For instance, H- H--Tnning demonstrates an average MAP of 0.535 across four datasets, outperforming the best competitor (Addax) by 2.8%. Additionally, it can be observed that H- H-Tuning achieves comparable performance to fine-tuning methods with the first-order optimization (Full FT and LoRA) on three datasets but fails to achieve this in the Ningbo dataset. This phenomenon indicates that its performance might not be stable in certain datasets, which deserves further investigation in future works. As for the computational costs, H-Tuning consumes significantly less GPU memory than Full FT and LoRA. Similar to LoRA(Hu et al., 2022), it can fine-tune pre-trained models in a 23.7 times compared with the Full FT. One limitation that H- H-Tuning cannot accelerate the fine-tuning process.
manner, greatly decreasing the parameter storage costs by
14

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
(2) Detailed ablation studies of H-Tuning on more evaluation metrics for multi-label CVDs detection (Table 6). can be observed that removing gradient estimation, refinement, normalization module from the H-Tuning results in performance degradations on all datasets.
(3) Detailed CVDs detection performance under different ECG lead configurations on more evaluation metrics for multi-label CVDs detection (Table 7). Under all BCG lead configurations, the knowledge distillation process increases the CVDs detection performance of the student models on all evaluation metrics. In Fig.4, visualize the AUC of different student models each CVDs from the four datasets. The CVDs analyzed in our study and their abbreviations can be found in Table 8. For example, the models with 1-lead ECG perform well in detecting atrial fibrillation (AF) but poorly in recognizing left axis deviation (LAD). This because 1-lead ECG (lead I) has the ability to capture the patterns of determining AF, such as irregular RR intervals and the absence of P waves(Fuster et al., 2006). However, as the critical pattern LAD is abnormal QRS complexes in lead and lead aVKKasoo et al., 2023), not visible in 1-lead ECG (lead I).
A.5. Performance Comparisons of Different Models under Limited Supervision
In the domain of medical intelligence, collecting labeled labels expensive and time-consuming, limiting the sample size of the datasets for model fine-tuning. Therefore, evaluating the performance of different fine-tuning methods under minimal supervision is important, it provides stress test for their robustness in clinical practices. For the four datasets, we decrease the ratio between the training set and the held-out test set from 1:9 to 0.5:9.5, and present the average CVDs detection performance of different fine-tuning methods across the four datasets in Fig.5. For simplicity, we visualize the methods with the top eight performances. It can be observed that the superiority of H-Tuning compared with other memory-efficien methods persists under limited supervision. For instance, H-Tuning achieves an average MAP 0.503 across the four datasets, outperforming Addax by 3.03%. More importantly, outperforms the Full FT ranking loss, coverage, and macro AUC. Besides, its performance losses compared with Full FT are within 1.5% for the remaining evaluation metries, As shown in Fig.6, utilize paired t-test to evaluate whether the performance differences between H- H-Tuiing and all compared methods are significant. The results demonstrate that H- Tu-inng significantly outperforms other memory-cfficiin methods on six evaluation metrics, such as Addax. Additionally the performance differences between H- Tuning and memory-inefficier methods (Full FT, LoRA) are not significant on five metrics. These phenomena indicate the robustness of the proposed H- Tuninng framework under different supervision levels, highlighting its effectiveness in clinical practices.
15

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Table Performance of H-Tuning and the compared models four public datasets. The average performance and the standard deviation across four seeds are presented. The dashed lines separate the memory-efficient fine-tuning methods and the traditional fine-tuming fine-tuning methods.
Methods
Params
Memory
Timefiter
Ranking Loss |
Coverage
Macro AUC
MAP t
Macro G3=2
Macro Fa=2
G12EC Dataset
Full FT
50.493 M
9.214 GB
0.401
0.086
3.657_10.113
0.869110.066
0.51710.012
0.336.10.009
0.58810000
LoRA
2.135 M
8.754 GB
0.428
0.084.10.00
3.62110.134
0.87010.009
0.522 0.008
0.336.10.007
0.592_0.001
0,272
1.416
0.152
0.005=+0.044
3.837=00086
0.852++0.003
0.483=0.002
0.306=000004
0.545=00005
McZO
50.493 M
1.815 GB
0.339 s
0.513=00.050
11.325==0.811
0.532=+0.047
0.133=0.026
0.111
0.327=00012
MeZO + LoRA
LoRA
2.135 M
1.437 GB
0.386
0.409110.010
10.761.10.187
0.507
0.115100.09
0.10310.004
0.316110.009
Addax
50.493 M
2.000 GB
0.349
0.086.10.006
3.664.10.130
0.8631000006
0.497
0.31910.008
0.573110.008
Addax LoRA
2.135
1.453 GB
0.403 s
0.093.10.008
3.786110.166
0.857110.005
0.408
0.312110009
0,568110.006
LoHO
50.493 M
2.002 GB
0.303 8
0.098:10.004
3.877=20.098
0.851+000003
0.484=0.003
0.309++00004
0.554=0.009
LoHO + LORA
2.135 M
1.453 GB
0.341 8
0.095=+0.002
3.825=:0.029
0.851+00.002
0.480=0.002
0.008
0.557=00.009
H-Tuning
2.135 M
1.453 GB
0.409
0.085:t0.004
3.612:10.069
0.070
0.006=00006
0.330++0.005
0.586+000010
PTB-XI. Dataset
Full FT
50.494 M
9.212 GB
0.413 5
0.026::0.002
2.530=10.069
0.919::0.002
0.008
0.387+00006
0.618=0.011
LoRA
2.135 M
8.754 GB
0.442
0.026110.001
2.537-10.057
0.919110.006
0.008
0.385100006
0.61810.006
LP
0.272 M
1.46 GB
0.155ss
0.032-E0.001
2.721=+0.025
0.897+0.002
0.509=0.001
0.35410.003
0.58240.006
McZO
50.494 M
1.814 GB
0.347 s
0.368:+0113
9.890=22.169
0.515=+0.021
0.116=0.023
0.104=0.014
0.241=0.024
McZ0 + LoRA
2.135 M
1.444 GB
0.394 s
0.398=00.088
1..65
0.483=+0.024
0.104=0.013
0.096+0.006
0.227
Addax
50.494 M
2.003 GB
0.358
0.031100.005
2.677.10.163
0.898100.009
0.51010.014
0.35410.010
00,015
Addax LoRA
2.135 M
1.451 GB
0.408
0.029110.001
2.628
0.89910.0004
0.505
0.36110.006
0.58210.007
LotIO
50.494
2.000 GB
0.311
0.031110.001
2.67910.027
0.902110.02
0.51710.002
0.36010.003
0.588
LoHO + LORA
2.135
1.453 GB
0.355 s
0.032++0.001
2.700=10.038
0.900++00001
0.509=0.004
0.352+0.007
0.581
H H-Tuning
2.135 M
1.453 GB
0.412 8
0.025=+0.001
2.470-10.034
0.923+00.03
0.502=0.004
0.393++0000
0.628=0.002
Ningho Dataset
Full
50.496 M
9,211 GB
0.428 s
0.02711.001
2.639-10.034
0.9334100022
0.541=0.005
0.36611.005
0.591=0.004
LoRA
2.137 M
8.754 GB
0.457
0.027+100001
2.661:100335
0.934:10.02
0.536-0.006
0.355-+0.006
0.58010.008
LF
0.274 M
GB
0.172s
0..04410.001
2.899--0.040
0.906100001
0.462-0.002
0.314100006
0.530-10.008
McZO
50.496
1.814 GB
0.359
0.509-0.006
13.938:+1.235
0.504:1.031
0.091=0.018
0.086:0.012
0.231=0.011
McZO + LoRA
2.137
1.444 GB
0.417s
0.489-10.094
13.560++2.056
0.530:10.018
0.085-0.006
0.08240.006
0.234=0.014
Addax
50.496 M
2.003 GB
0.370
0.041110.010
3.155
0.909110.018
0.443.1.004
0.30310.019
0,509
Addax LoRA
2.137 M
1.451 GB
0.425
0.040.10.005
3.128.10.188
0,901
0.42611.0029
0.29310.014
0.4951100022
LoTIO
50.496 M
2.001 GB
0.328
0.034.10.001
2.916.10.030
0.907
0.466_10.00
0.316110.066
0.535110.01
Lol1O + LORA
2.137
1.453 GB
0.369 s
0.034.10.011
2.8751
0.90910.001
0.463
0.320
0.53510.004
H-Tuning
2.137
1.453 GB
0.429
0.028:10.001
2.689-20.039
0.931+0.003
0.497=0.010
0.33340.006
0.550=0.006
Chapman Dataset
Full FT
50.492 M
9.211 GB
0.363
0.032.10.002
2.245:10.057
0.930-10.005
0.587=0.006
0.417:10.008
0.623-10.08
LoRA
2.134
8.754 GB
0.394 s
0.02910.001
2.189-10.036
0.932:1.003
0.596-0.003
0.428:10.002
0.636-10.005
L.
0.271 M
L416 GB
0.140ss
0.038-F0.002
2.352++0.023
0.898++0.008
0.530=0.003
0.381+-00007
0.579-10.018
McZO
50.492 M
1.814 GB
0
0,388
7.951=1.338
0.499110.04
0.114=0.019
0.107+0.011
0.277-200114
McZO LoRA
2.134
1.444 GB
0.361
0.518:10.022
9.926=11.011
0.481:100009
0.129=0.020
0.115+00009
0.278-000044
Addax
50.492 M
2.004 GB
0.324 s
0.001
2.216:10.112
0.929:10.005
0.579
0.39810.006
0.596+0.007
Addax LoRA
2.134
1.451 GB
0.368 s
0.034:10.007
2.302-10.202
0.907-10.025
0.529=0.042
0.091
0.588
LoHO
50.492 M
1.998 GB
0.280
0.036:10.001
2.309:10.029
0.00020..02
0.005
0.37410.006
0.574100013
LoT1O LORA
2.134 M
1.453 GB
0.313
0.035:10.001
2.303:10.021
0.904
0.000
0.382-10.002
0.580-10.005
H-Tuning
2.134
1.453 GB
0.381
0.030=:0.002
2.195=0.034
0.929:00002
0.586=0.011
0.420+000005
0.634=000009
16

---

H- H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Table 6. Detailed ablation study of H-Tuning.
Methods
Memory
Time/tter
Ranking Loss + Coverage
Macro AUC
MAP1
Macro G3=2 f Macro F3=2
G12EC Dataset
Without SPSA gradient estimation
1.451 GB
0.388
0.090:20.002
3.760:1:0067
0.866:t0.002
0.500=0.0044
0.324+000006
0.575=10.009
Without gradient refinement
1,453
0.341
0.095:10.002
3.825:10.029
0.851100002
0.480-10.002
0.308110.003
0.557-10.009
Without gradient normalization
1,453 GB
0.395
0.091110.005
3.715110061
0,870.10.002
0.504_10.005
0.328110.005
0,588
Without low-rank adaptation
2.002
0.096.10.003
3.900.10.088
0.856110.005
0.492_1.0006
0.3181000006
0.56410.012
H-Tuning
1.453 GB
0.409
0.085=000044
3.612:#0.069
0.870=0.002
0.506=0.006
0.330+0.0005
0.586=0.011
PTB-XL Dataset
Without SPSA gradient estimation
1.451
0.394
0.027:10.002
2.5311:0.078
0.919110005
0.542:00009
0.3791100009
0.6161000100
Without gradient re fefinmmett
1.453 GB
0.355
0.032:10.001
2.700:10.038
0.900:0.0011
0.509=1.004
0.352:10.007
0.501
Without gradient normalization
1.453
0.403
0.027110000
2.5531:.004
0.9171:00003
0.537=0.006
0.37640.005
0.609
Without low-rank adaptation
2.000 GB
0.368
0.032:10.004
2.727=10.400
0.907110.006
0.526-0.013
0.364110.010
0.5881100115
H-Tuning
1,453
0.412
0.025:10.01
2.470110.334
0.923-10.003
0.552-0.004
0,393310.003
0.628-0.002
Ningbo Dataset
Without SPSA gradient estimation
1.451 GB
0.406
0.034:20.033
2.910=:0115
0,921#0.013
0.469=0.023
0.318+0.006
0.525=0.011
Without gradient refinement
1.453
0.369
0.034=0.001
2.875=0.016
0.909#0.001
0.463=0.002
0.321
0.535=0.004
Without gradient mormalization
1.453 GB
0.420
0.033:10.002
2.859110.062
0,924:10.004
0.478110013
0.322:10.013
0.536110.014
Without low-rank adaptation
2.000
0.381
0.035.10.003
2.9711:0.144
0.9241100004
0.474:10004
0.31610.0010
0.52711.017
H-Tuning
1.453 GB
0.429
0.028:10.001
2.689-10039
0.93110.003
0.497=0.010
0.333-10.006
0.550=0.006
Chapman Dataset
Without SPSA gradient estimation
1.451 GB
0.357
0.034:10007
2.275::0.145
0.92710.004
0.578-0.013
0.40240.010
0.612-00015
Without gradient refinement
1.453 GB
0.313
0.035:10.001
2.303::0.021
0.904+0.001
0.541=0.003
0.382+0.002
0.580=10.005
Without gradient normalization
1.453
0.369
0.034:2.002
2.300+:0.044
0.928+0.000
0.572=0.010
0.398+0.013
0.596+100021
Without low-rank adaptation
2.001 GB
0.327
0.005.10.004
2.289100082
0.902.10.003
0.56110..11
0.401 10.011
0.603 0.603.10.014
Il-Tuning
1.453
0.381
0.030.10.002
2.195.10.03
0.02920.002
1.001
0.420
0.63410.009
17

---

H- H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Table Detailed CVDs detection performance of the student models with various numbers ECG leads.
Teacher
Student
Ranking Loss $
Coverage
Macro AUC
MAP
Macro Ga=2
Macro F
G12EC Dataset
None
12-Lead
0.091=0.002
3.779=:0.077
0.847-+0.004
0.481+0.011
0.311=200008
0.558=0.003
12-Lead
12-Lead
0.085=0.003
3.627=00.799
0.868=+0.003
0.510=0.007
0.335=10002
0.582==0.005
None
3-Lead
0.102:+0.003
4.050=00.087
0.834:50.003
0.465
0.295=10.005
0.537=00005
12-Lead
3-Lead
0.088=10.002
3.712:20.046
0.860++0.002
0.497:000004
0.325=:0.003
0.574=00.008
None
I-Lead
0.159::0.015
5.120=00.12
0.772++0.011
0.351
0.225:10.013
0.457=000088
12-Lead
1-Lead
0.133.10.009
4.68510.194
0.795:10.011
0.399110.012
0.260.10.009
0.495.10.012
Teacher's performance
0.085=000004
3.612+00.069
0.870
0.506
0.330:+0.005
0.586=0.0010
PTB-XL Dataset
None
12-Lead
0.029110001
2.620.1..032
0,903 10.004
0.507
0.360.10.008
0.592100.14
12-Lead
12-Lead
0.004.1..000
1..016
0.901000.03
0.55910..055
0.394 0.003
0.628110.009
None
3-Lcad
0.032:10.001
2.721
0.886=+0.007
0.496:10.006
0.336:+0.004
0.559=000115
12-Lead
3-Lead
0.027=100011
2.570:10.020
0.905
0.5281100004
0.372=#00003
0.596
None
I-Lead
0.055=t0.003
3.329=t0.079
0.830=+00004
0,381 #00005
0.240=t0.007
0.462=+0.009
12-L.ad
I-Lead
0.048:10.002
3.175:000066
0.843:t0.007
0.431
0.285++0.002
0.509:10.004
Teacher's performance
0.025:10.001
2.470.10.034
0.923-10.003
0.552-10.004
0.393-10.003
0.628-10.02
Ningbo Dataset
None
12-Lead
0.030:10.001
2.781
0.921
0.485=0.0003
0.323+400022
0.530=10.004
12-Lcad
12-Lcad
0.026=000011
2.623::0.047
0,937=t0.002
0.520=:0.007
0.350=
0.568=0.010
None
3-Lead
0.036.10.001
2.987:10.037
0,903.10.001
0.450.10.008
0.300.10.008
0.500.10.009
12-Lead
3-Lead
0.02911.000
2.756110.022
0.928:10.001
0.495
0.337:10.000
0.54910.006
None
1-Lead
0.057=10003
3.630:10.124
0,844:10.000
0,340=10.004
0.234+100003
0.420=10.00
12-Lead
1-Lead
0.050:10.001
3.501-10.037
0.866-40.003
0.305
0.268-100004
0.469-10.007
Teacher's performance
0.028:10.001
2.689-10.039
0.9000
0.4971000100
0.333-10.006
0.550-10.006
Chapman Dataset
None
12-Lead
0.029-10.011
2.176:10.021
0.927-10.002
0.574:10.004
0.410:10.011
0.617-10.008
12-Lead
12-Lead
0.030:10.001
2.197=0.031
0.933++0.003
0.587+0.003
0.423-0.009
0.632+00000
None
3-Lead
0.037+0.001
2.356=40.041
0.911
0.532+0.011
0.381
0.577=0.020
12-Lead
3-Lead
0.033-10.001
2.283
0,919-10.003
10.000
0..004
0.614.10001
None
I-Lead
0.053:10.004
2.677:10.124
0.847:10.007
0.441110.010
0.308:10.006
0.498:10.011
12-Lead
I-Lead
0.054:10.002
2.779:10.045
0.862-10.004
0..018
0.346:10.004
0.538-10.001
Teacher's performance
0.030:10.002
2.195:10.034
0.929:t0.002
0.586:10.011
0.420:10.005
0.634:10.009
18

---

H- H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Table 8. Description of the cardiovascular diseases analyzed in our study.
Original annotation
Abbreviations Original annotation
Abbreviations
G12EC Dataset
atrial fibrillation
AF
1st degree av block left axis deviation
TAVB
incomplete right bundle branch block
IRBBB
LAD
left bundle branch block
LBBB
low qrs voltages sinus rhythm
LQRSV
nonspecific intraventricular conduction disorder premature atrial contraction
NSIVCB
NSR
PAC
prolonged qi interval
LQT
qwave abnormal
QAb
right bundle branch block
RBBB
sinus arrhythmia
SA
sinus bradycardia
SB
sinus tachyeardia inversion
STach
abnormal
TAb
ventricular premature beats
VPB
PTB-XL Dataset
atrial fibrillation
AF
complete right bundle branch block
CRBBB
1st degree av block
LAVB
incomplete right bundle branch block
IRBBB
left axis deviation
LAD
left anterior fascicular block
LAnFB
left bundle branch block
LBBB
nonspecific intraventricular conduction disorder
NSIVCB
simus rhythm
NSR
premature atrial contraction
PAC
pacing rhythm
PR
prolonged interval
LPR
qwave abnormal
QAb
right axis deviation
RAD
sinus arrhythmia
SA
sinus bradycardia
SB
sinus tachycurdia inversion
STach
abnormal
TAb
Tlav
Ningbo Dataset
atrial flutter
AFL
bundle branch block complete right bundle branch block
BBB
complete left bundle branch block
CLBBB
CRBBB
1st degree av block
LAVB
incomplete right bundle branch block
IRBBB
left axis deviation
LAD
left anterior fascicular block
LAnFB
low qrs voltages
LQRSV
nonspecific intraventricular conduction disorder
NSIVCB
sinus rhythm
NSR
premature atrial cuntraction
PAC
pacing rhythm
PR
poor R wave Progression
PRWP
premature ventricular contractions
PVC
prolonged interval
LQT
qwave abnormal
QAb
right axis deviation
RAD
sinus arrhythmia
SA
sinus bradycardia
SB
sinus tachycardia inversion
STach
wave abnornal
TAb
Thnv
Chapman Dataset
atrial fibrillation
atrial flutter
AFL
1st degree av block
TAVB
left axis deviation
LAD
left bundle branch block
LBBB
low qrs voltages
LQRSV
nonspecific intraventricular conduction disorder
NSIVCB
sinus rhythm
NSR
premature atrial contraction
PAC
qwave abnormal
QAb
right axis deviation
RAD
right bundle branch block
RBBB
sinus hradycardia
SB
sinus tachycardia
STach
wave abnormal
TAb
ventricular premature beats
VPB
19

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
G12EC Dataset
PTB-XL Dataset
LQRSV LBBB
LAnFB
LAD
LBBB
NSIVCB
LAD
IRBBB
NSIVCB
NSR
IRBBB
IAVB
NSR
PAC
IABB
CRBBB
PAC
LQT
AF
AF
0.5
0.5
PR
0.6
0.6
QAb
0.7
VPB
0.7
Tinv
0.8
LPR
0.8
RBBB
0.9
Tinv
0.9
TAb
QAb
SA
TAb
STach
RAD
SB
STach
SA
SB
Ningbo Dataset
Chapman Dataset
LAD IRBBB
LBBB
LAnFB
IAVB
LQRSV
LAD
LQRSV
CRBBB
NSIVCB
IAVB
NSIVCR
CLBBB
NSR
NSR
AFL
BBB
PAC
AFL
PAC
0.5
0.5
PR
0.6
0.6
Tinv
0.7
PRWP
QAb
0.7
VPB
0.8
TAb
0.8
0.9
PVC
0.9
STach
RAD
TAb
LQT
SB
QAb
RAD
SA
RBBB
SB
STach
Single-Lead
Three-Lead
Twelve-Lead
Figure AUC different student models on various CVDs. The green lines denote student model trained with 12-lead ECG signals, the orange dashed-dotts lines denote the student model trained with 3-lead ECG signals, and the blue dashed lines denote the student model trained with 1-lead ECG signals.
20

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
Ranking Loss
Coverage
Macro AUC
0.910
0.058
3.194
0.056
0.056
3.150
0.898
0.8940.894
0.0550 055
3.124
3.103 3.10930098
0.880
0.054
0.053
3.100
3.067
0.880
0.880
3.050
0.875 0,875
0.052
3,010
0.870
3,000
2.975
0.050
0.0500.050
0.049
2.950
2.956
0.860
2.800
0.860
MAP
Macro GB
Macro Fp-2
0.520
0.516
0.336
0.580
0.552
0.505 0.503
0.390
0.3290.329
0. 0.550
0.5 0.544
0.500
0. 3.540
0.320
0
0,400
0.4700.470
0.467
0.463
0.310
0.311
0.308
0.3060.306
0.520
0.5190.516
0.516
0.513
0,460
0.510
0.443
0.294
0.500
0.440
0.290
0.490
Addax LoRA
LoHO
LoHO LoRA
Addax
Full FT
LoRA
H-Tuning
Figure 5. CVDs detection performance of different models under very limited supervision (training data: testing data 0.5 9.5)..
p 0.01
0.05
0.05
training data: testing data 1:9
training data: testing data 0.5:9.5
LoHO LoRA
LoHO LoRA
LoHO
LoHO
Addax LoRA
Addax LoRA
Addax
Addax
MeZO LoRA
MeZO LoRA
MeZO
MeZO
LP
LoRA
LoRA
Full FT
Full
Ranking Loss
Coverage
AUC
MAP
Macro Gp
Macro F
Ranking Loss
Coverage
AUC
MAP
Macro Gpa
Macro F
Figure 6. Paired t-test results for the model performance under diffcrent supervision levels. We check if the averaged performance differences on four datasets between H-Tuning the compared methods are significant. Each circle indicates t-test result between H- Tuning and compared method after false discovery rate correction. The circle colors denote different significant levels. The black dashed lines separate memory-efficient and memory-inefficien fine-tuning methods.
21

---

H-Tuning: Toward Low-Cost and Efficient ECG-based Cardiovascular Disease Detection with Pre-Trained Models
raw signal
preprocessed signal
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Time (s)
Figure Visualization of raw and the preprocessed ECG.
22


Reviews:
[{"review_id": "Rl2951n2uS", "reviewer": "Reviewer_ChMh", "summary": "The authors propose H-tuning, a model pipeline for efficiently fine-tuning pre-trained models for ECG classification to enable cardiac diagnosis under limited computation resources. \nThey combine zeroth- order backpropagation, low-rank adaptation, and knowledge distillation to reduce computation times and memory requirements at fine-tuning and inference time. They demonstrate superior runtime, memory footprint, and classification performances for numerous related methods considering four public datasets for evaluation. Design choices are justified with thorough ablation studies.\n\n## update after rebuttal: I am happy with the authors responses and believe this submission to be worthy of publication. I am happy to change my score to reflect this.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for all your questions and suggestions.\n- **Methods And Evaluation Criteria:** To assess the generalizability of our classifiers in mobile cardiac healthcare, an external validation set consisting of 7000 wearable 12-lead ECG signals, provided by [1], is used for testing. We first utilized the G12EC, PTB-XL, Ningbo, and Chapman datasets for downstream training, where only 10% of labeled ECG signals are used to fine-tune the backbone. We employ two fine-tuning methods (Full FT and LoRA) and H-Tuning to train three teacher models, followed by knowledge distillation to create three corresponding student models. The CVD detection performance of the six classifiers on the external dataset is shown in Table R1. \n\n**Table R1 External validation on a wearable ECG dataset**\n|Methods|Macro AUC|Macro $F_{\\beta=2}$|\n|-|-|-|\n|**Teacher Models**|\n|Full FT|0.870|0.570|\n|LoRA|0.879|0.579|\n|H-Tuning |0.866|0.567|\n|**Student Models**|\n|Full FT|0.867|0.534|\n|LoRA|0.874|0.543|\n|H-Tuning|0.880|0.551|\n\nThe results demonstrate that the teacher generated by H-Tuning achieves comparable performance to Full FT and LoRA, but with significantly less GPU memory consumption, as shown in our manuscript. Additionally, our student model performs better than the compared methods. \n\nWe cannot compare our classifiers with the winning challenge team on the Physionet 2020/2021 test datasets, which are not publicly available. However, we can report the performance of H-Tuning using the model designed by the winning team (See our response to Reviewer m5NF, Table R4, reference [2]).\n\n[1] Lai, J., etal. Practical intelligent diagnostic algorithm for wearable 12-lead ECG via self-supervised learning on large-scale dataset. Nature Communications, 14(1), 3741.\n\n- **Experimental Designs Or Analyses:**  Please refer to the above section.\n\n- **Relation To Broader Scientific Literature:** (1) We need to clarify that the proposed H-Tuning and the compared SOTA methods in our manuscript are all fine-tuning methods, which do not utilize unlabeled data for semi-supervised or self-supervised learning. In addition, the experiments presented in our manuscript are all conducted on the downstream datasets. External validation was performed on a wearable 12-lead ECG dataset (Table R1). (2) Comparisons between H-Tuning and CE-SSL. Please see *Questions For Authors:*.  \n\n- **Essential References Not Discussed:** We thank the reviewer for this comment. The downstream datasets used in our study were also included in the PhysioNet challenge. We will add this reference in next version of our manuscript.\n\n- **Other Comments Or Suggestions**:\nWe are grateful for your valuable suggestions. We will correct the issues in the next version of our manuscript.\n\n - **Questions For Authors:**\n   (1)  LoHO applied first-order optimization and zeroth-order optimization on different trainable parameters. It did not explore how to utilize their advantages in a more flexible way to fully optimize all parameters.  Addax and the proposed mix-order optimization both utilize first-order gradients to refine the direction of the zeroth-order gradients. However, our mix-order optimization proposed a gradient normalization technique to regulate the norm of the zeroth-order gradients, which has not been explored by Addax. Our ablation studies in section 3.4 demonstrated that the proposed technique has an obvious positive impact on fine-tuning performance. At the same time, Table 1 in our manuscript demonstrated the superior performance of H-Tuning compared with Addax and LoHO.\n   (2) The differences between CE-SSL and H-Tuning can be listed as:\n\n        a. CE-SSL is a semi-supervised method, which utilizes unlabeled data to achieve robust CVDs classification performance. In contrast, H-Tuning is a supervised method focusing on low-cost and efficient fine-tuning and inference.\n        b. CE-SSL utilizes first-order optimization for model training, which needs extensive activation outputs for gradient backpropagation. H-Tuning avoids this drawback by designing a mix-order optimization method, which greatly reduces the GPU memory costs.\n       c. In our study, we integrated H-Tuning with a knowledge distillation technique to reduce the model inference costs on wearable devices, which was not explored by CE-SSL.\n       d. To ensure a fair performance comparison between CE-SSL and H-Tuning, we remove the semi-supervised learning module from the CE-SSL while preserving the other modules, which makes CE-SSL a supervised fine-tuning method. We report their average performance across four downstream datasets (following the data split method and the backbone used in our manuscript).\n**Table R2**\n|Methods|Memory (GB)|MAP|Macro $F_{\\beta=2}$|\n|-|-|-|-|\n| CE-SSL (without unlabeled data)|9.024|0.562|0.616|\n| H-Tuning|1.453|0.535|0.600|\n\nThe results show that H-Tuning performs similarly to CE-SSL with much less GPU memory costs.", "claims_and_evidence": "The claims are convincing as the study introduces a thorough experiment design and evaluation. The proposed method has been evaluated on multiple datasets with convincing performance impact and compared with with multiple related state-of-the-art techniques. Finally the proposed model design choices is supported by various ablation studies.", "methods_and_evaluation_criteria": "The proposed method has not been assessed on external databases, and therefore it is impossible to assess the generalisability of the final classifiers, and that was one of the key challenges in the 2020 PhysioNet Challenge.\n\nMoreover, it would have been interesting to compare the proposed with the approaches proposed by the winning challenge team to compare the models with a strong baseline.", "theoretical_claims": "No", "experimental_designs_or_analyses": "The generalizability of the learned representation has not been performed, testing of the classifier on an external database would have been interesting.\nComapring their appoach with other state-of-the-art techniques (2021 PhysioNet challenge entries) would have been informative", "relation_to_broader_scientific_literature": "It would have been interesting to compare the results with baseline approaches on the downstream tasks, and not only assess how the proposed technique compares with other SSL techniques. \n\nIt would have been also interesting to compare the performance of the proposed technique with CE-SSL (Zhou, R., Liu, Z., Clifton, L., Clifton, D. A., Chan, K. W., Zhang, Y.-T., and Dong, Y. Computation-efficient semisupervised learning for ecg-based cardiovascular diseases detection. arXiv preprint arXiv:2406.14377, 2024.), which has a very similar study design.", "essential_references_not_discussed": "The authors have omitted the literature of past 2020 and 2021 PhysioNet/challenge on rhythm; classification.", "other_strengths_and_weaknesses": "The novelty of the proposed work is the combination of existing state-of-the-art model fine-tuning techniques.", "other_comments_or_suggestions": "Font size of Figure 2 could be increased\nTypo in L224? “Additionally, we tune the deep layers using the proposed mix-order\noptimization method…” -&gt; shallow layers?\nTable 3: Teacher “None” might be confusing before reading the manuscript, add a short\ndescription in the table caption", "questions_for_authors": "Could the authors highlight the differences of your proposed mix-order optimization with Addax and LoHO ?\n\nCould the authors also discuss more in depth the difference and added value of H-tuning compared to CE-SSL (Zhou, R., Liu, Z., Clifton, L., Clifton, D. A., Chan, K. W., Zhang, Y.-T., and Dong, Y. Computation-efficient semisupervised learning for ecg-based cardiovascular diseases detection. arXiv preprint arXiv:2406.14377, 2024.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96surzPJQL", "reviewer": "Reviewer_m5NF", "summary": "This paper aims to detect cardiovascular disease by fine-tuning large-scale pre-trained models using ECG signals. It focuses on low-cost and efficient fine-tuning through a mix-order optimization with low-rank adaptation and a novel layer-dependent model update scheme. Then, a knowledge distillation technique is introduced for smart devices. Experiments on the G12EC, PTB-XL, Ningbo, and Chapman datasets show the proposed method achieves comparable or even better performance with lower time cost and reduced memory consumption.\n\n## update after rebuttal\nI would like to keep the score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for all your questions and suggestions. \n- **Claims**:\n\nThe reasons for choosing the ECG setting: (1) The ECG community offers many open-access datasets for model training and evaluation. (2) Developing low-cost methods for accurate and mobile cardiac healthcare is an important topic in AI/ML for health. We apologize for not being able to extend the experiments to other signals within this timeframe, but we will include this as future work in the next version of the manuscript.\n\n- **Methods**:\n\n **Q3**: A bandpass filter (1-47 Hz) was used to filter out the noise, which is also adopted by [2].\n\n[2]  P. Nejedly et al., “Classification of ECG using ensemble of residual CNNs with attention mechanism,” in Proc. Comput. Cardiol., 2021, pp. 1–4.\n\n **Q4**: An ideal joint framework should not only reduce the computational costs associated with fine-tuning and deploying pre-trained models but also maintain the performance of the fine-tuned models on downstream datasets. Therefore, under the constraint of achieving comparable fine-tuning performance to full fine-tuning, the joint framework proposed in our study includes two subtasks: (1) reducing the fine-tuning cost and (2) reducing the inference cost.\n\nThe unit for Memory is GB. \n\n- **Experimental Designs:** \n\n **Q2**: Yes, a backbone with 50 million parameters can be trained or fine-tuned with high-level GPUs (> 8GB memory). **However, we want to point out that the costs for fine-tuning and inference become prohibitive on low-level devices (with 2-4 GB memory), which are affordable and typically deployed in clinics or home settings. Therefore, the motivation of our research is to maintain model performance under limited resources.**  We validated that H-Tuning achieves similar performance to Full FT while reducing GPU memory costs by 6.34 times, fulfilling our research objective. Additionally, we utilized a knowledge distillation technique to reduce the inference costs of the fine-tuned models, enabling them to be deployed on mobile devices (<20MB memory). \n\n **Q3**:  Motion artifacts often behave as baseline wander. Theoretically, a bandpass filter (1-47Hz) can filter out most of the baseline wander noises (generally below 1Hz). We will add a comparison between the raw and the filtered signals in the next version of the manuscript.\n\n**Q4**:   In the previous manuscript, no dataset was collected by wearable devices. To support our claims, we conducted an external validation of the final classifiers generated by H-Tuning on a wearable 12-lead ECG dataset. Please refer to our response to Reviewer ChMh (Table R1).\n\n**Q5**:  There is no 'standard layer' in the proposed model update scheme. All layers are trainable. Without this scheme, all layers can be optimized with first-order optimization, such as Full FT or LoRA. However, we agree that we should include an ablation study on the low-rank adaptation. The results are shown in Table R3. \n\n**Table R3**\n||Time/iter (s)|Memory (GB)|Macro $F_{\\beta=2}$|\n|-|-|-|-|\n|H-Tuning w/o low-rank adaptation|0.358|2.002|0.571|\n|H-Tuning|0.408|1.453|0.600|\n|Full FT|0.401|9.212|0.605|\n\n**Q6**:   The success of MeZO-based methods in fine-tuning pre-trained models relies on prompts[3]. However, prompts are not feasible in ECG analysis because there are no text inputs, making the MeZO unable to find a stable optimization path. The MeZO paper also reported a performance collapse (30.3%-47.2% loss) in the absence of prompts.\n\n[3]. Malladi, S.etal. (2023). Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36, 53038-53075.\n\n**Q7**: H-Tuning integrates low-rank adaptation to reduce the GPU memory consumption and decrease the variance for gradient estimation, which introduces extra time for the forward process. As shown in Table R3, training time is significantly reduced without this module at the expense of increased memory costs.\n\n- **Supplementary Material**: We will release our code after the publication of our manuscript.\n\n- **Relation To Literature**: The architecture of the backbone used in our study is provided by [4], which validated its effectiveness on various datasets. We also verified that the performance of H-Tuning is comparable to Full FT on another backbone provided by [2].\n\n**Table R4**\n|-|Params|Marco AUC|Macro $F_{\\beta=2}$|\n|-|-|-|-|\n|**H-Tuning with backbone**|-|-|-|\n|in [2]|15.8M|0.902|0.576|\n|in [4]|50.4M|0.913|0.600|\n|**Full FT with backbone**|-|-|-|\n|in [2]|15.8M|0.908|0.590|\n|in [4]|50.4M|0.918|0.605|\n[4] Zhou, R. etal. Computation-efficient semisupervised learning for ecg-based cardiovascular diseases detection. arXiv:2406.14377, 2024.\n\n- **Other Comments:**\n$\\eta$ is the learning rate. $\\mathbb{E}$ denotes the expectation of the zero-order gradients over vector $z$.\n\n- **Note:** \nAll evaluation metrics are averaged across 4 downstream datasets and 4 seeds.", "claims_and_evidence": "1. The proposed method is based on the ECG setting. But is there any reason this framework has to be specific for ECG but not other signals like EEG? It would be better if the authors expanded the experiments for other signals for core AI/ML contribution.", "methods_and_evaluation_criteria": "1. They propose a framework (H-Tuning) developed to integrate mix-order optimization with low-rank adaptation and a novel layer-dependent model update scheme, enhancing both computational efficiency and robustness.\n2. The mix-order optimization provides a low-cost solution and does not require text data.\n3. The study is set in the context of smart devices. However, as I understand it, the data collected by smart devices include a significant amount of noise. Is there any preprocessing applied to remove different types of noise?\n4. Unclear writing:\n- In line 023 of the abstract, what is meant by “a joint framework” and “a holistic method”? Do they mean a framework that includes both fine-tuning and downsampling tasks? What is being joined, and what subtasks should a non-holistic method focus on?\n- In Table 1 and Table 2, what is the unit for Memory? MB? GB?", "theoretical_claims": "The authors claim the effectiveness of the zero-th order optimization and the mixed-order (zero-th order and first order) optimization.", "experimental_designs_or_analyses": "- The authors investigate the performance of the student model under various lead configurations, considering that most mobile ECG devices have only 1–3 leads.\n- The authors claim in the abstract that the computational costs for fine-tuning are unaffordable, however, the backbone in the proposed method only consists of 50 million parameters, which is easy to train from scratch. It would be better if the authors conducted experiments using a larger model, except it cannot support this motivation.\n- The authors claim that for signal preprocessing, a band-pass filter (1–47 Hz) is applied to remove potential noise from the raw ECG recordings, such as power-line interference and motion artifacts (Section 3.1). It would be better if the authors had conducted additional experiments to show that the motion artifacts are removed.\n- The authors claim that the student model meets the requirements for mobile cardiac healthcare. Are any of the datasets used collected by mobile devices? If yes, which dataset? If not, how can it be proven that this method has the potential to be embedded into mobile phones?\n- It would strengthen the authors' argument if the authors included more ablation studies to demonstrate the effectiveness of the Low-Rank Adaptation module and the Model Update Scheme. A comparison between the current results and those obtained by removing the fine-tuning block would support their claims. Additionally, comparing the results with those from a standard layer (not the shallow and deeper layers) could provide further insights.\n- In Table 1, MeZo and MeZo+LoRA only achieved about 0.5 on AUROC, almost making a random decision. It is unusual and worth explaining why.\n- In Table 4, regarding Time/Iter, the efficiency of H-Tuning does not appear to be significantly improved.", "relation_to_broader_scientific_literature": "It would be better if the authors could provide the reason for selecting this particular backbone. Additionally, it would be useful to know whether the proposed framework can also work with other backbones.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Some notions are unclear. \n- In line 156, Equation (2), section 2.1, what does “η” represent? \n- In line 159, Section 2.1, what does “E” represent?", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9hEpNpIAPp", "reviewer": "Reviewer_j6yC", "summary": "This paper proposes H-Tuning, a novel framework that reduces the computational cost of fine-tuning large pre-trained models for ECG-based cardiovascular disease detection by integrating mix-order optimization, low-rank adaptation, and layer-dependent tuning. Additionally, it employs knowledge distillation to transfer knowledge to smaller models, significantly reducing inference costs and enabling efficient deployment on low-resource devices while maintaining high diagnostic performance.\n\n## update after rebuttal\nI'd keep the current score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments on our work. In this rebuttal round, we provide an external validation on a wearable ECG dataset, a more thorough ablation study to support the claims of our study. At the same time, experiments on different backbones strengthen the flexibility of the proposed method (Please refer to our responses to other reviewers).", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The paper provides quantitative experiments on four publicly available ECG datasets, demonstrating that H-Tuning significantly reduces GPU memory consumption (by 6.34×), inference latency (by 19.8×), and the number of model parameters (by 194.2×) while maintaining comparable performance to standard fine-tuning methods. It also includes comparisons against multiple baseline methods, such as Full Fine-Tuning (Full FT), LoRA, MeZO, and Addax, showing superior efficiency and performance. Furthermore, ablation studies confirm the contribution of different components (e.g., mix-order optimization, gradient refinement, and knowledge distillation), while sensitivity analyses demonstrate the robustness of the approach across different hyperparameter settings. Overall, the empirical results strongly support the paper’s core claims regarding computational efficiency and diagnostic performance.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of ECG-based cardiovascular disease (CVD) detection using pre-trained models.", "theoretical_claims": "This paper does not appear to present formal mathematical proofs.", "experimental_designs_or_analyses": "The experimental design is methodologically strong, using diverse datasets, meaningful baselines, and rigorous efficiency metrics.", "relation_to_broader_scientific_literature": "Although this paper focuses on pre-trained ECG models for CVD diseases, the proposed approach has a strong potential to be generalized and be applied to many other fields. Thus it’s worth being discussed by the general audience of ICML.", "essential_references_not_discussed": "Nothing particular.", "other_strengths_and_weaknesses": "Nothing particular.", "other_comments_or_suggestions": "Nothing particular.", "questions_for_authors": "Nothing particular.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received two accepts and one weak reject. While the core contribution of combining existing fine-tuning techniques for pre-trained ECG models may not be highly novel, the approach is well-motivated and shows strong potential for generalization beyond cardiovascular disease applications. The paper is methodologically sound, and its relevance to broader domains makes it a valuable contribution, although it lacks theoretical backup. Although one reviewer expressed limited enthusiasm due to the incremental nature of the work, the overall consensus supports acceptance. The authors are encouraged to clarify the broader impact and potential extensions of their method in the final version.
Decision: Accept (poster)