FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data

## Abstract

Abstract Cardiovascular diseases (CVDs) are currently the leading cause of death worldwide, highlighting the critical need for early diagnosis and treatment. Machine learning (ML) methods can help diagnose CVDs early, but their performance relies on access to substantial data with high quality. However, the sensitive nature of healthcare data often restricts individual clinical institutions from sharing data to train sufficiently generalized and unbiased ML models. Federated Learning (FL) is an emerging approach, which offers a promising solution by enabling collaborative model training across multiple participants without compromising the privacy of the individual data owners. However, to the best of our knowledge, there has been limited prior research applying FL to the cardiovascular disease domain. Moreover, existing FL benchmarks and datasets are typically simulated and may fall short of replicating the complexity of natural heterogeneity found in realistic datasets that challenges current FL algorithms. To address these gaps, this paper presents the first real-world FL benchmark for cardiovascular disease detection, named FedCVD. This benchmark comprises two major tasks: electrocardiogram (ECG) classification and echocardiogram (ECHO) segmentation, based on naturally scattered datasets constructed from the CVD data of seven institutions. Our extensive experiments on these datasets reveal that FL faces new challenges with real-world non-IID and long-tail data. The code and datasets of FedCVD are available https://github.com/SMILELab-FL/FedCVD .

## 1 Introduction

Figure: Figure 1: The overall architecture of the proposed FedCVD benchmark. We present two main settings (Fed-ECG, Fed-ECHO) and an experimental platform, highlighting three primary challenges. Green and blue circles in the challenges section indicate their presence in Fed-ECG and Fed-ECHO, respectively. The API section highlights user-facing APIs in orange boxes.
Refer to caption: x1.png

Cardiovascular Diseases (CVDs) cause over 18 million deaths globally each year, positioning them as one of the most significant global health challenges []. Early detection and accurate diagnosis of CVDs are crucial, as they allow for timely medical interventions and more effective treatment plans, which in turn significantly lower patient mortality rates []. In recent years, with the growing availability of electronic health records and other high-quality clinical data, researchers have increasingly utilized machine learning techniques to automate clinical diagnostics [; ], a strategy that has proven highly effective in the context of CVDs [; ]. This data-driven approach not only facilitates efficient early screening but also optimizes the allocation of healthcare resources, improving overall patient outcomes.

Compared to models trained in isolation at a single center, collaborations across multiple medical institutions enable the utilization of richer regional and demographic characteristics, fostering more precise and comprehensive research outcomes. However, medical data is considered highly sensitive, and recent privacy regulations (e.g., EU General Data Protection Regulation (GDPR) []) restrict its transfer, hindering the expansion of datasets through data sharing among institutions to train more efficient models, i.e., data isolation.

To address this issue, federated learning (FL) [; ] has been proposed as a more secure paradigm of distributed machine learning. A typical FL architecture involves a coordinator (Server) and several participants (Clients) with private data. By aggregating (e.g., FedAvg []) the model parameters or gradients from different Clients on the Server, participants collaboratively train high-performance models keeping private data within their respective domains. This process only involves transmitting model parameters, thus ensuring a certain degree of data privacy.

In medical applications such as CVD, integrating FL enables medical institutions to harness larger and more diverse datasets, collaboratively training models that are more unbiased and generalized, thereby enhancing diagnostic accuracy and clinical decision-making in real-world settings. For instance, [] applied FL for joint case analysis across three institutions during the Covid-19 pandemic, significantly improving CT segmentation performance and facilitating more accurate detection of Covid-19. Additionally, the effectiveness of FL has been demonstrated in multi-center research, such as a study involving three centers focused on the medical image analysis task of whole prostate segmentation [], further underscoring its relevance in realistic, large-scale medical scenarios.

Facilitating the application of FL in multi-center medical research, particularly in areas such as CVD, necessitates the creation of appropriate datasets and benchmarks to support the development of robust algorithms. However, publicly available cardiovascular disease datasets are limited, and those that do exist often suffer from incompatibility due to variations in data collection protocols. Furthermore, there is currently no comprehensive, publicly accessible benchmark specifically designed for FL on CVD data, which significantly impedes research progress in this domain. Additionally, most existing FL benchmarks simulate an FL scenario by manually partitioning data—often without considering geographic distribution—into smaller subsets, resulting in an overly idealized model that fails to capture the complexities and heterogeneity of real-world, multi-center CVD scenarios. This gap presents substantial challenges for the development and validation of effective FL algorithms in practical medical applications.

To address these gaps, we introduce the *first* multi-center FL benchmark specifically designed for CVD tasks, named FedCVD. Built from real-world CVD data collected from seven medical institutions (i.e., clients, the two terms will be used interchangeably), FedCVD utilizes a *natural partitioning* strategy. It comprises two primary datasets along with their corresponding tasks: electrocardiogram (ECG) classification and echocardiogram (ECHO) segmentation. FedCVD encapsulates three critical traits of FL in real-world CVD applications, each of which presents substantial challenges to FL algorithms:

Challenging Trait 1. Non-IID Data: The Non-independently and identically distributed (non-IID) data among institutions, including non-IID feature (e.g., variations in imaging quality due to different equipment across institutions) and non-IID label (e.g., differences in disease prevalence across regions). The non-IID data may significantly hindering global model convergence.

Challenging Trait 2. Long-tail Distribution: The labels of CVD data from various institutions exhibit a long-tailed distribution, where a few labels dominate while most labels are sparse. This challenges the model’s performance on tail classes, a problem that is exacerbated in FL scenarios.

Challenging Trait 3. Label Incompleteness: For the same type of medical images, hospitals with strong annotation capabilities can identify all key segmentation areas, while those with weaker capabilities can identify only some. This incomplete annotation can mislead the global model’s segmentation performance in areas unrecognized by certain institutions.

Focusing on these challenging traits, FedCVD provides new insights and evaluation metrics for designing FL algorithms in multi-center CVD scenarios. Our contributions are summarized as follows:

- 1.
We introduce FedCVD, an open-source federated multi-center healthcare dataset and benchmark specifically for the CVD domain. To the best of our knowledge, FedCVD is the largest multi-center CVD benchmark available. This dataset encompasses two critical tasks—multi-label classification and segmentation—within the CVD domain and includes data of varying scales. Crucially, all datasets are partitioned using natural splits.
- 2.
Our benchmark emphasizes three critical traits in the FL-CVD scenario: non-IID, long tail, and label incompleteness. These traits pose significant challenges to existing FL algorithms.
- 3.
We conducted extensive experiments on FedCVD to evaluate the performance of mainstream FL and centralized learning methods, validating the effectiveness of FL in the CVD context and the proposed three challenges.
Additionally, we have made the open-source code in [https://github.com/SMILELab-FL/FedCVD](https://github.com/SMILELab-FL/FedCVD) accessible for benchmark reproducibility and easy integration into different FL frameworks.

## 2 Related Work

**Table 1: Comparison of FedCVD with Other Federated Datasets or Benchmarks.**
|  | Long-tailedness<br>Considered | Natural<br>Partition | Incomplete<br>Label | Covers CVD<br>Domain | Code<br>Available |
| --- | --- | --- | --- | --- | --- |
| FedDTI [] | ✗ | ✗ | ✗ | ✗ | ✓ |
| FedTD [] | ✗ | ✗ | ✗ | ✗ | ✗ |
| Flamby [] | ✗ | ✓ | ✗ | ✗ | ✓ |
| NIPD [] | ✗ | ✓ | ✗ | ✗ | ✓ |
| FEDLEGAL [] | ✓ | ✓ | ✗ | ✗ | ✓ |
| FLHCD [] | ✗ | ✓ | ✗ | ✓ | ✓ |
| FedMultimodal [] | ✗ | * | ✗ | ✗ | ✓ |
| FedAudio [] | ✗ | * | ✗ | ✗ | ✓ |
| FedCVD | ✓ | ✓ | ✓ | ✓ | ✓ |

Numerous studies have utilized CVD data for disease detection and diagnostic support, with a particular focus on Electrocardiogram (ECG) and Echocardiogram (ECHO) data. ECG, recorded as time-series signals, captures the heart’s electrical activity over time, providing detailed insights into cardiac conditions and potential damage []. These studies typically formulate ECG-based tasks as classification problems aimed at disease diagnosis and heart metric analysis [; ; ; ; ]. ECHO, which consists of ultrasound images of the heart, offers real-time visualization of heart chambers and blood flow, aiding in the diagnosis of conditions such as heart valve problems and Congestive Heart Failure (CHF). For example, [] used ECHO data for Hypertrophic Cardiomyopathy detection through classification, while [] employed Convolutional Neural Networks (CNNs) for standard view classification to improve clinical efficiency. Additionally, ECHO segmentation plays a crucial role in assessing heart morphological changes, and supporting diagnoses of conditions like myocardial infarction. Given that manual segmentation is time-consuming and prone to subjectivity, many studies have employed AI models for automated segmentation of ECHO images, including ventricular segmentation [; ; ; ] and atrial segmentation [; ]. While these studies demonstrate the potential of machine learning models on CVD data, they are largely confined to single-institution settings.

CVDs often require multi-center collaboration for effective research, and FL offers a promising solution to this challenge. Most current studies simulate FL in multi-institution collaborative training by manually partitioning data from a single institution. For instance, [] trained a classification model to detect cardiac arrhythmia using ECG data within a federated architecture, partitioning data from the MIT-BIH Supraventricular Arrhythmia database []. Similarly, [] investigated congestive heart failure detection in a federated setting by splitting samples from the NSR-RR-interval and CHF-RR-interval databases [] into 2 to 4 clients for simulated training. FedCluster [] tackled the issue of unbalanced class distributions in ECG data by optimizing algorithms that cluster local parameters before performing intra- and inter-class aggregation, thus increasing the weight of minority classes. Their data were also partitioned from the MIT-BIH Arrhythmia database []. However, these partition-based simulations may not fully capture the true distribution characteristics of CVDs. In contrast, FLHCD [] demonstrated federated training for hypertrophic cardiomyopathy detection using ECG and ECHO data from four medical institutions (three in the US and one in Japan), showcasing the effectiveness of FL in a naturally partitioned, multi-center setting.

To further support research in FL, numerous datasets and benchmarks have been proposed for a wide range of applications. A comprehensive comparison of FedCVD with these benchmarks is shown in Table 1. Existing studies often manually partition centralized datasets and introduce perturbations or masking to features and labels to mimic the heterogeneity found in real-world FL scenarios. For instance, FedTD [] and FedDTI [] simulate non-iid data partitioning by altering feature distributions and sample sizes.

To better reflect real-world conditions, several FL benchmarks utilize real-world multi-institution data directly. For instance, NIPD [] employs data from cameras in different geographical locations as FL clients for person detection tasks, naturally exhibiting non-iid characteristics. Similarly, FEDLEGAL [] provides a FL benchmark for NLP tasks in the legal domain, using geographically distributed case-based text data for natural data partitioning. Another example is FLamby [], an FL benchmark for real-world distributed medical data, offering seven datasets naturally distributed by geography or institution, with corresponding tasks including segmentation and binary/multiclass classification for medical image analysis and diagnostic assistance. Some benchmarks combine natural partitioning with simulated partitioning. For instance, FedAudio [] applies simulated partitioning for certain audio data, while introducing perturbations to mimic noisy data and labels. FedMultimodal [] uses a Dirichlet distribution to partition multimodal data from various domains, incorporating missing modalities, labels, and erroneous labels to simulate real-world heterogeneity. Despite these advances, none of these benchmarks cover the CVD domain. Although FLHCD [], which utilizes multi-institution data for hypertrophic cardiomyopathy detection, has a setup most similar to ours, it does not address challenges such as the long-tail distribution and incomplete label issues, which are specifically tackled by FedCVD.

## 3 The Proposed FedCVD

In this section, we present the details of the proposed general FL framework for healthcare tasks as shown in Figure 1. Our framework is built upon the lightweight open-source framework FedLab [] for FL simulation. We present the details of datasets, metrics, and baseline models in Section 3.1. Then, we discuss the main FL challenges that FedCVD supported in Section 3.2.

### 3.1 Datasets, Metrics and Baseline Models

Figure 1 provides an overview of the datasets included in FedCVD. In this section, we provide a brief description of each dataset, corresponding metrics, and baseline models.

##### Fed-ECG.

The 12-lead ECG signals in Fed-ECG are sourced from four distinct datasets. The first and third datasets were collected from Shandong Provincial Hospital [] and Shaoxing People’s Hospital [] in China, respectively. The second dataset is from the PTB-XL database, released by Physikalisch Technische Bundesanstalt (PTB) [], and the fourth originates from the PhysioNet/Computing in Cardiology Challenge 2020 [], which represents a large population from the Southeastern United States. These four datasets, originating from geographically diverse regions, are naturally suited for the Federated Learning (FL) setting due to their separation by location.

The corresponding task on these four datasets involves multi-label classification for each institution, a challenging problem due to the large number of labels and the long-tail distribution inherent to the data. To provide a more fine-grained evaluation, we focus on detailed label distinctions, which are of particular interest to clinicians, rather than broader label categories. To thoroughly assess performance, we adopt two metrics: *micro-F1*, which evaluates the overall performance across all labels, and *mean average precision (mAP)*, which specifically measures the impact of the long-tail distribution on model performance.

The original four datasets consist of ECG data with varying lengths and labels, each based on different standards, such as AHA [], SCP-ECG, and SNOMED-CT, making them incompatible for use in a Federated Learning (FL) setting directly. To standardize ECG lengths, we truncate signals longer than 5000 samples and apply edge padding to those shorter than 5000. Additionally, we retain only samples whose labels appear in at least two datasets, ensuring alignment across labels. Figures 2(a) and 2(b) illustrate the heterogeneity in both age and label distributions among institutions. For our baseline model, we adopt a residual network, following the implementation from []. Appendix C provides further details on the dataset and the preprocessing pipeline.

Figure: (a) Feature non-IID of the Fed-ECG dataset, demonstrated with the non-identical distribution of patient age among institutions.
Refer to caption: x2.png

##### Fed-ECHO.

This dataset is derived from three sources: CAMUS [], ECHO-DYNAMIC [], and HMC-QU []. CAMUS provides a database of gray-scale 2D apical four-chamber echocardiographic images, acquired at the University Hospital of St. Etienne in France, fully annotated with the left ventricular endocardium ($\rm LV_{Endo}$), epicardium ($\rm LV_{Epi}$), and left atrial wall ($\rm LA$) regions. ECHO-DYNAMIC contains 2D echocardiogram videos collected at Stanford Medicine, with only the $\rm LV_{Endo}$ region annotated in two frames. HMC-QU, released through a collaboration between Qatar University (QU) and Hamad Medical Corporation (HMC) Hospital, includes 2D echocardiogram videos from Qatar, with annotations limited to the $\rm LV_{Epi}$ region in frames of a single cardiac cycle.

The common task across these three datasets is the automatic segmentation of cardiac structures in echocardiograms, a crucial step in further diagnosing cardiovascular diseases. This task is particularly challenging due to the varying quality of the original echocardiograms across datasets. To evaluate segmentation accuracy, we use both the *Dice similarity index (DICE)* and the *2D Hausdorff distance ($d_{H}$)*. The Dice index measures the overlap between the predicted segmentation and the ground truth, while the $d_{H}$ quantifies the local maximum distance between the two areas.

For consistency among each institution, we only select annotated frames for experiment. The followed image pre-processing pipeline includes picture resizing to $\rm 1\times 112\times 112$, and label alignment. As a baseline model, we use a 2D U-net model following the implementation from []. More details about this dataset are available in Appendix D.

### 3.2 Challenging Traits of FedCVD

##### Non-IID .

Non-independently and identically distributed (non-IID)is a typical characteristic in FL scenarios, encompassing non-IID features and non-IID labels, where clients’ data shows heterogeneity in both feature and label spaces. Quantity imbalance, where institutions hold uneven sample sizes, can further exacerbate these non-IID issues. Among these, non-IID labels have the most pronounced impact on FL model performance. This is because the quantity and types of labels held by each institution can vary greatly, misleading the local supervised training process and causing "Client Drift" [], which hinders global model convergence.

Fed-ECG naturally exhibits these three characteristics. In terms of feature distribution, Figure 2(a) shows significant age distribution differences among institutions’ patients, with Institution 1 notably younger, reflected in the ECG features. Regarding sample size, Figure 2(b) depicts significant differences among the four institutions, with Institution 4 having the fewest samples. For label distribution in the Fed-ECG multi-label classification task, each sample may belong to multiple categories, but the quantity and proportion of different labels vary significantly among institutions. For example, the most common label for Institution 1 and Institution 2 is NORM (Normal), while for Institution 3 and Institution 4 it is STACH (Sinus tachycardia). Some institutions may even lack samples with certain labels, such as both Institution 3 and Institution 4 lacking samples labeled as PAC (Atrial premature complex(es)). These non-IID characteristics challenge the four institutions in collaboratively training a multi-label prediction model, as institutions struggle to capture information about the labels they lack during local training, potentially leading to client drift.

##### Long-tail Distribution.

In addition to the inter-institution heterogeneity caused by non-IID labels, Fed-ECG also exhibits intra-institution and inter-institution heterogeneity in the form of a long-tail distribution of labels. As shown in Figure 2(b), each institution’s internal label distribution has a clear long-tail characteristic, with a few dominant labels having many samples and numerous labels having fewer samples (long tail). These tail categories are already troublesome during independent local training, as the model may focus more on the head categories and neglect the tail ones. In FL scenarios with quantity imbalance and non-IID labels, the long-tail problem is further exacerbated. For instance, categories mainly found in the disadvantaged institutions’ tails may be in an even worse position within the overall data of all institutions. The long-tail characteristic challenges FL algorithms in ensuring the effectiveness and fairness of handling samples from various categories across institutions.

##### Label Incompleteness.

Fed-ECHO presents the most challenging scenario: label-incomplete FL. In Fed-ECHO, three naturally formed institutions hold ECHO video data with annotations (image region segmentation). However, due to varying annotation capabilities, the completeness of labels among the three institutions differs, as shown in Figure 1. Institution 1 has the most complete labels (four labels) due to its ability to identify and annotate all four key regions (including the background). In contrast, Institution 2 and Institution 3 each have labels for only one key region (LV${}_{\text{Endo}}$ and LV${}_{\text{Epi}}$, respectively). This incompleteness introduces (1) label heterogeneity, similar to the label-non-IID in Fed-ECG, where Institution 2 and Institution 3 lack some labels, and (2) mislabeling, where Institution 2 and Institution 3 label unrecognized parts as background, conflicting with Institution 1’s labels and causing misleading information. This scenario significantly challenges FL algorithms to effectively utilize the different levels of label completeness from each Institution and leverage highly heterogeneous data to benefit the global model.

## 4 Experiment

### 4.1 Experiment Details

##### Baseline Algorithms.

Our experiments utilize seven typical FL algorithms across both datasets. The first four are classical global FL algorithms: *FedAvg* [], the oft-cited FL algorithm, collaboratively trains a global model across participants. *FedProx* [] addresses statistical heterogeneity in FL by introducing an L2 proximal term during local training, while *Scaffold* [] mitigates client drift through control variates and server-side learning rate adjustments. *FedInit* [] also tackles client drift by employing a personalized, relaxed initialization at the start of each local training stage. The last three are personalized FL methods: *Ditto* [], which excels in balancing accuracy, fairness, and robustness in FL; *FedSM* [], which combines model selection with personalized methods to avoid client drift; and *FedALA* [], which reduces the impact of statistical heterogeneity by adaptively aggregating both the global and local models. For the Fed-ECHO dataset, we further evaluate two Federated Semi-Supervised Learning (FSSL) methods: *Fed-Consist* [], which uses a consistency-based method for segmentation, and *FedPSL* [], which applies separate model aggregation and meta-learning techniques for classification. In addition to the FL family, we include two other baseline algorithms: *Client*, which refers to training models using only local data without collaboration among participants, and *Central.*, which represents the ideal centralized training scenario where the server has access to all participants’ data.

##### Setup.

The number of institutions involved in federated training for each task is listed in Appendix C. Our experiments mainly focus on the multi-center FL scenario (i.e., cross-silo), where all institutions participate in training at each communication round. Considering the trade-off between computation and communication, we set the local training epoch to 1 and the communication rounds to 50 throughout experiments except Fed-Consist. Since Fed-Consist requires extra rounds for training on clients with full labels before starting federated learning, we set the communication rounds to 100, where 50 rounds are for labeled clients training and another 50 rounds are normal FL training.

##### Evaluation Strategies.

For a comprehensive evaluation, we build a local and global evaluation set for both datasets in FedCVD. For the local one, we divide each local data into train/test sets by 8:2. For the global one, we collect each local test set together. Our experiments test all algorithms using two evaluation strategies: 1) Global test performance (GLOBAL) is evaluated on the global test set and used to determine whether the model has learned knowledge from other clients in the FL setting. The better results of GLOBAL indicate that the model is closer to the centralized training. 2) Local test performance (LOCAL) is evaluated on each local test set. The LOCAL is more practical in real-world applications than GLOBAL because it indicates performance improvement for its task without centralizing all local data.

### 4.2 Benchmark on Fed-ECG

**Table 2: The performance of different FL methods on Fed-ECG is reported using two metrics: Micro F1-Score (Mi-F1) and Mean Average Precision (mAP), both expressed as percentages (%). The best results for each configuration are highlighted in bold, while the second-best results are underlined.**
|  | LOCAL | GLOBAL |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Client1 | Client2 | Client3 | Client4 |  |  |  |  |  |  |
|  | Mi-F1$\uparrow$ | mAP$\uparrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ |
| Client1 | 85.8$\pm$1.9 | 58.1$\pm$2.6 | 52.7$\pm$3.4 | 37.8$\pm$2.2 | 61.5$\pm$1.2 | 19.8$\pm$1.2 | 49.8$\pm$4.2 | 26.7$\pm$3.0 | 64.3$\pm$2.1 | 32.3$\pm$2.0 |
| Client2 | 69.9$\pm$50.0 | 38.9$\pm$30.0 | 76.8$\pm$90.0 | 55.7$\pm$50.0 | 26.3$\pm$80.0 | 22.7$\pm$30.0 | 42.2$\pm$80.0 | 31.6$\pm$60.0 | 50.4$\pm$30.0 | 35.9$\pm$70.0 |
| Client3 | 22.7$\pm$0.2 | 29.8$\pm$0.7 | 17.0$\pm$0.4 | 27.2$\pm$0.3 | 88.1$\pm$0.2 | 37.7$\pm$0.4 | 56.9$\pm$0.4 | 29.4$\pm$0.6 | 51.5$\pm$0.2 | 32.7$\pm$0.2 |
| Client4 | 23.7$\pm$2.0 | 31.7$\pm$2.7 | 24.7$\pm$3.3 | 30.5$\pm$1.5 | 61.6$\pm$5.5 | 25.3$\pm$2.1 | 72.3$\pm$10.2 | 38.5$\pm$2.8 | 44.7$\pm$4.3 | 29.3$\pm$2.5 |
| FedAvg | 69.0$\pm$10.1 | 58.5$\pm$1.2 | 50.3$\pm$5.3 | 54.4$\pm$0.5 | 77.6$\pm$0.7 | 37.2$\pm$0.3 | 66.3$\pm$0.9 | 39.5$\pm$0.5 | 67.9$\pm$3.8 | 50.8$\pm$0.4 |
| FedProx | 74.0$\pm$7.5 | 60.3$\pm$2.9 | 55.6$\pm$2.7 | 56.4$\pm$0.6 | 73.2$\pm$1.0 | 36.0$\pm$0.8 | 70.2$\pm$2.3 | 43.8$\pm$1.8 | 68.8$\pm$2.6 | 52.3$\pm$0.9 |
| Scaffold | 77.5$\pm$2.6 | 58.0$\pm$1.2 | 56.9$\pm$1.7 | 55.9$\pm$0.7 | 73.3$\pm$1.0 | 36.2$\pm$0.6 | \ul70.7$\pm$2.9 | 42.7$\pm$1.1 | 70.1$\pm$0.8 | \ul52.1$\pm$0.7 |
| FedInit | 73.0$\pm$6.6 | 58.2$\pm$0.7 | 54.1$\pm$5.2 | 55.6$\pm$1.3 | 73.5$\pm$0.5 | 36.6$\pm$0.1 | 67.8$\pm$2.0 | 41.5$\pm$1.0 | 68.1$\pm$3.0 | 51.5$\pm$0.9 |
| Ditto | \ul82.8$\pm$4.4 | 63.1$\pm$4.2 | 74.8$\pm$1.4 | 58.3$\pm$0.6 | \ul86.5$\pm$1.5 | 38.1$\pm$0.6 | 73.4$\pm$6.7 | 42.2$\pm$4.0 | 68.1$\pm$2.9 | 48.7$\pm$1.4 |
| FedSM | 77.2$\pm$7.2 | 58.8$\pm$1.3 | 59.1$\pm$4.5 | 56.4$\pm$1.4 | 69.8$\pm$0.8 | 35.0$\pm$0.5 | 67.7$\pm$3.6 | \ul42.9$\pm$2.4 | \ul68.9$\pm$2.5 | 51.2$\pm$0.7 |
| FedALA | 84.4$\pm$4.0 | \ul62.0$\pm$7.0 | \ul71.7$\pm$5.7 | \ul57.1$\pm$2.2 | 88.2$\pm$0.1 | \ul37.4$\pm$0.2 | 66.7$\pm$5.9 | 41.2$\pm$2.3 | 67.8$\pm$1.9 | 50.8$\pm$1.3 |
| Central. | 84.9$\pm$0.5 | 54.8$\pm$0.5 | 71.4$\pm$5.0 | 55.2$\pm$2.9 | 84.1$\pm$1.6 | 36.5$\pm$1.1 | 72.2$\pm$3.7 | 41.5$\pm$1.3 | 80.0$\pm$2.1 | 63.2$\pm$2.8 |

**Table 3: Demonstration of FedECG’s *long-tail challenge* through the performance (F1-Score (%)) differences (measured by relative performance drop) on head and tail class groups of varying sizes. “Top K” denotes the selection of K classes with the most/fewest samples as the head/tail group. Comparisons are made among various FL algorithms, with the algorithm achieving the best result (minimum drop) highlighted in bold and the second-best results underlined.**
| Method | Top k F1 Performance |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| k=1(5%) | k=3(15%) | k=5(25%) | k=10(50%) |  |  |  |  |  |  |  |  |  |
| Head | Tail | Drop | Head | Tail | Drop | Head | Tail | Drop | Head | Tail | Drop |  |
| FedAvg | \ul71.9$\pm$12.5 | \ul38.1$\pm$5.0 | \ul47.0 | \ul85.8$\pm$4.2 | \ul16.0$\pm$2.2 | \ul81.3 | 74.1$\pm$2.5 | 25.5$\pm$1.2 | 65.6 | 57.7$\pm$2.5 | 28.7$\pm$1.1 | 50.3 |
| FedProx | 76.8$\pm$4.5 | 30.8$\pm$2.6 | 59.9 | 87.6$\pm$1.4 | 13.1$\pm$1.2 | 85.0 | 71.2$\pm$0.9 | 22.7$\pm$2.6 | 68.1 | 57.5$\pm$1.8 | 31.1$\pm$2.2 | 46.0 |
| Scaffold | 77.4$\pm$3.2 | 33.8$\pm$4.9 | 56.4 | 87.7$\pm$1.2 | 14.6$\pm$2.2 | 83.4 | 71.3$\pm$0.7 | 24.1$\pm$1.4 | 66.3 | 58.4$\pm$1.7 | 32.1$\pm$2.3 | 45.0 |
| FedInit | 77.2$\pm$2.8 | 29.7$\pm$2.7 | 61.5 | 87.8$\pm$1.0 | 12.9$\pm$1.2 | 87.9 | 71.3$\pm$0.6 | 23.2$\pm$0.5 | 69.8 | \ul56.4$\pm$2.5 | \ul29.0$\pm$0.8 | \ul45.8 |
| Ditto | 73.5$\pm$7.4 | 23.8$\pm$5.8 | 67.6 | 86.4$\pm$2.5 | 10.2$\pm$1.7 | 88.2 | 70.6$\pm$2.1 | 21.4$\pm$1.1 | 69.7 | 54.4$\pm$2.6 | 27.6$\pm$1.7 | 49.3 |
| FedSM | 75.5$\pm$8.2 | 25.8$\pm$3.6 | 65.8 | 87.2$\pm$2.9 | 10.5$\pm$2.2 | 85.3 | 69.5$\pm$1.8 | 21.0$\pm$0.9 | 67.5 | 57.0$\pm$1.9 | 30.9$\pm$2.3 | 48.5 |
| FedALA | 72.4$\pm$5.4 | 38.9$\pm$5.8 | 46.2 | 86.0$\pm$1.8 | 16.2$\pm$2.1 | 81.1 | \ul73.7$\pm$1.5 | \ul25.2$\pm$1.2 | \ul65.7 | 57.9$\pm$1.7 | 28.8$\pm$1.0 | 50.3 |
| Central. | 88.6$\pm$2.3 | 35.6$\pm$5.9 | 59.8 | 92.4$\pm$1.6 | 19.5$\pm$7.0 | 78.9 | 84.1$\pm$1.9 | 29.9$\pm$5.5 | 64.5 | 71.3$\pm$3.6 | 44.5$\pm$4.4 | 37.6 |

We present the Fed-ECG dataset, which poses significant challenges for FL scenarios, namely *non-IID data* and *long-tailed distribution*. We first compared the overall performance of mainstream FL algorithms on Fed-ECG, with the evaluated local and global performance shown in Table 2. The results indicate that FL has advantages over local training. Additionally, FL algorithms designed for heterogeneous scenarios (FedProx, Scaffold, FedInit) outperform the FedAvg algorithm. The personalized algorithms Ditto, FedSM, and FedALA exhibit excellent performance in local tests for certain clients. However, these FL algorithms still lag behind centralized training.

To better illustrate the impact of these two challenges on FL performance, we introduced additional experimental settings and evaluation metrics. For the *non-IID challenge*, we compared the performance differences between natural partitioning and two simulated partitioning methods (random and non-IID), with the simulated non-IID partitioning method described in Appendix C. Figure 3 compares the performance (percentage relative to centralized training) between FL algorithms trained under the three partitioning settings. The results reveal that Fed-ECG’s natural partitioning poses significantly greater challenges compared to the two simulated partitioning methods.

For the *long-tail challenge*, we used the mAP metric in Table 2 to evaluate the overall performance of algorithms across different classes. In general, FL algorithms designed for heterogeneous scenarios demonstrate an advantage in addressing long-tail issues, with personalized algorithms like Ditto and FedALA showing better results in local tests. However, in global tests, the FedProx algorithm more effectively handles long-tail problems. Comparisons with centralized training reveal that FL scenarios tend to amplify the impact of long-tail distributions. To further illustrate this challenge, we introduced the F1-STD metric, which measures the standard deviation of F1 scores across classes. This metric reflects the algorithm’s ability to manage long-tail problems; the larger the STD, the poorer the algorithm’s performance in this regard. The GLOBAL F1-STD results of different FL algorithms are visually presented in Figure 4, showing a pattern consistent with Table 2 and underscoring the challenges posed by long-tail distributions.

To more granularly evaluate the performance of algorithms in long-tail scenarios, we introduced a new metric, *Top-K*. Top-K refers to selecting the K classes with the most samples and the K classes with the fewest samples, calculating the average F1 score for each group, and then computing the relative performance drop between them. A larger performance drop indicates a more severe long-tail problem. Table 3 presents the Top-K metrics for various K values, highlighting the significant long-tail characteristics of Fed-ECG. The results show that mainstream FL algorithms struggle to address long-tail issues effectively, performing worse compared to centralized training.

Figure: Figure 3: Demonstration of Fed-ECG’s *non-IID challenge*: Comparisons of performance (relative Mean Average Score %) between artificial partitions (simulated random and non-IID partitions) and Fed-ECG’s natural partition across different FL algorithms.
Refer to caption: x4.png

Figure: Figure 4: Demonstration of Fed-ECG’s *long-tail challenge*: Average Macro F1-Score (%) and Standard Deviation across classes for various FL Algorithms.
Refer to caption: x5.png

### 4.3 Benchmark on Fed-ECHO

The proposed Fed-ECHO dataset presents one of the most challenging FL settings: *label incompleteness*, which can be viewed as an enhanced version of label-non-IID. Specifically, all annotated video frames from Institution 1 are completely segmented into four regions: BG, LV${}_{\text{Endo}}$, LV${}_{\text{Epi}}$, and LA. In contrast, Institution 2 and Institution 3 can only recognize the LV${}_{\text{Epi}}$ and LV${}_{\text{Endo}}$ regions in their annotated video frames, respectively, with the remaining regions simply labeled as "BG." For convenience, we refer to the BG labels from Institution 2 and Institution 3 as "Maybe-BG," indicating these segmentations may be unreliable. This discrepancy introduces potential conflicts between the "Maybe-BG" labels of Institution 2 and Institution 3 and the corresponding "reliable" labels of Institution 1, resulting in misleading labels that affect model convergence.

To mitigate the impact of misleading labels, we propose a straightforward baseline strategy, *supervised-only*. During supervised model training, we input the data from all three Institutions into the model without additional processing, allowing the model to benefit from the rich data features. However, when calculating the loss, we mask out the "Maybe-BG" regions in the video frames from Institution 2 and Institution 3. This means that for samples from Institution 2 and Institution 3, we only compute the training loss on the "reliable foreground". This strategy ensures the model learns segmentation capabilities from completely reliable labels. Additionally, during model segmentation performance evaluation, we also exclude the "Maybe-BG" regions from the test samples, preventing them from influencing the model’s performance metrics.

**Table 4: The performance of different FL methods on Fed-ECHO, with DICE (%) and $d_{H}$ representing DICE index and Hausdorff distance respectively. The best results for each configuration are highlighted in bold, while the second-best results are underlined.**
|  | LOCAL | GLOBAL |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mthods | Client1 | Client2 | Client3 |  |  |  |  |  |
|  | Dice$\uparrow$ | $d_{H}$$\downarrow$ | Dice$\uparrow$ | $d_{H}$$\downarrow$ | Dice$\uparrow$ | $d_{H}$$\downarrow$ | Dice$\uparrow$ | $d_{H}$$\downarrow$ |
| Client1 | 88.2$\pm$0.8 | 5.196$\pm$0.360 | 46.5$\pm$3.9 | 24.246$\pm$0.442 | 63.4$\pm$4.2 | 22.000$\pm$12.914 | 66.1$\pm$2.8 | 17.147$\pm$4.187 |
| Client2 | 24.4$\pm$6.0 | 71.917$\pm$2.832 | 88.9$\pm$5.5 | 5.577$\pm$1.413 | - | - | 37.8$\pm$3.8 | 59.165$\pm$1.411 |
| Client3 | 15.8$\pm$1.0 | 76.368$\pm$0.988 | - | - | 94.1$\pm$0.7 | 7.110$\pm$2.900 | 36.6$\pm$0.3 | 61.159$\pm$0.931 |
| FedAvg | 26.2$\pm$3.7 | 48.343$\pm$8.719 | 56.4$\pm$8.8 | 33.127$\pm$10.721 | 67.9$\pm$3.5 | 34.004$\pm$5.287 | 50.2$\pm$5.3 | 38.491$\pm$8.058 |
| FedProx | 74.8$\pm$18.7 | 13.928$\pm$11.742 | 82.3$\pm$3.5 | 13.402$\pm$2.975 | 66.7$\pm$12.4 | 16.181$\pm$16.329 | 74.6$\pm$11.4 | 14.504$\pm$9.134 |
| Scaffold | 81.5$\pm$2.1 | 9.981$\pm$2.482 | 81.0$\pm$2.1 | \ul12.543$\pm$2.157 | 74.6$\pm$2.1 | \ul7.551$\pm$0.885 | \ul79.0$\pm$0.7 | \ul10.025$\pm$1.467 |
| FedInit | 83.5$\pm$0.9 | \ul7.799$\pm$0.665 | \ul81.6$\pm$2.2 | 12.240$\pm$1.091 | \ul73.4$\pm$3.0 | 7.542$\pm$0.918 | 79.5$\pm$0.5 | 9.193$\pm$0.558 |
| Ditto | 88.2$\pm$0.4 | 4.796$\pm$0.085 | 56.9$\pm$3.3 | 28.381$\pm$4.043 | 56.3$\pm$2.2 | 27.321$\pm$15.627 | 78.1$\pm$1.8 | 10.658$\pm$2.372 |
| FedSM | 80.2$\pm$6.0 | 11.339$\pm$5.868 | 81.1$\pm$1.5 | 12.580$\pm$1.288 | 72.7$\pm$2.0 | 10.913$\pm$4.128 | 78.0$\pm$2.2 | 11.611$\pm$2.308 |
| FedALA | 80.5$\pm$1.6 | 8.700$\pm$1.245 | 51.3$\pm$2.4 | 36.472$\pm$2.686 | 47.1$\pm$0.9 | 52.128$\pm$4.356 | 52.3$\pm$2.0 | 36.811$\pm$2.630 |
| Fed-Consist | \ul85.9$\pm$0.2 | 11.904$\pm$0.442 | 75.2$\pm$0.9 | 27.480$\pm$1.440 | 66.3$\pm$0.2 | 34.037$\pm$1.777 | 75.8$\pm$0.3 | 24.474$\pm$1.155 |
| FedPSL | 53.5$\pm$9.3 | 37.277$\pm$9.166 | 77.0$\pm$2.9 | 12.873$\pm$1.589 | 67.8$\pm$14.1 | 29.166$\pm$15.660 | 66.1$\pm$7.5 | 26.439$\pm$7.831 |
| Central.(sup) | 89.9$\pm$0.4 | 4.643$\pm$0.097 | 48.5$\pm$22.2 | 43.684$\pm$19.659 | 65.0$\pm$14.6 | 30.557$\pm$14.831 | 67.8$\pm$12.1 | 26.295$\pm$11.379 |
| Central.(ssup) | 90.3$\pm$0.2 | 3.872$\pm$0.067 | 91.7$\pm$0.5 | 4.370$\pm$0.181 | 91.1$\pm$1.7 | 3.005$\pm$0.732 | 91.0$\pm$0.6 | 3.749$\pm$0.242 |

Table 4 compares the performance of mainstream FL algorithms with centralized/isolated learning on Fed-ECHO, evaluated using Dice and Hausdorff distance ($d_{H}$). Except for the semi-supervised learning algorithms Centralized (semi-sup) and Fed-Consist, all algorithms use the previously mentioned supervised-only strategy. The results underscore the viability of FL in the Fed-ECHO setting, as most FL algorithms exhibit superior global performance compared to models trained independently by individual institutions. However, due to high degree of data heterogeneity, none of the evaluated FL algorithms outperform locally trained models on each client’s test dataset, indicating a need for more personalized and heterogeneity-resistant FL strategies.

On the global test set, FL algorithms specifically designed to address heterogeneity, such as FedInit and Scaffold, consistently demonstrate significant advantages over simpler algorithms like FedAvg. Notably, these algorithms also outperform the Centralized (sup) model, which we attribute to FL effectively mitigating the impact of intra-batch heterogeneity(e.g., within the same batch, there are four-label data from Institution 1 and single-label data from Institution 2 or Institution 3).

Additionally, to leverage the substantial amount of partially labeled data from Institution 2 and Institution 3 and potentially mitigate label heterogeneity, we introduced semi-supervised learning algorithms for comparison. These include the centralized semi-supervised model, Centralized (ssup), and federated semi-supervised algorithms such as Fed-Consist and FedPSL. The Centralized (ssup) model significantly outperformed its fully supervised counterpart, underscoring the value of utilizing unlabeled video frames.
Similarly, Fed-Consist outperformed FedAvg and FedProx, although it still exhibited a noticeable performance gap compared to the centralized semi-supervised algorithm and lagged behind fully supervised FL algorithms like Scaffold and FedInit. While FedPSL performed well on certain participating client, it showed greater instability overall, largely due to its sensitivity to client-side feature heterogeneity.

Therefore, the highly heterogeneous Fed-ECHO scenario poses significant challenges for FL algorithms, requiring them to adapt to heterogeneous data and effectively leverage unlabeled data across different data domains.

## 5 Conclusion

This paper has introduced FedCVD, the first real-world multi–center FL benchmark for CVD data, which consists of two datasets and their respective tasks: Fed-ECG and Fed-ECHO. It presents three major challenges due to the heterogeneous distribution of real-world data: non-IID, long-tailed labels, and label incompleteness. We conducted extensive comparative and validation experiments, testing mainstream FL algorithms and centralized training on these tasks. Experimental results show that the natural non-IID characteristics in FedCVD are more challenging than the manually partitioned setups in most previous federated benchmarks, and mainstream algorithms perform poorly in the long-tail tests of FedCVD. For the most difficult task, i.e., the label-incomplete Fed-ECHO, mainstream FL algorithms barely maintain utility, but are still better than non-cooperative algorithms that only utilize unlabeled data on each client. Federated semi-supervised learning algorithms that leverage unlabeled data achieve some performance improvement. Beyond, as a flexible and extensible framework, FedCVD is meant to be a step towards the development of FL on CVD domain.

Limitations and Future Work. FedCVD presents a realistic and challenging scenario that tests FL algorithms’ ability to mitigate data heterogeneity, handle long-tailed classes, and utilize unlabeled data. However, FedCVD currently offers a limited variety of data types and only two tasks. Additionally, the FL algorithms compared in experiments, particularly semi-supervised ones, are limited. In future work, we will expand the data range of FedCVD, aiming for it to inspire future FL research in real-world medical contexts, especially with CVD data.

## Appendix A Broader Impact

Considering that this research exclusively involves the repurposing of existing open-source databases, the associated risks are limited. However, it is important to acknowledge that all datasets utilized in this study may be influenced by biases inherent in the original data collection processes, such as those related to gender, age, or race.
Unfortunately, identifying the sources of potential biases is challenging because the data have been appropriately pseudonymized. Moreover, records such as electrocardiograms and echocardiograms cannot be easily linked to specific demographic attributes such as age, ethnicity, or gender by non-medical experts. Nonetheless, our work discloses certain metadata of the datasets, including geographical origin, gender distribution, and age distribution. This exposure may aid in identifying underlying geographical biases, which are anticipated in real-world federated learning scenarios.

While prioritizing simplicity and utility, the current benchmark does not include privacy metrics. Nevertheless, privacy remains critically important in the cardiovascular disease domain, and we strongly encourage the research community to address these considerations. Thanks to the modularity of FedCVD, we can add privacy components easily.
Therefore, we anticipate that FedCVD will address privacy concerns related to federated learning within the cardiovascular disease domain in the future.

## Appendix B Datasets repository and Maintenance plane

### B.1 Dataset repository.

The code is now available at [https://github.com/SMILELab-FL/FedCVD](https://github.com/SMILELab-FL/FedCVD).Considering licenses, users need to download the data manually through the original dataset link.

### B.2 Maintenance plan

We shall adhere to a maintenance plan to uphold the integrity of the codebase and ensure the conformity of supplied datasets to requisite standards.
In particular, this maintenance plan encompasses:

- •
Fixing bugs affecting the correctness of our code, whether identified by the community or ourselves;
- •
Introducing additional variants of federated learning techniques, including alternative methods within the scope of cross-silo federated learning and federated semi-supervised learning methodologies;
- •
Adding new functional modules, such as privacy protection components.
- •
Regarding datasets, reviewing potential updates of the datasets referenced in the FedCVD, including but not limited to introducing new tasks or modalities;

## Appendix C Fed-ECG

### C.1 Description

Fed-ECG consists of four datasets: SPH, PTB-XL, SXPH, and G12EC. The order of leads of each dataset is I, II, III, aVR, aVL, aVF, V1, V2, V3, V4, V5, V6. The overview of Fed-ECG is shown in Table 5. Table 6 shows demographics information for four datasets in Fed-ECG.

**Table 5: Overview of the datasets, tasks, metrics and baseline models in FedCVD.**
| Dataset | Fed-ECG | Fed-ECHO |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Task Type | Multi-label Classification | 2D Segmentation |  |  |  |  |  |
| Input | 12-lead ECG Signal | Echocardiogram |  |  |  |  |  |
| Prediction (y) | Diagnostic Statement | Cardiac Structure Mask |  |  |  |  |  |
| Data source | SPH | PTB-XL | SXPH | G12EC | CAMUS | ECHONET-DYNAMIC | HMC-QU |
| Original Patient Size | 24,666 | 18,885 | 45,152 | UNKNOWN | 500 | 10,030 | 109 |
| Original Sample Size | 25,770 | 21,837 | 45,152 | 10,344 | 1000 | 20,060 | 2,349 |
| Preprocessing | Label Alignment | Resizing and Label Alignment |  |  |  |  |  |
| Patient Size | 21,530 | 16,699 | 36,272 | UNKNOWN | 500 | 10,024 | 109 |
| Sample Size | 22,425 | 19,019 | 36,272 | 6,205 | 1000 | 20,048 | 2,349 |
| Model | ResNet | U-net |  |  |  |  |  |
| Metrics | Micro F1 / mAP | DICE / Hausdorff distance |  |  |  |  |  |
| Input Dimension | 12 $\times$ 5000 | 112 $\times$ 112 |  |  |  |  |  |

##### SPH.

The original Shandong Provincial Hospital (SPH) database contains 25,770 12-lead ECG records from 24,666 patients, which were acquired from Shandong Provincial Hospital between 2019/08 and 2020/08. The record length is between 10 and 60 seconds. The sampling frequency is 500 Hz. All ECG records are in full compliance with the AHA standard, which aims for the standardization and interpretation of the electrocardiogram and consists of 44 primary statements and 15 modifiers as per the standard. 46.04% records in this dataset contain ECG abnormalities. Moreover, 14.45% records have multiple diagnostic statements.

##### PTB-XL.

The original PTB-XL database contains 21,837 12-lead ECG records from 18,885 patients of 10 seconds length at the Physikalisch Technische Bundesanstalt (PTB) between October 1989 and June 1996. The original records are resampled to both 100 Hz and 500 Hz. For consistency, we only use the records whose frequency is 500 Hz. Each data is annotated by up to two cardiologists with the SCP-ECG standard.

##### SXPH.

This database contains 12-lead ECGs of 45,152 patients with a 500 Hz sampling rate under the auspices of Chapman University, Shaoxing People’s Hospital (Shaoxing Hospital Zhejiang University School of Medicine), and Ningbo First Hospital. The record length is 10 seconds. All records are labeled by professional experts with the SNOMED-CT standard.

##### G12EC.

This Georgia 12-lead ECG Challenge (G12EC) database is provided by the PhysioNet/Computing in Cardiology Challenge 2020. Only 10,344 training data from this database are open to the public. The record length is not longer than 10 seconds with a sample frequency of 500 Hz. All records are labeled with the SNOMED-CT standard as well.

**Table 6: Demographics information for Fed-ECG.**
| Client | Sex | Dataset size | Age | Age Range |
| --- | --- | --- | --- | --- |
| Client1 | Female | 9,502 | 48.73 $\pm$ 15.67 | 18 - 92 |
| Male | 12,923 | 50.35 $\pm$ 15.49 | 18 - 95 |  |
| Client2 | Female | 8,930 | 59.80 $\pm$ 18.42 | 3 - 89 |
| Male | 10,089 | 58.40 $\pm$ 15.66 | 2 - 89 |  |
| Client3 | Female | 14,830 | 58.36 $\pm$ 20.11 | 4 - 89 |
| Male | 21,442 | 60.28 $\pm$ 19.10 | 4 - 89 |  |
| Client4 | Female | 2,668 | 61.37 $\pm$ 16.51 | 20 - 89 |
| Male | 3,537 | 61.35 $\pm$ 15.04 | 14 - 89 |  |

### C.2 License and Ethics

All four databases are open-access. The SPH database is open access at Figshare, while the rest databases are open access at PhysioNet under a Creative Commons Attribution 4.0 International Public License.

The PTB-XL database was supported by the Bundesministerium für Bildung und Forschung (BMBF) through the Berlin Big Data Center under Grant 01IS14013A and the Berlin Center for Machine Learning under Grant 01IS18037I and by the EMPIR project 18HLT07 MedalCare. The EMPIR initiative is cofunded by the European Union’s Horizon 2020 research and innovation program and the EMPIR Participating States.

The institutional review board of Shaoxing People’s Hospital and Ningbo First Hospital of Zhejiang University approved the study of the SXPH database, granted the waiver application to obtain informed consent, and allowed the data to be shared publicly after de-identification. The requirement for patient consent was waived.

### C.3 Download and preprocessing

#### C.3.1 Download

The four datasets can be downloaded using the URLs below:

- 1.
SPH: [https://springernature.figshare.com/collections/A_large-scale_multi-label_12-lead_electrocardiogram_database_with_standardized_diagnostic_statements/5779802/1](https://springernature.figshare.com/collections/A_large-scale_multi-label_12-lead_electrocardiogram_database_with_standardized_diagnostic_statements/5779802/1)
- 2.
PTB-XL: [https://physionet.org/content/ptb-xl/1.0.3/](https://physionet.org/content/ptb-xl/1.0.3/)
- 3.
SXPH: [https://physionet.org/content/ecg-arrhythmia/1.0.0/](https://physionet.org/content/ecg-arrhythmia/1.0.0/)
- 4.
G12EC: [https://physionet.org/content/challenge-2020/1.0.2/](https://physionet.org/content/challenge-2020/1.0.2/)

#### C.3.2 Preprocessing

Raw 12-lead ECG signals have varying sequence lengths and raw 12-lead ECG signals have varying sequence lengths and annotated standards which must be standardized before FL training. Therefore, we first set a signal length to 10 seconds. We pad the signal with edge value at the edge for those whose length is shorter than 10 seconds and cut off the signal at 10 seconds for those whose length is longer than 10 seconds.
Next, we only save the records whose label occurs in at least two databases. Finally, we align the labels of records in different databases. The relationship between the original label and our label is shown in Table7.

**Table 7: Label relationship between original label and ours.**
|  | Original Label |  |  |  |
| --- | --- | --- | --- | --- |
| ours | SPH | PTB-XL | SXPH | G12EC |
| NORM (Normal) | Normal | Normal | - | - |
| STACH (Sinus tachycardia) | Sinus tachycardia | Sinus tachycardia | Sinus tachycardia | 427084000 |
| SBRAD (Sinus bradycardia) | Sinus bradycardia | Sinus bradycardia | Sinus bradycardia | 426177001 |
| SARRH (Sinus arrhythmia) | Sinus arrhythmia | Sinus arrhythmia | - | 427393009 |
| PAC (Atrial premature complex(es)) | Atrial premature complex(es) | Atrial premature complex | - | - |
| AFIB (Atrial fibrillation) | Atrial fibrillation | Atrial fibrillation | Atrial fibrillation | 164889003 |
| AFLT (Atrial flutter) | Atrial flutter | Atrial flutter | Atrial flutter | 164890007 |
| SVTAC (Supraventricular tachycardia) | - | Supraventricular tachycardia | Supraventricular tachycardia | 426761007 |
| PVC (Ventricular premature complex) | Ventricular premature complex(es) | Ventricular premature complex | - | 164884008 |
| 1AVB (First degree AV block) | - | First degree AV block | 1 degree atrioventricular block | 270492004 |
|  | Second-degree AV block, Mobitz type I (Wenckebach) |  | 2 degree atrioventricular block(Type one) | 54016002 |
|  | Second-degree AV block, Mobitz type II |  | 2 degree atrioventricular block(Type two) | 28189009 |
|  | 2:1 AV block |  |  | 164903001 |
|  | AV block, varying conduction |  |  | 195042002 |
| 2AVB (Second degree AV block) | AV block, advanced (high-grade) | Second degree AV block | 2 degree atrioventricular block | 284941000119107 |
| 3AVB (Third degree AV block) | AV block, complete (third-degree) | Third degree AV block | 3 degree atrioventricular block | 27885002 |
|  | Left anterior fascicular block | Left anterior fascicular block |  | 445118002 |
|  | Left posterior fascicular block | Left posterior fascicular block |  | 445211001 |
| LBBB (Left bundle branch block) | Left bundle-branch block | Complete left bundle branch block | Left bundle branch block | 164909002 |
|  | Incomplete right bundle-branch block | Incomplete right bundle branch block |  | 713426002 |
|  |  |  |  | 59118001 |
| RBBB (Right bundle branch block) | Right bundle-branch block | Complete right bundle branch block | Right bundle branch block | 164907000 |
| LAO/LAE (Left atrial overload/enlargement) | Left atrial enlargement | Left atrial overload/enlargement | - | 67741000119109 |
| LVH (Left ventricular hypertrophy) | Left ventricular hypertrophy | Left ventricular hypertrophy | - | 164873001 |
| RVH (Right ventricular hypertrophy) | Right ventricular hypertrophy | Right ventricular hypertrophy | - | - |
| AMI (Anterior myocardial infarction) | Anterior MI | Anterior myocardial infarction | - | - |
| IMI (Inferior myocardial infarction) | Inferior MI | Inferior myocardial infarction | - | - |
| ASMI (Anteroseptal myocardial infarction) | Anteroseptal MI | Anteroseptal myocardial infarction | - | - |

### C.4 Baseline, loss function and evaluation

##### Baseline Model.

We implement a ResNet1d model with 34 layers. The final layer output is passed through a sigmoid function to encode the probability that each label corresponds to one 12-lead ECG signal.

##### Loss function.

The model was directly trained for the Binary CrossEntropy Loss (BCELoss), defined as:

$$ $\mathrm{BCE}(\mathbf{y},\hat{\mathbf{y}})=-[\sum_{i=1}^{n}y_{i}\log(\hat{y}_{i })+\sum_{i=1}^{n}(1-y_{i})\log(1-\hat{y}_{i})]$ (1) $$

##### Evaluation Metrics.

In multi-label classification for Fed-ECG, the micro F1 score is used as the main metric to evaluate the performance of the model. Given $N$ labels, the micro-precision ($P_{\text{micro}}$) and micro-recall ($R_{\text{micro}}$) are calculated as $P_{\text{micro}}=\frac{\sum_{i=1}^{N}\text{TP}_{i}}{\sum_{i=1}^{N}(\text{TP}_{
i}+\text{FP}_{i})}$ and $R_{\text{micro}}=\frac{\sum_{i=1}^{N}\text{TP}_{i}}{\sum_{i=1}^{N}(\text{TP}_{
i}+\text{FN}_{i})}$, where $\text{TP}_{i}$ is the number of true positives for label $i$,$\text{FP}_{i}$ is the number of false positives for label $i$,$\text{FN}_{i}$ is the number of false negatives for label $i$. The micro F1 score ($F1_{\text{micro}}$) is then calculated as:

$$ $F1_{\text{micro}}=\frac{2\cdot P_{\text{micro}}\cdot R_{\text{micro}}}{P_{ \text{micro}}+R_{\text{micro}}}$ (2) $$

For Fed-ECG’s Multi-Label Classification task, the Mean Average Precision (mAP) is adopted to measure the classification performance across all labels (including long-tailed labels), calculated by averaging the average precision (AP) for each label, defined as:

$$ $\text{mAP}=\frac{1}{L}\sum_{i=1}^{L}\sum_{k=1}^{n}P_{i}(k)\Delta r_{i}(k)$ (3) $$

where $L$ is the total number of labels, and $\text{AP}_{i}$ is the average precision for the $i$-th label, $P_{i}(k)$ is the precision for label $i$ at the $k$-th threshold, and $\Delta r_{i}(k)$ is the change in its recall at the $k$-th threshold.

### C.5 Training Detail

##### Optimization parameters.

We optimize the ResNet1d using SGD optimizer, with a batch size of 32. We train our model for 50 epochs on one NVIDIA A100-PCIE-40GB.

##### Hyperparameter Search

For centralized and local model training, we first conduct a search for optimal learning rates from the set {1e-5, 1e-4, 1e-3, 1e-2, 1e-1} during centralized model training. The learning rate that yields the best micro-F1 score is then used for local model training. For the federated learning strategies, we employ the following hyperparameter grid:

- •
For clients’ learning rates (all strategies): {1e-5, 1e-4, 1e-3, 1e-2, 1e-1}.
- •
For server size learning rate (Scaffold strategy only): {1e-2, 1e-1, 1.0}.
- •
For FedProx and Ditto strategies, the parameter $\mu$ is selected from {1e-2, 1e-1, 1.0}.
- •
For FedInit, the parameter $\beta$ is chosen from {1e-1, 1e-2, 1e-3}.
- •
For FedSM, the parameters $\gamma$ and $\lambda$ are set to values from {0, 0.1, 0.7, 0.9} and {0.1, 0.3, 0.5, 0.7, 0.9}, respectively.
- •
For FedALA, the parameters layer index, $\eta$, threshold, and num_per_loss are fixed at 1, 1.0, 0.1, and 10, respectively, while rand_percent is selected from {5, 50, 80}.

**Table 8: Hyperparameters used for the Fed-ECG.**
| Fed-ECG |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | learning rate | optimizer | learning rate server | mu | beta | lambda | gamma | rand_percent |
| Central. | 0.1 | torch.optim.SGD | - | - | - | - | - | - |
| FedAvg | 0.1 | torch.optim.SGD | - | - | - | - | - | - |
| FedProx | 0.1 | torch.optim.SGD | - | 0.01 | - | - | - | - |
| Scaffold | 0.1 | torch.optim.SGD | 1.0 | - | - | - | - | - |
| FedInit | 0.1 | torch.optim.SGD | 1.0 | - | 0.01 | - | - | - |
| Ditto | 0.1 | torch.optim.SGD | - | 0.01 | - | - | - | - |
| FedSM | 0.1 | torch.optim.SGD | 1.0 | - | - | 0.1 | 0 | - |
| FedALA | 0.1 | torch.optim.SGD | 1.0 | - | - | - | - | 80 |

##### Non-IID partition.

For the non-IID partition, we first pool the training data from the four clients. Then, we cluster the samples into 10 categories based on the cosine similarity and order them according to the number of samples contained in each category. Next, the sorted samples are divided into 32 shards. finally, 8 random shards are distributed to one client. The label distribution of each client with the non-IID partition is shown in Figure 5.

Figure: Figure 5: Label non-IID of the Fed-ECG dataset with the artificially non-IID partition, shown as the variation in the number of each label (right axis) across different clients (left axis).
Refer to caption: x6.png

### C.6 Supplementary Experiment Results

We provide additional evaluation metrics here. Table 9 presents an extensive array of evaluation metrics for various federated learning approaches applied to Fed-ECG. The Micro F1-Score (Mi-F1) and Hamming Loss (HL) serve as indicators of the overall performance, given their insensitivity to long-tail distributions. In contrast, the mean Average Precision score (mAP) provides insight into the average performance across individual labels. In addition, Figure 6 presents the evaluation metrics for each label, encompassing F1 score, precision, and recall, which more clearly demonstrates the impact of the long-tail distribution on each label.

**Table 9: The performance of different FL methods on Fed-ECG, with Mi-F1, mAP, and HL representing Micro F1-Score, mean Average Precision score, and Hamming Loss, respectively. All metrics are present in percentage (%). The best results for each configuration are highlighted in bold, while the second-best results are underlined.**
|  | LOCAL | GLOBAL |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Client1 | Client2 | Client3 | Client4 |  |  |  |  |  |  |  |  |  |  |  |
|  | Mi-F1$\uparrow$ | mAP$\uparrow$ | HL$\downarrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | HL$\downarrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | HL$\downarrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | HL$\downarrow$ | Mi-F1$\uparrow$ | mAP$\uparrow$ | HL$\downarrow$ |
| Client1 | 85.8<br>$\pm$1.9 | 58.1<br>$\pm$2.6 | 1.5<br>$\pm$ 0.2 | 52.7<br>$\pm$3.4 | 37.8<br>$\pm$2.2 | 5.8<br>$\pm$ 0.4 | 61.5<br>$\pm$1.2 | 19.8<br>$\pm$1.2 | 4.4<br>$\pm$ 0.1 | 49.8<br>$\pm$4.2 | 26.7<br>$\pm$3.0 | 6.4<br>$\pm$ 0.6 | 64.3<br>$\pm$2.1 | 32.3<br>$\pm$2.0 | 4.1<br>$\pm$ 0.2 |
| Client2 | 69.9<br>$\pm$50.0 | 38.9<br>$\pm$30.0 | 3.2<br>$\pm$ 0.1 | 76.8<br>$\pm$90.0 | 55.7<br>$\pm$50.0 | 3.1<br>$\pm$ 0.1 | 26.3<br>$\pm$80.0 | 22.7<br>$\pm$30.0 | 9.0<br>$\pm$ 0.2 | 42.2<br>$\pm$80.0 | 31.6<br>$\pm$60.0 | 8.1<br>$\pm$ 0.1 | 50.4<br>$\pm$30.0 | 35.9<br>$\pm$70.0 | 6.1<br>$\pm$ 0.1 |
| Client3 | 22.7<br>$\pm$0.2 | 29.8<br>$\pm$0.7 | 8.2<br>$\pm$ 0.0 | 17.0<br>$\pm$0.4 | 27.2<br>$\pm$0.3 | 10.3<br>$\pm$ 0.1 | 88.1<br>$\pm$0.2 | 37.7<br>$\pm$0.4 | 1.3<br>$\pm$ 0.0 | 56.9<br>$\pm$0.4 | 29.4<br>$\pm$0.6 | 5.4<br>$\pm$ 0.1 | 51.5<br>$\pm$0.2 | 32.7<br>$\pm$0.2 | 5.5<br>$\pm$ 0.0 |
| Client4 | 23.7<br>$\pm$2.0 | 31.7<br>$\pm$2.7 | 8.4<br>$\pm$ 0.9 | 24.7<br>$\pm$3.3 | 30.5<br>$\pm$1.5 | 10.1<br>$\pm$ 1.2 | 61.6<br>$\pm$5.5 | 25.3<br>$\pm$2.1 | 5.0<br>$\pm$ 1.2 | 72.3<br>$\pm$10.2 | 38.5<br>$\pm$2.8 | 4.1<br>$\pm$ 1.8 | 44.7<br>$\pm$4.3 | 29.3<br>$\pm$2.5 | 7.0<br>$\pm$ 1.1 |
| FedAvg | 69.0<br>$\pm$10.1 | 58.5<br>$\pm$1.2 | 3.4<br>$\pm$ 1.1 | 50.3<br>$\pm$5.3 | 54.4<br>$\pm$0.5 | 6.2<br>$\pm$ 0.7 | 77.6<br>$\pm$0.7 | 37.2<br>$\pm$0.3 | 2.5<br>$\pm$ 0.1 | 66.3<br>$\pm$0.9 | 39.5<br>$\pm$0.5 | 4.2<br>$\pm$ 0.1 | 67.9<br>$\pm$3.8 | 50.8<br>$\pm$0.4 | 3.7<br>$\pm$ 0.5 |
| FedProx | 74.0<br>$\pm$7.5 | 60.3<br>$\pm$2.9 | 2.9<br>$\pm$ 1.0 | 55.6<br>$\pm$2.7 | 56.4<br>$\pm$0.6 | 5.5<br>$\pm$ 0.5 | 73.2<br>$\pm$1.0 | 36.0<br>$\pm$0.8 | 3.0<br>$\pm$ 0.1 | 70.2<br>$\pm$2.3 | 43.8<br>$\pm$1.8 | 3.8<br>$\pm$ 0.3 | 68.8<br>$\pm$2.6 | 52.3<br>$\pm$0.9 | 3.6<br>$\pm$ 0.4 |
| Scaffold | 77.5<br>$\pm$2.6 | 58.0<br>$\pm$1.2 | 2.3<br>$\pm$ 0.2 | 56.9<br>$\pm$1.7 | 55.9<br>$\pm$0.7 | 5.2<br>$\pm$ 0.2 | 73.3<br>$\pm$1.0 | 36.2<br>$\pm$0.6 | 3.0<br>$\pm$ 0.1 | \ul70.7<br>\ul$\pm$2.9 | 42.7<br>$\pm$1.1 | \ul3.7<br>\ul$\pm$ 0.3 | 70.1<br>$\pm$0.8 | \ul52.1<br>\ul$\pm$0.7 | 3.4<br>$\pm$ 0.1 |
| FedInit | 73.0<br>$\pm$ 6.6 | 58.2<br>$\pm$ 0.7 | 3.1<br>$\pm$ 1.0 | 54.1<br>$\pm$ 5.2 | 55.6<br>$\pm$ 1.3 | 5.9<br>$\pm$ 0.9 | 73.5<br>$\pm$ 0.5 | 36.6<br>$\pm$ 0.1 | 3.0<br>$\pm$ 0.1 | 67.8<br>$\pm$ 2.0 | 41.5<br>$\pm$ 1.0 | 4.1<br>$\pm$ 0.3 | 68.1<br>$\pm$ 3.0 | 51.5<br>$\pm$ 0.9 | 3.8<br>$\pm$ 0.5 |
| Ditto | \ul82.8<br>\ul$\pm$4.4 | 63.1<br>$\pm$4.2 | \ul1.8<br>\ul$\pm$ 0.4 | 74.8<br>$\pm$1.4 | 58.3<br>$\pm$0.6 | 3.5<br>$\pm$ 0.2 | \ul86.5<br>\ul$\pm$1.5 | 38.1<br>$\pm$0.6 | \ul1.5<br>\ul$\pm$ 0.2 | 73.4<br>$\pm$6.7 | 42.2<br>$\pm$4.0 | 3.6<br>$\pm$ 0.9 | 68.1<br>$\pm$2.9 | 48.7<br>$\pm$1.4 | \ul3.6<br>\ul$\pm$ 0.3 |
| FedSM | 77.2<br>$\pm$ 7.2 | 58.8<br>$\pm$ 1.3 | 2.3<br>$\pm$ 0.6 | 59.1<br>$\pm$ 4.5 | 56.4<br>$\pm$ 1.4 | 5.1<br>$\pm$ 0.5 | 69.8<br>$\pm$ 0.8 | 35.0<br>$\pm$ 0.5 | 3.5<br>$\pm$ 0.1 | 67.7<br>$\pm$ 3.6 | \ul42.9<br>\ul$\pm$ 2.4 | 4.1<br>$\pm$ 0.4 | \ul68.9<br>\ul$\pm$ 2.5 | 51.2<br>$\pm$ 0.7 | 3.6<br>$\pm$ 0.3 |
| FedALA | 84.4<br>$\pm$ 4.0 | \ul62.0<br>\ul$\pm$ 7.0 | 1.6<br>$\pm$ 0.4 | \ul71.7<br>\ul$\pm$ 5.7 | \ul57.1<br>\ul$\pm$ 2.2 | \ul3.8<br>\ul$\pm$ 0.6 | 88.2<br>$\pm$ 0.1 | \ul37.4<br>\ul$\pm$ 0.2 | 1.3<br>$\pm$ 0.0 | 66.7<br>$\pm$ 5.9 | 41.2<br>$\pm$ 2.3 | 4.4<br>$\pm$ 0.7 | 67.8<br>$\pm$ 1.9 | 50.8<br>$\pm$ 1.3 | 3.7<br>$\pm$ 0.3 |
| Central. | 84.9<br>$\pm$0.5 | 54.8<br>$\pm$0.5 | 1.6<br>$\pm$ 0.1 | 71.4<br>$\pm$5.0 | 55.2<br>$\pm$2.9 | 3.8<br>$\pm$ 0.6 | 84.1<br>$\pm$1.6 | 36.5<br>$\pm$1.1 | 1.7<br>$\pm$ 0.2 | 72.2<br>$\pm$3.7 | 41.5<br>$\pm$1.3 | 3.6<br>$\pm$ 0.3 | 80.0<br>$\pm$2.1 | 63.2<br>$\pm$2.8 | 2.3<br>$\pm$ 0.2 |

Figure: (a) F1-Score for each label
Refer to caption: x7.png

## Appendix D Fed-ECHO

### D.1 Description

Fed-ECHO consists of three datasets: CAMUS, ECHONET-DYNAMIC, and HMC-QU. The overview of Fed-ECHO is shown in Table 5.

##### CAMUS.

This database consists of clinical exams from 500 patients, acquired at the University Hospital of St Etienne (France). All images are labeled with three areas: endocardium of the left ventricle ($\rm LV_{Endo}$), epicardium of the left ventricle ($\rm LV_{Epi}$), and left atrium wall ($\rm LA$). The image size varies from $584\times 354$ to $1945\times 1181$.

##### ECHONET-DYNAMIC.

This database contains 10,0230 echocardiogram videos where two frames are annotated with only $\rm LV_{Endo}$ area. All frames are resized to $112\times 112$.

##### HMC-QU.

This database contains 109 echocardiogram videos collected at the Hamad Medical Corporation Hospital in Qatar. The frames of one cardiac cycle in each video are annotated with $\rm LV_{Epi}$ area. The video frame size varies from $422\times 636$ to $768\times 1024$ while all labels are resized to $224\times 224$.

### D.2 License and Ethics

Both CAMUS and HMC-QU datasets are open-access. HMC-QU database requires the user to have a Kaggle account, while the ECHONET-DYNAMIC database requires the user to have a Stanford AIMI account and to accept its agreement. It is licensed under the Stanford University Dataset Research Use Agreement.

### D.3 Download and preprocessing

#### D.3.1 Download

The three datasets can be downloaded using the URLs below:

- 1.
CAMUS: [https://humanheart-project.creatis.insa-lyon.fr/database/#collection/6373703d73e9f0047faa1bc8](https://humanheart-project.creatis.insa-lyon.fr/database/#collection/6373703d73e9f0047faa1bc8)
- 2.
ECHONET-DYNAMIC: [https://echonet.github.io/dynamic/index.html#access](https://echonet.github.io/dynamic/index.html#access)
- 3.
HMC-QU: [https://www.kaggle.com/datasets/aysendegerli/hmcqu-dataset/data](https://www.kaggle.com/datasets/aysendegerli/hmcqu-dataset/data)

#### D.3.2 Preprocessing

Raw echocardiograms have varying frame sizes, modalities, and mask labels, which must be standardized before training. Therefore, as a first step, we extract frames that are annotated and store them as images. We then resize them to a common ($112\times 112$) shape. Finally, we align the labels of records in different databases. We use 1, 2, 3 representing $\rm LV_{Endo}$, $\rm LV_{Epi}$ and $\rm LA$ respectively. The samples of Fed-ECHO are shown in Figure7.

Figure: (a) Sample from Institution 1.
Refer to caption: x10.png

### D.4 Baseline, loss function and evaluation

##### Baseline Model.

A U-net architecture is employed in this study, utilizing echocardiographic images as input to forecast masks delineating four distinct cardiac regions. The U-net model represents a conventional convolutional neural network design frequently deployed in the realm of biomedical image segmentation endeavors. Its application is tailored towards semantic segmentation, a process wherein individual pixels within an image are categorized based on semantic content.

##### Loss function.

We use a CrossEntropy Loss (CELoss) for training. Note that, for centralized supervised learning and client training in FedAvg, FedProx, Scaffold, and Ditto strategies, we ignore label with value 0 when calculating CELoss for data from client 2 or 3, since region with label 0 may not be true ground truth in these clients.

##### Evaluation Metrics.

We use the Dice similarity index and 2D Hausdorff distance ($d_{H}$) to measure the accuracy of the segmentation output. Dice index is calculated as:

$$ $\mathrm{DICE}(\mathbf{y},\hat{\mathbf{y}})=\frac{2\sum_{i=1}^{n}y_{i}\hat{y}_{ i}}{\sum_{i=1}^{n}y_{i}+\sum_{i=1}^{n}\hat{y}_{i}}$ (4) $$

The Hausdorff distance is calculated as:

$$ $\mathrm{d_{H}}(\mathbf{y},\hat{\mathbf{y}})=\max\{d(\mathbf{y},\hat{\mathbf{y} }),d(\hat{\mathbf{y}},\mathbf{y})\},$ (5) $$

where $d(\mathbf{y},\hat{\mathbf{y}})$ represents the minimum distance among points at the edge of $\mathbf{y}$ and points at the edge of $\hat{\mathbf{y}}$.

Note that, to better measure the model segmentation performance, for clients 2, and 3, we select only 200 labeled frames for testing.

### D.5 Training Detail

##### Optimization parameters.

We optimize our model using the SGD optimizer, with a batch size of 32. We train our model for 50 epochs on one NVIDIA A100-PCIE-40GB.

##### Hyperparameter Search

For centralized and local model training, we first explore learning rates from the set {1e-4, 1e-3, 1e-2, 1e-1.5, 1e-1} during centralized model training. The learning rate that achieves the best Dice index is then utilized for local model training. For the federated learning strategies, we employ the following hyperparameter grid:

- •
For clients’ learning rates (all strategies except Fed-Consist): {1e-4, 1e-3, 1e-2, 1e-1.5, 1e-1}.
- •
For server size learning rate (Scaffold strategy only): {1e-2, 1e-1, 1.0}.
- •
For FedProx and Ditto strategies, the parameter $\mu$ is selected from {1e-2, 1e-1, 1.0}.
- •
For FedInit, the parameter $\beta$ is chosen from {1e-1, 1e-2, 1e-3}.
- •
For FedSM, the parameters $\gamma$ and $\lambda$ are set to {0, 0.1, 0.7, 0.9} and {0.1, 0.3, 0.5, 0.7, 0.9}, respectively.
- •
For FedALA, the parameters layer index, $\eta$, threshold, and num_per_loss are fixed at 1, 1.0, 0.1, and 10, respectively, while rand_percent is chosen from {5, 50, 80}.

For Fed-Consist, we introduce Gaussian noise with a variance of 0.1 as augmentation. The learning rates for labeled clients are searched from {1e-2, 1e-3, 1e-4}, while those for unlabeled clients are explored within {1e-3, 1e-4, 1e-5, 5e-6, 1e-6}. The parameter $\tau$ is varied from {0.5, 0.7, 0.9}.

Additionally, for FedPSL, we further search the parameters $\alpha$ and $\beta$ from {1e-0.5, 1e-1, 1e-1.5, 1e-2, 1e-3} and {1e-1, 1e-1.5, 1e-2, 1e-3, 1e-4, 1e-5}, respectively. The optimal values found are $\alpha=1e-1.5$ and $\beta=1e-5$.

**Table 10: Hyperparameters used for the Fed-ECHO.**
| Fed-ECHO |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | learning rate | optimizer | learning rate server | mu | beta | lambda | gamma | rand_percent | $\tau$ |
| Central.(sup) | 0.1 | torch.optim.SGD | - | - | - | - | - | - | - |
| Central.(ssup) | 0.1 | torch.optim.SGD | - | - | - | - | - | - | - |
| FedAvg | 0.1 | torch.optim.SGD | - | - | - | - | - | - | - |
| FedProx | 0.1 | torch.optim.SGD | - | 0.1 | - | - | - | - | - |
| Scaffold | 0.1 | torch.optim.SGD | 1.0 | - | - | - | - | - | - |
| FedInit | 0.1 | torch.optim.SGD | 1.0 | - | 1e-2 | - | - | - | - |
| Ditto | 0.1 | torch.optim.SGD | - | 0.1 | - | - | - | - | - |
| FedSM | 0.1 | torch.optim.SGD | 1.0 | - | - | 0.1 | 0 | - | - |
| FedALA | 0.1 | torch.optim.SGD | 1.0 | - | - | - | - | 5 | - |
| FedPSL | 0.1 | torch.optim.SGD | 1.0 | - | 1e-5 | - | - | - | - |
| Fed-Consist | 0.0001(labeled client)<br>1e-6(unlabeled client) | torch.optim.SGD | - | - | - | - | - | - | 0.9 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see above
soundness: 3
strengths: - The paper aggregates several publicly available datasets that are useful for advancing research in this area in future work. 
- The repository is well organized and easy to navigate.
- The analysis is thorough and spans three different challenges related to FL (long-tail distribution, label incompleteness, and non-IID data).
- The authors include several SOTA baselines and describe their hyperparameter tuning strategy. 
- They also consistently provide confidence intervals across all results tables. 
- The figures and tables are neat and easy to understand.
summary: In this paper, the authors present a benchmark for federated learning applied to two cardiovascular disease related modalities: ECG and ECHO. The ECG dataset comprises 4 publicly available datasets, while the ECHO dataset comprises 3 publicly available datasets. The authors evaluate several existing SOTA approaches, including FedAvg, FedProx, Scaffold, FedInit, Ditto, FedSM, and FedALA, and investigate several settings: local vs global test set performance, long-tail challenge, non-IID challenge, and label incompleteness.
weaknesses: - Can you clarify the key differences and potential impact of your benchmark compared to existing work like Goto et al. 2022. Can you also discuss how the focus on long-tailedness and incomplete labels advances the field beyond prior benchmarks?
- To improve the paper's accuracy, can you change the title so that it better reflects the scope, e.g. "FedCVD: A Federated Learning Benchmark for ECG Classification and Echocardiogram Segmentation"?
- The paper requires significant proofreading, i.e. inconsistencies with abbreviations / capitalization / spelling mistakes.
- The abstract should make it clear that the benchmark is using publicly available datasets, rather than presenting new datasets. 
- The API functions in Figure 1 are more like functions in their repository so I'm not sure if the term API is the best fit description. 
- The dataset sizes and characteristics should be summarized in the main text.
- To better interpret the results, could you discuss why certain FL methods performed better on the long-tail challenge or label incompleteness scenarios?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section
soundness: 3
strengths: 1/ The manuscript was reasonably written. Most of the key concepts were explained clearly.

2/ Although there is some related work to evaluate the performance federated learning algorithms over multiple ECG datasets, this can be the first benchmarking exercise that have conducted  detailed and extensive experiments to evaluate seven FL algorithms over both the ECG-based arrhythmia classification task and also the ECHO video segmentation task.

3/ Major datasets for both tasks that are publicly available are included.
summary: This is a benchmark study aiming to evaluate the performance of the commonly used Federated Learning strategies for both the ECG-based arrhythmia classification task and also the echocardiogram video segmentation task. The study specifically focused to investigate the impact from non-IID in data from different sites, the long-tail distribution (where distributions among labels are very imbalanced) and the label incompleteness.

Four ECG datasets and three echo datasets were included for the benchmarking exercise. Seven federated learning algorithms were included, and their performances were compared with the one from a centralized scenario and the ones from individual sites.
weaknesses: 1/ As the authors also mentioned, the reported study is closely related to the Goto et al 2022 (https://www.ahajournals.org/doi/10.1161/CIRCULATIONAHA.121.058696) study. Although the author did mentioned about that study in the manuscript, think a more detailed explanations on how different and the new contributions of this work compared to the Goto study is still needed. Would suggest the authors to clearly compared the datasets included in both studies, the FL algorithms included, the different evaluation metrics measured, and this work's unique emphasis on non-IID in data, the long-tail distribution and the label incompleteness.

2/ For the ECG-based arrhythmia classification study, ResNet was chosen to be the base model. However, the authors did not explain the rational behind why model was chosen to be the only ONE base model for this study. Simply citing a previous paper won't justify that. Could explain what was the rational in choosing the current model?

More importantly, to report a more robust benchmarking study, suggest the author may investigate how different base models may or may not impact the performance of the included FL algorithms over the benchmarking datasets. It will useful to investigate why there may or may not be an impact?

Therefore, suggest the authors to include other base models too. Believe some customized LSTM and Transformers have been developed for ECG classification as well.  Would it be possible to include them as base models too?

3/ Similar to point 2, 2-D Unet was chosen to be the only base model in this study. Again, could explain the rationale behind picking the current base model?

Additionally, it will be also interesting to study how different segmentation base models may have different strengths in addressing the non-IID in data, the long-tail distribution and the label incompleteness challenges.

Therefore, suggest the authors to consider including one or two more base models?

Some of these possibilities could be:
Leclerc, Sarah, et al. "Deep learning for segmentation using an open large-scale dataset in 2D echocardiography." IEEE transactions on medical imaging 38.9 (2019): 2198-2210.

Wu, Huisi, et al. "Super-efficient echocardiography video segmentation via proxy-and kernel-based semi-supervised learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.

Yang, Jiewen, et al. "GraphEcho: Graph-Driven Unsupervised Domain Adaptation for Echocardiogram Video Segmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

and 3D models e.g.

Hatamizadeh, Ali, et al. "Unetr: Transformers for 3d medical image segmentation." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2022.

4/ For ECG datasets, would it make sense to include the Icentia11K dataset too? Although it was build mainly for self-supervised algorithms, part of the datasets do have labels? Additionally, it helps to expand the study's technical coverage if the team can expand their work to investigate how the FL learning mechanism can be combined with the newly proposed self-supervised learning models over CVD datasets.

Tan, Shawn, et al. "Icentia11k: An unsupervised representation learning dataset for arrhythmia subtype discovery." arXiv preprint arXiv:1910.09570 (2019).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: Could you please clarify what aspects make this benchmark-focused paper suitable for this conference, particularly in terms of its contributions to evaluating real-world healthcare data?
soundness: 4
strengths: The primary strength of this paper is its systematic approach to evaluating CVD-related tasks using federated learning. It effectively addresses the practical challenges of Non-IID and long-tail distributions in healthcare data, which are critical issues for achieving robust model performance. Non-IID data distribution, particularly when labels are collected independently from multiple institutions rather than through a consortium, introduces complexities in model evaluation. The paper adeptly tackles this challenge, offering a realistic evaluation approach that reflects the heterogeneous data across different institutions, thus enhancing the credibility of the proposed FL setting.

Moreover, the long-tail challenge is approached innovatively using a Top-K metric, which allows for detailed performance evaluation of model accuracy across classes with varying sample distributions. This approach provides valuable insights into the model’s handling of label imbalance in real CVD data. Additionally, FedCVD rigorously analyzes various FL methodologies, demonstrating how each method performs under the specific characteristics of CVD data. With open-source code available, it contributes a practical and reproducible resource for FL research in healthcare.
summary: This paper proposes FedCVD, the first real-world FL benchmark for CVD diagnosis. By utilizing FL, it aims to enhance the accuracy of CVD diagnosis while preserving data privacy, enabling collaborative model training across multiple institutions. Traditional FL research has largely relied on simulated data, often failing to capture the complexity of real-world data distributions. To address this limitation, FedCVD incorporates two primary tasks, ECG classification and ECHO segmentation, based on real ECG and ECHO data collected from seven institutions, reflecting the challenges of federated learning in a real-world healthcare setting.
weaknesses: A limitation of this paper is that, in cases of label-non-IID, the increased label sparsity across institutions may pose challenges to quantitative evaluation. As the label distribution varies more widely across institutions, certain labels may not be adequately represented, potentially compromising evaluation reliability.

Another significant limitation is whether the domain of this study aligns with the conference scope. Despite the academic value of the proposed framework, it may not be fully aligned with the themes of the conference, potentially impacting its acceptance and reception.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Major suggestions: 
- Variability in the algorithms' performance estimates seemed quite substantial at times (e.g., Table 2 GLOBAL Mi-F1 FedAvg 67.9+/-3.8 compared to the leading Scaffold 70.1+/-0.8) , making it unclear whether the differences between algorithms are actually meaningful or just due to chance. I was unable to find information on how variability was calculated. Please include that information and consider adding formal tests for differences. 
- It is not immediately obvious to me how some algorithms like Ditto in Table 2 can have a much lower Mi-F1 in the GLOBAL set (68.1) than in any single local set (min 73.4). Do they authors have an intuitive explanation for this?
- I was unable to find any details on the semi-supervised algorithms for the ECHO data. Please add a clear description of what was done in those cases. 

Minor suggestions: 
- it might be worth replacing Client 1-4 with the actual dataset names to make sure which local performance is shown.
soundness: 3
strengths: Medical machine learning still lags behind many other domains, in part owing to the difficulties of collecting sufficiently large and diverse data. Federated learning is increasingly important to address this challenge and allow the training of larger models in secure and privacy-preserving manner. However, studies that evaluate the performance of different FL algorithms rely on artificially federated data that does not reflect conditions encountered in the wild. This work introduces two complex datasets that can help facilitate more realistic benchmarking of FL algorithms.
summary: This work collates and harmonises 7 open-access cardiology datasets (4 ECG, 3 ECHO) to create a realistic benchmark for federated learning. The authors show that existing federated learning benchmarks may not adequately reflect real-world challenges in cardiovascular research: 1) natural partitions are needed to reflect inter-institution heterogeneity; 2) clinical outcomes of interest are often not binary but multi-class and long-tailed; 3) labelling may differ significantly between
weaknesses: The paper tries to do many things at once, which makes it difficult to read at times. Here are some examples: 
- Scope: while the work presents itself as a benchmark for CVD, in terms of modalities it contains separate benchmarks for a time series (ECG) and 2D images/videos (ECHO). The benchmarks in Table 1 span additional modalities including tabular data (FedTD), time-to-event data (Flamby), graphs (FedDTI), NLP (FEDLEGAL), and multimodal data (FedMultimodal). It is unclear whether they are even comparable. 
- Related work: the related work section does not have a clear logical flow. As far as I can see, the main arguments are: 1) AI is important in cardiology and spans ECGs and images among other modalities (could be argued more concisely); 2) most existing applications are trained on single centre data but multicentre data is needed for reasons XYZ (currently not clear why multicentre data is needed); 3) federated learning provides a privacy-preserving way to share data between institutions but most studies to date only show proof-of-concept with artificially federated data; 4) realistic benchmarks for FL are needed (why?) and some work has been done in that direction, but they do not completely consider the challenges faced in FL for CVD. Section headings may help to guide the reader through this section. 
- The Proposed FedCVD: This section introduces the datasets, metrics, baseline models, and challenges. However, it does not actually introduce the framework. Here I would expect to see the flow of an experiment, how the framework/API supports adding new datasets or models, etc. The metrics and baseline models, on the other hand, belong into the experiment section and are indeed repeated there.
- Experiments: The current version is structured by dataset (Fed-ECG and Fed-ECHO). It might make it more readable if the authors group the results by challenges (non-IID, long-tail, incomplete) instead. The experiment section also mixes methods with results. For example, on page 8 F1-STD and Top-K are introduced right next to the discussion of the results. Consider explaining your metrics earlier in the experiment details section.

Meta-review: The paper introduces a benchmark for federated learning for cardiovascular disease data, including ECG classification and echo segmentation. The benchmark uses several already-available public datasets and analyzes long-tail behavior and label incompleteness.


Strengths:
 - The benchmark uses public datasets to demonstrate the performance of federated learning on common data challenges in CVD settings
 - The authors were very responsive during the discussion. They actively performed new experiments and updated confusing sections of the manuscript.

Weaknesses
 - The largest weakness is that it is unclear how this fits with the ICLR conference scope. The specificity of the paper on one narrow set of FL algorithms on two CVD tasks means that it is not clear how much the larger ICLR community will benefit. For example, if researchers are similarly interested in non-IID labels in federated learning settings, it is not obvious whether this paper has generalizable insights.
 - Additionally, reviewers consistently raised concerns about clarity issues in the work. The lack of detailed analysis about comparisons to prior work (e.g., Goto et al, 2022) was brought up several times. 
 - There were also several issues about the definitions of variability, the SSL experiment setup, the limited number of architectures examined, and the lack of details in the framework of the experiment setup.


I applaud the authors for extensive responses to reviewer concerns, including entirely new Transformer-based architecture experiments. However, the fit with ICLR and the extent of modifications needed weighed heavily in my decision.

Decision: Reject