JMIR FORMATIVE RESEARCH

Pirmani et al

Original Paper

Accessible Ecosystem for Clinical Research (Federated Learning
for Everyone): Development and Usability Study
Ashkan Pirmani1,2,3,4, MSc; Martijn Oldenhof1, MSc; Liesbet M Peeters2,3,4, PhD; Edward De Brouwer1*, PhD; Yves
Moreau1*, PhD
1ESAT-STADIUS, KU Leuven, Leuven, Belgium

2

Data Science Institute, Hasselt University, Diepenbeek, Belgium

3

University Multiple Sclerosis Center, Hasselt University, Diepenbeek, Belgium

4

Biomedical Research Institute, Hasselt University, Diepenbeek, Belgium*

these authors contributed equally

Corresponding Author:
Yves Moreau, PhD
ESAT-STADIUS
KU Leuven
Kasteelpark Arenberg 10
Leuven, 3001
Belgium
Phone: 32 16 32 86 45
Email: Yves.Moreau@esat.kuleuven.be

Abstract
Background: The integrity and reliability of clinical research outcomes rely heavily on access to vast amounts of data. However,the fragmented distribution of these data across multiple institutions, along with ethical and regulatory barriers, presents significant
challenges to accessing relevant data. While federated learning offers a promising solution to leverage insights from fragmented
data sets, its adoption faces hurdles due to implementation complexities, scalability issues, and inclusivity challenges.
Objective: This paper introduces Federated Learning for Everyone (FL4E), an accessible framework facilitating multistakeholder
collaboration in clinical research. It focuses on simplifying federated learning through an innovative ecosystem-based approach.
Methods: The “degree of federation” is a fundamental concept of FL4E, allowing for flexible integration of federated andcentralized learning models. This feature provides a customizable solution by enabling users to choose the level of data
decentralization based on specific health care settings or project needs, making federated learning more adaptable and efficient.
By using an ecosystem-based collaborative learning strategy, FL4E encourages a comprehensive platform for managing real-world
data, enhancing collaboration and knowledge sharing among its stakeholders.
Results: Evaluating FL4E’s effectiveness using real-world health care data sets has highlighted its ecosystem-oriented andinclusive design. By applying hybrid models to 2 distinct analytical tasks—classification and survival analysis—within real-world
settings, we have effectively measured the “degree of federation” across various contexts. These evaluations show that FL4E’s
hybrid models not only match the performance of fully federated models but also avoid the substantial overhead usually linked
with these models. Achieving this balance greatly enhances collaborative initiatives and broadens the scope of analytical possibilities
within the ecosystem.
Conclusions: FL4E represents a significant step forward in collaborative clinical research by merging the benefits of centralized
and federated learning. Its modular ecosystem-based design and the “degree of federation” feature make it an inclusive, customizable
framework suitable for a wide array of clinical research scenarios, promising to revolutionize the field through improved
collaboration and data use. Detailed implementation and analyses are available on the associated GitHub repository.
(JMIR Form Res 2024;8:e55496) doi: 10.2196/55496
KEYWORDS
federated learning; multistakeholder collaboration; real-world data; integrity; reliability; clinical research; implementation;
inclusivity; inclusive; accessible; ecosystem; design effectivenesshttps://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e55496 | p. 1
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Introduction
Real-World Data in Health Care: Toward Federated
Learning Solutions
Real-world data (RWD), which encompass clinical and healthdata, are a fundamental asset in health care, providing invaluable
insights for research and clinical applications [1,2]. However,for privacy, legal, collaborative, or practical reasons, RWD are
typically fragmented into different data sources, such as
electronic health records, patient registries, and other clinicalrepositories. This fragmentation, characterized by the dispersion
of RWD across numerous health care systems and geographic
locations,hampers
theFindability,
Accessibility,
Interoperability, and Reusability (“FAIRness”) of the RWD [3].
Thus, the capacity to harness these fragmented data sources forevidence-based health care decision-making is hindered andmust be improved [4,5].
While limited data sharing impedes the clinical research progressin general, the impact is particularly significant for research
into low-prevalence diseases, where data scarcity already posesa challenge [6]. This issue becomes especially critical in urgent
situations, such as the COVID-19 pandemic [7,8], where rapid
data sharing is vital to quickly understand the impact of a novel
disease.
Nevertheless, many challenges stemming from RWD
fragmentation are systemic and can be seen as constraints thatare difficult to eliminate. Legal data protection frameworks,
ethical considerations, and organizational policies of different
countries often limit the dissemination of such data both within
and across countries [9-12]. These challenges are further
complicated by different data registries using different data
formats, standards, and infrastructures to reflect their specificneeds [13]. This context raises the need for a global data sharing
framework to facilitate the secure and efficient sharing of RWD,
while accommodating most of the complexities of the clinicaldata ecosystem.
A key component of such a global data sharing framework isfederated learning (FL), which offers a solution by training
machine learning models on distributed data sets without
centralization [14,15]. Nevertheless, current FL packages crucial
for implementing this system fail to cater to the multifaceted
requirements of health care stakeholders.

Related Works
The FL framework ecosystem is characterized by its
heterogeneity, prompted by contributions from both the
open-source and industry communities, with each frameworkcasted to address specific facets of FL’s inherent challenges.
From enhancing security and privacy to ensuring scalability
and promoting framework agnosticism to facilitating ease ofuse and customization, these frameworks collectively foster a
conducive environment for the flourishing of FL applications.
Industry solutions such as IBM FL [16], the Clara Training
Framework [17], and Sherpa.ai [18] offer specialized features
but suffer from restricted accessibility due to their proprietarynature, limiting widespread application and innovation.
https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et alOpen-source FL frameworks, despite their appeal of broader
accessibility, grapple with challenges unique to their design andintended use cases. Syft, also known as PySyft [19], in thecontext of offering advanced privacy-enhancing technologies,
focuses on secure and private data analysis across various
domains. Despite it emphasizes structured transparency systems,
the complex setup process associated with Syft could impede
its broad adoption. As a pioneering framework in the realm ofprivacy technology, Syft has undergone substantial revisions
across its iterations. While aimed at improvement, thesecontinuous overhauls may pose challenges for early adoptersin terms of staying abreast with the framework’s evolution.
Frameworks such as FATE [20] and OpenFL [21] aredistinguished by their strong emphasis on security and privacy,
which are critical in sectors such as health care, where data
confidentiality is essential. FATE is noted for its robust security
and privacy features. However, it requires substantial
infrastructural investment and has a steep learning curve, as
there are some discrepancies in documentation clarity andavailability across different parts of the framework. OpenFL
also offers Intel-enabled trusted execution environment known
as Safe Guard and mutual Transport Layer Security for allcommunications to ensure maximum privacy. However,
implementing this framework on a broader scale is hindered
and may necessitate specific hardware, software, or theactivation of certain services [22]. TensorFlow Federated [23],while offering a specialized toolkit for TensorFlow models,limits its use to those committed to a singular modelingframework, potentially sidelining a broader audience that
operates across diverse machine learning frameworks.
Flower [24] and FedLab [24] are recognized for their frameworkagnosticism and scalability. Flower emerges as a user-friendly
and lightweight option, albeit still progressing through its
developmental stages with real-world applications yet to be
fully demonstrated. This reflects a common developmental
trajectory among open-source FL frameworks, where achieving
a balance between accessibility, ease of use, and broad
applicability poses a significant challenge.
FedML [25] distinguishes itself through its versatility and recent
architectural improvements, positioning it alongside frameworks
such as Plato [26], known for supporting a diverse array ofcomputing paradigms. This dual focus on facilitating FL
research and the development of practical applications
underscores its strengths. However, integrating significant
updates may necessitate substantial maintenance for applications
developed on its platform. Additionally, developers may needto reacquaint themselves with the framework’s updated features,presenting a potential challenge in adaptability and ongoingengagement.
Innovative approaches are evident in PaddleFL [27] and Galaxy
FL [28], with Galaxy FL pioneering blockchain technology
integration for improved data privacy and model confidentiality.
This innovation broadens the FL framework landscape,introducing novel data protection and utilization aspects.
However, blending FL with blockchain remains a debated topic
within the literature, especially concerning the significant
computation and communication overheads [29].
JMIR Form Res 2024 | vol. 8 | e55496 | p. 2
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Pirmani et al

Lastly, FLSim [30] specializes in FL simulations, providing acomprehensive framework that incorporates differential privacy
and secure aggregation scenarios. Similarly, Leaf [31] focuses
primarily on simulation studies. While this specialization insimulation offers valuable insights, it may also narrow theirapplicability, potentially restricting their direct translation into
real-world deployments.

•

For a more in-depth discussion of the FL frameworks, we guidereaders to Multimedia Appendix 1, where Table S1 in
Multimedia Appendix 1 [32-51] offers a comprehensive
summary of the highlights and key features of various FL
frameworks. Additionally, Table S2 in Multimedia Appendix
1 details the developmental progress and recent advancements
within these frameworks, providing a clear snapshot of theirevolution and current state.

Methods

Navigating the Path Forward: The Need for an
Accessible Ecosystem for Clinical Research
Beyond the challenges already covered, a recent review of FL
in the biomedical domain reveals that only some research studies
have provided reusable or reproducible source code alongside
their research [52]. This highlights a common issue: thetendency to start a unique project for each separate data sharing
initiative. This practice not only consumes valuable time andresources but also hinders the evolution of a universal
ecosystem. Furthermore, the opportunity to build upon andcontribute to an existing initiative is often overlooked.
Despite recent progress in the field, there is a critical demand
for a more inclusive, user-friendly, and holistic framework inhealth care collaborative research [53,54]. A framework thatempowers stakeholders from diverse backgrounds to fully
engage in collaborative learning opportunities is essential. Thisrequires FL solutions characterized by intuitive interfaces,
streamlined workflows, comprehensive documentation, and adesign process that actively seeks and incorporates feedback
from a wide range of stakeholders. By prioritizing usability andinclusivity in developing FL solutions, we can significantly
lower technical barriers. This approach enables wider
participation and amplifies the contributions of collaborative
health care research.
In this work, we present Federated Learning for Everyone(FL4E)—a dynamic, data-centric framework designed withinclusivity and adaptability at its core. FL4E is dedicated toefficiently converting RWD into actionable evidence, facilitating
engagement from a broad spectrum of stakeholders, andenhancing collaborative learning experiences. Its hierarchicalarchitecture offers a customizable data sharing ecosystem,
facilitating a tunable balance between centralization andfederation. With its modular approach, FL4E is general andprovides an ecosystem-based environment that can support awide range of collaborative research projects. This modularity
further allows all stakeholders to contribute to the project byselectively engaging in the relevant part of the data sharing
pipeline.

Key Contributions•

RenderX

Overview
In this section, we rigorously investigate the fundamental
principles underlying FL4E, systematically articulating its core
concepts. This entails a meticulous examination of the pivotal
modules and essential components that constitute FL4E, witha particular focus on their functionalities and interrelationships.
Moreover, this section highlights the architectural nuances of
FL4E, comprehensively detailing its structural configurationand the dynamics of its operational framework.

Key Concepts of FL4E
First, the concept of “degree of federation” within the FL4E
framework introduces a novel, adaptable, and pragmatic
approach to collaborative data analysis designed to meetstakeholders’ varied needs and limitations. This feature enables
participants to customize their level of involvement and data
sharing based on their specific infrastructural capacities,
regulatory requirements, and privacy considerations.
FL4E acknowledges that while federated models offer
significant privacy advantages, their practicality may vary due
to different constraints. Therefore, FL4E is designed toaccommodate various levels of participation for stakeholders,
ensuring flexibility if one data sharing stream proves unfeasible.
Therefore, the choice between a centralized or federated model
within FL4E is presented not as a dichotomy but as a continuum.
On the one hand, centralization may simplify operations andenhance efficiency, but it could compromise privacy to somedegree. Conversely, full federation prioritizes privacy, possibly
adding operational and analytical complexity. FL4E’s design
empowers stakeholders to find a balance that best suits theirneeds, governance frameworks, and regulatory environments.
This adaptable strategy has proven effective in accommodating
diverse data sharing needs, as demonstrated by the Global Data
Sharing Initiative for Multiple Sclerosis and COVID-19 [55,56].
The deployment of a versatile hybrid data acquisition system,
merging traditional data sharing with federated methods, has
not only broadened inclusivity and flexibility but also led to anotable (3527/7757, 45.47%) increase in data collection volume
compared to initial efforts [57]. These results underscore the“degree of federation” practical value, showcasing theadvantages of integrating such adaptability into data acquisitionand sharing frameworks. Different configurations within the
degree of federation are depicted in Figure 1.

We propose a unique FL framework designed to cater tothe diverse needs of health care stakeholders and to adapt

https://formative.jmir.org/2024/1/e55496

XSL• FO•to various study designs. This design is further enhanced
by offering a tunable balance between centralization andfederation.
We formalize this balance by introducing the concept of“degree of federation.”
Our evaluation of the framework, using 2 real-world data
sets, underscores the value of federation. Specifically,
hybrid experiments conducted at different degrees offederation deliver performance comparable to fully
federated setups.

JMIR Form Res 2024 | vol. 8 | e55496 | p. 3
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Pirmani et al

Figure 1. The degree of federation characterizes the balance between fully centralized and fully federated setups, leading to hybrid solutions wheresome of the stakeholders centralize their data while others prefer a federated approach. FL4E: Federated Learning for Everyone.

However, it is essential to emphasize that the framework fosters
regulatory compliance across all participation methods. Whether
stakeholders prefer the simplicity and direct oversight of a more
centralized model or the enhanced privacy of a federated
approach, stakeholders are reminded of their collective
obligation to adhere to prevailing data protection regulations.
Second, ecosystem-based collaborative learning: FL4E
framework introduces a paradigm shift from conventional
project-specific FL to an “ecosystem-based” approach,underscoring a strategic pivot toward creating a unified,adaptable platform tailored for conducting various analyses
centered on specific diseases or health conditions. At the heartof the “ecosystem-based” methodology is its holistic perspective,
aiming to formulate a comprehensive and synergistic
environment for data sharing and collaborative research.
An ecosystem, in this context, is envisioned not just as acollection of data or series of disjointed projects but as adynamic, collaborative network of stakeholders. This includes
researchers, health care professionals, patients, and policy
makers, all united by a common goal: understanding andaddressing a particular health issue. By fostering such an
ecosystem, FL4E encourages the pooling of diverse resources,
expertise, and perspectives. This collaborative efficiency breaks
down traditional silos, allowing for more efficient use ofresources through shared data and findings.
Moreover, encompassing a wide array of stakeholders andresearch questions enables the ecosystem to provide a morenuanced and comprehensive understanding of the healthhttps://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

condition under investigation. The diversity of perspectives andexpertise enriches the research outcomes, offering holistic
insights far beyond what could be achieved in isolation.
This approach is inherently adaptable and capable of evolving
in response to new research findings, emerging healthchallenges, and shifts in the regulatory landscape. Suchflexibility ensures the long-term sustainability and relevance ofthe ecosystem, enabling it to continue generating valuable
insights over time.
Regarding its structure, FL4E features a nested design. Withineach ecosystem, several studies are carried out, each focusing
on specific research questions related to the overarching theme.
These studies, while operating under their unique regulatory
and consent protocols, incorporate the degree of federation.
They can include a data set, data quality scripts, and analyses
tailored to their specific research queries.
FL4E’s structure also incorporates simplicity and accessibility
in its design. The intuitive user interface encourages
contributions from stakeholders, fostering a collaborativeenvironment that is integral to the ecosystem’s success. Thissystematic approach to collaboration and data sharing is visually
depicted in Figure 2.
The FL4E framework builds upon the foundational concepts of
the Flower framework [32]. Flower’s flexibility, support for
multinode execution, and agnosticism to both machine learning
frameworks and programming languages significantly contributeto the overall adaptability and versatility of FL4E.

JMIR Form Res 2024 | vol. 8 | e55496 | p. 4
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Pirmani et al

Figure 2. FL4E’s nested approach schema: the schema presents the ecosystem approach of the FL4E, facilitating the incorporation of diverse studies
framed by different research questions. It further highlights the support for a wide range of analyses, leading to the execution of statistical experiments.
FL4E: Federated Learning for Everyone.

Modules and Functionality
Overview
This section provides an in-depth overview of the core modules
within the FL4E framework: the Study Center, Repository
Center, Model Center, and Data Center. Each module’s role
and anticipated functionality are detailed, showcasing how thedivision into distinct centers contributes to effectively deploying
the overarching ecosystem. Figure 3 complements this byvisually representing the architectural arrangement of serverand client components. By detailing the anticipated interactions
among stakeholders and illustrating the operational flow within
the framework, the architecture provides a solid foundation forunderstanding the potential and practical application of FL4E
in advancing health care data-driven research and insights.

Figure 3. The high-level architecture of FL4E: This figure showcases a comprehensive framework designed to accommodate the nuanced interactionsof diverse stakeholders. The architecture includes the expected user stories across 3 primary categories of participants: data providers, data scientists,and downstream users. Each stakeholder group engages with FL4E platform through distinct pathways. The architecture diagram features a table
outlining the unique interactions of the stakeholders, mapping their respective roles and activities within the FL4E framework. This detailed mapping
clearly explains how each stakeholder contributes to and benefits from the FL4E, highlighting the platform's versatility and user-centric design. At thecore of our architecture lie 3 fundamental components: the server, client, and executor machine, each vital to the execution of FL tasks. The diagram
we provide elucidates their interconnected roles, showcasing the seamless flow of data, scripts, and analytical results across the system. The server acts
as the orchestrator for FL tasks, hosting the primary web application within a Docker container as an ASP.NET application. It securely manages theplatform's data, housed on an SQL server in a dedicated hosting environment. This component is crucial for coordinating tasks and uses a Python
environment to manage secure data sharing and preprocessing of “data center” module of the framework. On the client side, implemented as a Docker-based
image, it runs on the data contributor's machine. This component is essential for integrating RWD into the FL process. Developed using Python and
ASP.NET for web applications, the client-side component establishes a connection to the executer machine. On the other hand, the executor machineplays a crucial role in conducting the analysis. It is designed to receive client updates. This adaptable component configuration allows data scientists
to tailor it according to their specific analytical needs and preferences. FL: federated learning; FL4E: Federated Learning for Everyone; RWD: real-world
data.

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e55496 | p. 5
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Study Center
The Study Center is a critical piece in the ecosystem design ofthe framework. It coordinates research activities via the Study
Catalogue and the Analysis Center, highlighting theinterconnectivity and synergy within the system.
The Study Catalogue functions as a cooperative repository where
various stakeholders can share and access overarching
information or metadata about different studies. Collaborations
can be initiated by a data scientist with an intriguing researchquestion but needing more data or by a data provider possessing
the required data but lacking the analytical tools. Downstream
users can further enrich the catalogue by contributing scientificarticles or proposing research questions relevant to otherstakeholders. Any new entry to the catalogue should include
the title, a brief description, the study lead or author, and thecurrent status of the entry (ie, active or not). If the entry involves
any data, this should also be noted, as the data would be
subsequently managed within the Data Center.
In contrast, the Analysis Center is primarily designed for data
scientists. Here, they can create an analysis entry derived froma study catalogue record. While doing so, they must furnish
comprehensive details, including the exact analysis title, adescriptive summary, and the privacy status of the analysis(private or public). In a private setting, data scientists canselectively grant access to specific users (data providers). Thismeans only authorized users can access these entries.
In the FL4E framework, data scientists play a crucial role inorchestrating the analysis by providing essential scripts forclient-side execution. Given that the FL4E backend is built upon
the Flower framework, these scripts must align with thisframework’s requirements. The machine learning–agnostic
nature of this foundation offers a substantial flexibility to data
scientists, allowing them to craft these scripts within the ambitof their expertise and the specific needs of the analysis. This
flexibility is further augmented by Flower’s support for securechannels, enabling the addition of privacy-preserving
configurations to the analysis execution. The integration of
Flower’s capabilities within FL4E not only simplifies thetechnical implementation of FL projects but also empowers data
scientists to innovate and experiment with various analyticalmodels and privacy configurations. This approach ensures that
FL4E remains a versatile and secure platform for FL,
accommodating a wide range of research objectives and data
privacy standards.
To ensure transparency, the process of sharing scripts among
the stakeholders is not automated. Clients may review and, if
agreed, run these scripts locally using the FL4E-clientcomponent. In keeping with the principles of modularity andcustomization, the server does not include a built-in compiler
to execute the analysis. Data scientists should disclose the IP
and port address of the designated machine to data providers
for analysis execution, enabling FL process participation. It isvital to note that no analysis should proceed in the Analysis
Center without explicit consent from participating centers. If aparticular study is associated with any data on central server,
data scientists should incorporate the data as a centralized client
https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et al
into the federated analysis, effectively acting as an additional
presumptive federated client in the data scientist’s machine.

Repository Center
The Repository Center serves as a warehouse for all scripts andmodels essential for stakeholders’ research and collaboration.
It serves both as a repository to disseminate the scripts andmodels to the relevant stakeholders, and as an archive that stores
past analyses and provides examples facilitating the creation of
new studies. It seamlessly integrates with the Study Center andModel Center, enabling direct and organized access to all shared
scripts and models across these centers.

Model Center
The Model Center is designed to facilitate the coordination andsharing of trained models. Similar to the Study Center, theModel Center is structured around 2 main elements: the Model
Catalogue and the Model Repository.
The Model Catalogue serves as a database for high-level
information about various trained models. This can include thetitle, description, type of model, study lead, and status. The
Model Catalogue is particularly useful for downstream users
and data scientists who come across an interesting model andwish to share it within the ecosystem, or for those who want toshare the outcome of an analysis conducted within the Study
Center.
Furthermore, the Model Repository is where detailed
information about each trained model is stored. This includes
the relationship of the model to a specific Model Catalogue
entry or Analysis Center entry. Like the Study Center, the Model
Center also adopts authorization layers to ensure that only
authorized users can access certain models. A typical scenario
is a trained model that would only be shared among theparticipants that contributed with data.
However, if needed, the Model Center can also be used todisseminate a model more broadly. Indeed, users can attachtrained models to the results of an analysis. This means that auser who may not have been able to join an analysis can still
benefit from the trained models, download them, and test themon their local data. This is especially useful for data providers
who want to validate models against their data sets or for data
scientists who wish to use them as pretrained models in a similar
domain.
Furthermore, the Model Center allows data scientists to provide
a decision support system as a result of a trained model. They
can inject an HTML web page into the Model Repository entry,
serving as a web-based dashboard. This user-friendly interface
enables downstream users, even those without expert knowledge,
to engage with the system effectively.

Data Center
The Data Center plays a crucial role in actualizing the conceptof the “degree of federation” within the FL4E framework.
Beyond providing data sharing capabilities, it also enables data
integration and aggregation, meaning this module stores thedata in a central server. Data providers and data scientists arecentral to the sharing process within the Data Center, using a
JMIR Form Res 2024 | vol. 8 | e55496 | p. 6
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
meticulous 3-step routine to ensure data quality, along witheffective data sharing and aggregation. The first step is (1) the
raw data sharing scheme, followed by (2) the data cleaning andenhancement, and (3) the data sharing and aggregation.

Raw Data Sharing Scheme
The process initiates when the data provider shares a raw data
scheme via a data dictionary, a static file describing the data
structure. This dictionary must be attached to an existing study
catalogue entry along with details like title, description, anddata type (public or private). For private data, the provider mustlist authorized users.

Data Cleaning and Enhancement Script
Once the data scheme is established, data scientists must createa Python script for data cleaning and enhancement. This script
ensures data quality by handling duplicates, missing values, andvalidating value ranges based on information of the data schema.
This script should be uploaded to the Data Center and linked
to the data dictionary.

Data Sharing and Aggregation
After setting up the data dictionary and cleaning script, data
sharing can occur. Each data share triggers the cleaning script
execution, with the outcomes appended to a core dictionary andeach upload saved as a distinct file. This setup permits various
data providers to share data in line with the data dictionary, withthe script guaranteeing aggregation into a single file. Data
scientists can then access this aggregated file for analytical use
if access is granted to them.

Implementation of the Framework
The FL4E framework is divided into server-side and client-side
implementations. Server-side, the framework is a
microservice-based web application, designed using ASP.NET
Core [58]. It adheres to the N-tier architecture and uses the
Model-View-Controller design, along with a repository pattern,
leading to a scalable and maintainable application. The
application is segmented into 4 layers: Application, Model,
Utility, and Data Access. Each layer performs distinct tasks,
such as managing user authorization, defining data structures,
handling requests, and interacting with the database. ASP.NET
Core Identity module is used for authentication andauthorization, while an admin control panel supervises user
registrations in the interface layer of the framework [58].
Client-side, the framework aims for simplicity and ease of use,also leveraging the ASP.NET Core framework. It incorporates
3 primary interfaces for data mounting, script uploading, andscript execution. The data mounting interface facilitates user
selection and mounting of data files; the script uploadinginterface enables users to upload scripts acquired from theserver; and the script execution interface allows for the runningof these scripts, using the Flower framework to secure thecommunication between the client and server.
Ensuring the seamless operation and accessibility of the FL4E
platform, both the client and server components are encapsulated
as Docker containers. This strategic choice underpins theplatform’s commitment to ease of deployment, consistency
across different environments, and portability, thereby,
https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et al
significantly enhancing the user experience for participants in
FL projects.
A fully functional prototype of the FL4E server component has
been successfully deployed to Microsoft Azure, showcasing theplatform’s robustness [59]. This deployment not only
demonstrates FL4E’s capability to leverage cloud infrastructure
but also its readiness to support FL projects.
To further acquaint potential users and stakeholders with FL4E’s
functionalities and user interface, a comprehensive video
demonstration has been made available [60]. This visual guideserves as an invaluable resource for understanding the platform’s
operational flow, features, and the simplicity of initiating andparticipating in FL studies.
For those interested in a deeper exploration or customizationof FL4E, the source code for both the server and clientcomponents is readily accessible [61]. This openness ensurestransparency, fosters a community of collaboration, and enables
continuous improvement of the platform by allowing
researchers, developers, and data scientists to contribute to its
development.

Ethical Considerations
This study conducted secondary analyses of openly licensed
repository data sets for demonstration purposes in the results
section. No sensitive data were used for the cocreation of theframework. All data set sources are cited and adhere to open
licensing agreements. For more detailed information about thedata sets, please refer to the original sources.

Results
Data Sets
We evaluated our framework using 2 real-world clinical data
sets from FLamby [62]: Fed-Heart-Disease and
Fed-TCGA-BRCA. The Fed-Heart-Disease data set contains
740 records from 4 centers, detailing 13 clinical features and abinary heart disease indicator for each patient. We used a logistic
regression model for prediction. The Fed-TCGA-BRCA dataset, sourced from The Cancer Genome Atlas’s Genomics Data
Commons portal, has data from 1088 patients with breast cancer
across 6 centers. With 39 clinical features and each patient’s
time of death, we used a Cox model to predict the risk of death.
Additionally, the Wisconsin Breast Cancer data set wasincorporated into the GitHub repository as an extra use case[61,63]. It serves as a set of foundational scripts that can begeneralized and act as guidelines for hybrid experiment settings
in the FL4E framework.

Degree of Federation Scenarios
To demonstrate how FL4E adapts to various research questions,and how it modulates the degree of federation, we conducted 3
experiments using these data sets:

Fully Federated Experiment
In this scenario, every client participates in the FL process witheach one contributing to the global model using their respective
local data sets. This scenario aligns with the conventional FL
JMIR Form Res 2024 | vol. 8 | e55496 | p. 7
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
setup. The data flow for this experiment is depicted in Figure
3.

Hybrid Experiment
In the hybrid setup, while all clients are part of the training
process, some of them contribute their data centrally to theserver. For instance, with the Fed-Heart-Disease data set, clients
1 and 2 were randomly selected to centrally contribute theirdata, while the rest participated as federated clients. For the
Fed-TCGA-BRCA data set, clients 2, 3, and 5 joined as
centralized clients.

Centralized Experiment
For this experiment, all clients send their data to the central
server, and the model is trained centrally. This design aims toprovide a benchmark for comparing the performance of thefederated and hybrid approaches with the centralized model.

Experimental Findings
We evaluated our FL models using area under the receiver
operating characteristic curve (ROC-AUC) and accuracy metrics
for the Fed-Heart-Disease data set and the concordance index
for the Fed-TCGA-BRCA data set. These metrics were chosen
according to their ability to assess model performance in binaryclassification and survival analysis task, while accounting forpotential class imbalance. For federated experiments, we
implemented multiple FL strategies: FedAVG [14], FedOpt[64] (including FedAdagrad and FedYogi), and FedProx [65].
Hyperparameter tuning was performed using grid search on acentralized setting and applied to each experiment of the data
sets. To ensure consistency, each experiment was repeated 5
times. The results of these experiments are summarized in Table
1.
In our analysis of the Fed-Heart-Disease data set, as illustrated
in Multimedia Appendix 2 and detailed in Table 1, notablebehavior patterns emerge from clients 2 and 3. These patterns
indicate the effects of class imbalance and quantity skew,
suggesting that smaller clients with imbalanced data sets may
benefit more from participating in an FL environment thanoperating in isolation. This insight is harmonious with thefoundational principles of the FL paradigm, which aspires toharness the potential of diverse data sources to enhance model
robustness and generalizability.
To clarify the evaluation process, we introduce the notation
where K represents the total number of clients participating in
the FL process, and ni denotes the size of the data set for thei-th client, where i ranges from 1 to K. Furthermore, Ei signifies
the evaluation metric (such as accuracy or ROC-AUC) obtained
from assessing the global model by the i-th client. The sum of
the data set sizes across all clients is represented by N, calculatedas. Therefore, the overall evaluation metric E for the

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et al
global FL model is calculated using the formula,
offering a comprehensive measure of performance across thefederated network.
This approach significantly impacts the evaluation process byprioritizing the characteristics of individual clients over thegeneralizability of the FL model. This provides a morecomprehensive understanding of model performance acrossdiverse environments. However, the potential for skewness infederated evaluation, often stemming from imbalanced clients,
arises from its reliance on a weighted average of individual
client metrics. This introduces challenges, particularly when
directly comparing it to centralized performance.
It is important to note that for a direct comparison between
federated and centralized evaluations, the ideal scenario involves
assessing the FL model using a separate global test set on theserver side. While this approach aligns more closely with themethodology used in centralized evaluations, it is not always
feasible, especially in real-world settings where maintaining aseparate test set on the server can be impractical due to data
governance and privacy concerns.
Contrary to initial expectations that centralized models would
outperform others, our analysis unveiled that federated andhybrid models exhibit superior performance based on the
ROC-AUC measure, even though the accuracy metrics mightsuggest a different narrative. To ensure the reliability andconsistency of our findings, we have conducted rigorouscomparisons with the original benchmark [62]. These
comparisons demonstrate that our observations, especiallyconcerning the performance metrics of centralized and federated
models such as accuracy, are in line with existing research. Thisalignment reinforces the validity of our analysis despite theinherent challenges in conducting direct comparisons between
federated and centralized performance evaluations [62].
Similarly, the Fed-TCGA-BRCA study substantiates theadvantage of FL. The heterogeneity of clients in the data set
analysis, as highlighted by FLamby [62], indicates that thisdiversity significantly influences training performance,
underscoring the potential benefits of FL.
Notably, the hybrid experiment yields a crucial insight: it offers
performance that matches fully federated models while
significantly outperforming local settings for smaller clients.
Importantly, this is achieved with considerably less complexity
and overhead associated with managing federation acrossmultiple data centers. This setup enables centralized
management of in-house data, providing analysts with directaccess (within regulatory boundaries) and flexibility foradvanced analyses. By having data information at their disposal,
analysts can develop more effective scripts, enabling advanced
analyses with greater flexibility.

JMIR Form Res 2024 | vol. 8 | e55496 | p. 8
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Pirmani et al

Table 1. Experiments carried out using FL4Ea with the Fed-Heart-Disease and Fed-TCGA-BRCA data sets, demonstrating adaptability to varying
degrees of federation.
Experiment and parameter

Fed-Heart-Disease, mean (SD)

Fed-TCGA-BRCA, mean (SD)ROC-AUCb

Accuracy

C-indexc

FedAvg0.846 (0.002)

0.733 (0.007)

0.732 (0.030)

FedAdagrad

0.841 (0.029)

0.726 (0.05)

0.748 (0.016)

FedYogi

0.803 (0.051)

0.715 (0.032)

0.745 (0.037)

FedProx0.846 (0.003)

0.741 (0.006)

0.725 (0.007)

FedAvg0.825 (0.004)

0.740 (0.054)

0.656 (0.06)

FedAdagrad

0.821 (0.008)

0.741 (0.005)

0.776 (0.036)

FedYogi

0.794 (0.013)

0.710 (0.039)

0.726 (0.041)

FedProx0.822 (0.004)

0.737 (0.012)

0.439 (0.227)

Client 00.842 (0.009)

0.753 (0.011)

0.668 (0.064)

Client 1

0.882 (0.007)

0.800 (0.013)

0.445 (0.237)

Client 2

0.546 (0.271)

0.550 (0.199)

0.635 (0.166)

Client 3

0.542 (0.054)

0.559 (0.096)

0.570 (0.140)

Client 4—d—0.851 (0.078)

Client 5——

0.666 (0.001)

Clients 1 and 2

0.819 (0.003)

0.752 (0.008)—

Clients 2, 3, and 5——

0.578 (0.125)

0.812 (0.003)

0.753 (0.007)

0.609 (0.207)

Fully federated

Hybrid experiment

Local experimentCentralizedaFL4E: Federated Learning for Everyone.

b

ROC-AUC: area under the receiver operating characteristic curve.

c

C-index: concordance index.

d

Not applicable.

Discussion
FL4E presents a novel approach to collaborative learning in ahealth care context, by innovatively balancing centralizationand federation. This allows health care organizations to engagein FL according to their infrastructure and policy capabilities.
Composed of the Study Center, Repository Center, Model
Center, and Data Center, FL4E’s modular, ecosystem-oriented
design caters to a wide variety of stakeholders, thereby
facilitating multiparty clinical research collaborations.
However, FL4E is not alone in promoting deployment ease andinclusivity in FL environments. FeatureCloud [66] advances
FL by offering a platform that simplifies deployment acrosshealth care and research sectors. Aimed at democratizing
multi-institutional analyses, FeatureCloud ensures that advanced
programming skills or complex setups do not impede
participation. The AI Store module, facilitating algorithm
sharing and access, highlights FeatureCloud’s commitment tobroadening FL adoption.

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

In parallel, the emergence of UniFed [67] underscores a similarcommitment to simplifying FL’s application by unifying diverse
open-source frameworks. UniFed’s approach standardizes FL
experimentation and deployment, offering a configuration-based,
schema-enforced task specification that eases the distributed
execution of FL projects. By supporting various FL frameworks,
UniFed addresses the challenges of workflow diversity, data
format discrepancies, and interpretation variations, thereby
streamlining the FL experimentation process.
What distinguishes FL4E in the evolving landscape is its focus
on the “degree of federation” and the adaptability it brings,
ensuring the framework is inclusive, adaptable, and can host
hybrid experiments. Unlike FeatureCloud, which was developed
from scratch, FL4E builds on the well-regarded Flower
framework, providing a robust, accessible foundation and using
this base, an adaptive, ecosystem-based model, designed to caterto diverse stakeholder preferences and requirements. FL4E
extends the democratization vision of FL, incorporating a
modular, customizable design that enhances engagement andensures versatility across various applications. Additionally,
FL4E adheres to key coding principles such as N-tier
JMIR Form Res 2024 | vol. 8 | e55496 | p. 9
(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
architecture and the repository pattern. These design choices
promote ease of use, flexibility for modifications, andfuture-proofing the framework, further distinguishing FL4E bymaking it more accessible, maintainable, and adaptable toadvancements in FL.
As the FL field continues to mature, platforms such as
FeatureCloud and UniFed play crucial roles in lowering
technical barriers and fostering an environment conducive tocollaborative innovation. Their contributions, along with FL4E’s
nuanced understanding of stakeholder engagement, collectively
propel the FL ecosystem toward a future where data-driven
insights are accessible to a broader spectrum of the health careand research community, thereby amplifying the impact of FL
in advancing health care outcomes and research endeavors.
Nonetheless, FL4E still presents some limitations. Version
control can pose challenges due to the Flower package
dependency, and significant version changes may requireinternal updates. Moving from technical concerns to regulatory
issues, ensuring adherence to regional data privacy regulations
necessitates diligence from initiators and study leads, who must
establish proper agreements for data sharing. Additionally, theframework’s dependence on manual communication among
stakeholders could potentially impede its universal accessibility.
This last point is particularly critical, as the system’s effectivefunctioning heavily relies on the presence of a pre-establishednetwork of collaborators where members have a certain levelof familiarity or knowledge of each other.
Although FL4E and other FL systems offer potential technicalsolutions to collaborative learning in health care, the influenceof societal barriers should not be underestimated. Hence,
continuous education, stakeholder engagement, andproof-of-concept demonstrations within controlled environments
are crucial for successful deployment.
Future development path to FL4E, the demand for advanced
privacy measures becomes increasingly apparent. While FL
introduces a novel approach to leveraging patient-level data
across decentralized sources, it is essential to recognize that thismethod is not without its vulnerabilities [68-70]. The processof sharing impersonal statistics during FL can inadvertently
encode sensitive information, thus introducing potential risks
and challenges to data privacy [68,71,72].
The architecture of FL, which is designed to enhance privacy
by eliminating the need for direct data sharing, is neverthelesssusceptible to sophisticated attacks and privacy breaches. Thesevulnerabilities include inference attacks, where malicious entities

Pirmani et al
may attempt to reconstruct personal data from model updates,
as well as direct cyber threats that target the integrity andconfidentiality of the FL process [68]. The body of literature
on FL privacy highlights the critical importance ofacknowledging these risks, with studies identifying various
attack vectors that could potentially compromise the privacy
guarantees of federated model sharing [72].
Therefore, despite FL’s significant reduction in the risk of direct
data exposure, it introduces new complexities in safeguarding
against indirect data inference and cyberattacks. This nuanced
understanding necessitates a cautious approach in theimplementation of FL, particularly when handling sensitive
health care information. The design of FL workflows, including
the development of infrastructure and study plans, must
incorporate robust security measures and privacy-preserving
techniques.
To this end, it is crucial to leverage the latest advancements insecure computing, differential privacy, and encryption
technologies to strengthen the FL process against potential
vulnerabilities [73-75]. Acknowledging these risks andadequately preparing for them is paramount, ensuring that theadvantages of using decentralized data through FL do not comeat the expense of patient privacy or data security. The
implementation of industry-scale privacy measures is criticallyimportant in FL applications, as emphasized by recent studies[74]. However, these measures must be carefully balanced toensure that data protection does not compromise thecomputational efficiency of FL analysis.
To ease deployment, recent research in practical FL points
toward promising techniques, such as soft computing’s fusion
with FL [76] and the use of FL in cloud-edge collaborativearchitectures [77]. Such strategies can address concerns overdata privacy, high communication costs, and data management
difficulties, offering a promising direction for future
investigations.
While the manual steps (for instance, script exchanges) in FL4E
are designed for transparency, automating these steps could
improve the framework’s adaptability and usability.
Additionally, transitioning from the current ASP.Net to Python
could amplify the framework’s appeal, especially given Python’s
strong foothold in the data science realm. As health care data
expand in volume and complexity, frameworks such as FL4E,
with further advancements and community involvement, standto be vital tools in catalyzing transformative progress andcollaboration in the domain.

Acknowledgments
YM, MO, and AP are funded by VLAIO PM: Augmenting Therapeutic Effectiveness through Novel Analytics (HBC.2019.2528);
Research Council KU Leuven: Symbiosis 4 (C14/22/125) and Symbiosis3 (C14/18/092); and CELSA—Active Learning(CELSA/21/019). YM, MO and AP are affiliated to Leuven.AI and received funding from the Flemish Government (AI Research
Program), FWO SBO (S003422N), and ELIXIR Belgium (I002819N). EDB was funded by an FWO-SB grant.

Data Availability
The data sets used for the analysis purposes of this study are available at the FLamby repository on GitHub [78]. FLamby serves
as a benchmark for cross-silo data sets for federated learning applications, with a focus on health care application [62].
https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e55496 | p. 10(page number not for citation purposes)

JMIR FORMATIVE RESEARCH

Pirmani et al

Authors' Contributions
EDB and YM contributed equally.

Conflicts of Interest
EDB is an employee of Genentech, Inc.

Multimedia Appendix 1
Landscape analysis: prominent frameworks for federated learning.
[DOCX File , 60 KB-Multimedia Appendix 1]

Multimedia Appendix 2
Delineating client heterogeneity in real-world applications: exploration of class imbalance in the Fed-Heart-Disease analysis.
[PNG File , 103 KB-Multimedia Appendix 2]

References
1.
2.
3.

4.
5.

6.

7.
8.

9.

10.
11.

12.

13.

14.

15.
16.

Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, et al. Real-world evidence - what is it and what
can it tell us? N Engl J Med. 2016;375(23):2293-2297. [doi: 10.1056/NEJMsb1609216] [Medline: 27959688]
Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning
approaches. Med Care. 2010;48(6 Suppl):S106-S113. [doi: 10.1097/MLR.0b013e3181de9e17] [Medline: 20473190]
Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles forscientific data management and stewardship. Sci Data. 2016;3:160018. [FREE Full text] [doi: 10.1038/sdata.2016.18][Medline: 26978244]
Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care.
Nat Rev Genet. 2012;13(6):395-405. [doi: 10.1038/nrg3208] [Medline: 22549152]
Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse forclinical research. J Am Med Inform Assoc. 2013;20(1):144-151. [FREE Full text] [doi: 10.1136/amiajnl-2011-000681][Medline: 22733976]
Kodra Y, Weinbach J, Posada-de-la-Paz M, Coi A, Lemonnier S, van Enckevort D, et al. Recommendations for improving
the quality of rare disease registries. Int J Environ Res Public Health. 2018;15(8):1644. [FREE Full text] [doi:10.3390/ijerph15081644] [Medline: 30081484]
Moorthy V, Henao Restrepo AM, Preziosi M, Swaminathan S. Data sharing for novel coronavirus (COVID-19). Bull World
Health Organ. 2020;98(3):150. [FREE Full text] [doi: 10.2471/BLT.20.251561] [Medline: 32132744]
Peeters LM, Parciak T, Walton C, Geys L, Moreau Y, De Brouwer E, et al. COVID-19 in people with multiple sclerosis:
a global data sharing initiative. Mult Scler. 2020;26(10):1157-1162. [FREE Full text] [doi: 10.1177/1352458520941485][Medline: 32662757]
Aitken M, de St Jorre J, Pagliari C, Jepson R, Cunningham-Burley S. Public responses to the sharing and linkage of healthdata for research purposes: a systematic review and thematic synthesis of qualitative studies. BMC Med Ethics. 2016;17(1):73.
[FREE Full text] [doi: 10.1186/s12910-016-0153-x] [Medline: 27832780]
Holm S, Ploug T. Big data and health research-the governance challenges in a mixed data economy. J Bioeth Inq.
2017;14(4):515-525. [doi: 10.1007/s11673-017-9810-0] [Medline: 28980135]
Taruscio D, Mollo E, Gainotti S, Posada de la Paz M, Bianchi F, Vittozzi L. The EPIRARE proposal of a set of indicatorsand common data elements for the European platform for rare disease registration. Arch Public Health. 2014;72(1):35.
[FREE Full text] [doi: 10.1186/2049-3258-72-35] [Medline: 25352985]
Dreyer NA, Blackburn SC, Mt-Isa S, Richardson JL, Thomas S, Laursen M, et al. Direct-to-patient research: piloting a
new approach to understanding drug safety during pregnancy. JMIR Public Health Surveill. 2015;1(2):e22. [FREE Fulltext] [doi: 10.2196/publichealth.4939] [Medline: 27227140]
Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, et al. A harmonized data quality assessment
terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244.
[FREE Full text] [doi: 10.13063/2327-9214.1244] [Medline: 27713905]
McMahan B, Moore E, Ramage D, Hampson S, Arcas BAY. Communication-efficient learning of deep networks fromdecentralized data. 2017. Presented at: 20th International Conference on Artificial Intelligence and Statistics; April 20-22,
2017:1273-1282; Fort Lauderdale, FL. URL: https://proceedings.mlr.press/v54/mcmahan17a.html
Konečný J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D. Federated learning: strategies for improving
communication efficiency. arXiv. Preprint posted online on October 30, 2017. [doi: 10.48550/arXiv.1610.05492]
Ludwig H, Baracaldo N, Thomas G, Zhou Y, Anwar A, Rajamoni S, et al. IBM Federated Learning: an enterprise framework
white paper V0.1. arXiv. Preprint posted online on July 22, 2020. [doi: 10.48550/arXiv.2007.10987]

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e55496 | p. 11(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
17.
18.

19.
20.
21.
22.
23.
24.
25.
26.

27.
28.
29.
30.
31.
32.
33.
34.

35.
36.
37.
38.

39.
40.
41.
42.
43.
44.
45.
46.
47.

48.

Wen Y, Li W, Roth H, Dogra P. Federated Learning powered by NVIDIA Clara. NVIDIA Technical Blog. Dec 1, 2019.
URL: https://developer.nvidia.com/blog/federated-learning-clara/ [accessed 2023-12-15]
Rodríguez-Barroso N, Stipcich G, Jiménez-López D, Ruiz-Millán JA, Martínez-Cámara E, González-Seco G, et al. Federated
learning and differential privacy: software tools analysis, the Sherpa.ai FL framework and methodological guidelines forpreserving data privacy. Information Fusion. Dec 2020;64:270-292. [doi: 10.1016/j.inffus.2020.07.009]
Ziller A. PySyft: a library for easy federated learning. Studies in Computational Intelligence. 2021;965:111-139.
Yang L, Tao F, Tianjian C, Qian X, Qiang Y. FATE: an industrial grade platform for collaborative learning with dataprotection. J Mach Learn Res. 2021;22(226):1-6. [FREE Full text]
Foley P, Sheller MJ, Edwards B, Pati S, Riviera W, Sharma M, et al. OpenFL: the open federated learning library. Phys
Med Biol. 2022;67(21):214001. [FREE Full text] [doi: 10.1088/1361-6560/ac97d9] [Medline: 36198326]
Try federated learning with OpenFL. Intel. URL: https://www.intel.com/content/www/us/en/developer/articles/technical/
try-federated-learning-with-openfl.html [accessed 2024-03-18]
TensorFlow federated: machine learning on decentralized data. TensorFlow. URL: https://www.tensorflow.org/federated[accessed 2023-12-15]
Zeng D, Liang S, Hu X, Wang H, Xu Z. FedLab: a flexible federated learning framework. arXiv. Preprint posted online
on April 22, 2022. [doi: 10.48550/arXiv.2107.11621]
He C, Li S, So J, Zeng X, Zhang M, Wang H, et al. FedML: a research library and benchmark for federated machinelearning. arXiv. Preprint posted online on November 8, 2020. [doi: 10.48550/arXiv.2007.13518]
Li B, Su N, Ying C, Wang F. Plato: an open-source research framework for production federated learning. 2023. Presented
at: ACM TURC '23: ACM Turing Award Celebration Conference - China 2023; July 28-30, 2023:1-2; Wuhan, China. [doi:10.1145/3603165.3607364]
PaddlePaddle/PaddleFL: federated deep learning in PaddlePaddle. GitHub. URL: https://github.com/PaddlePaddle/PaddleFL[accessed 2024-03-18]
Liu L, Wu C, Xiao J. Blockchain-based platform for distribution AI. EasyChai. Preprint posted online on February 2, 2019.
[doi: 10.29007/4BZ1]
Qu Y, Uddin MP, Gan C, Xiang Y, Gao L, Yearwood J. Blockchain-enabled federated learning: a survey. ACM Comput
Surv. Nov 21, 2022;55(4):1-35. [doi: 10.1145/3524104]
Li L, Wang J, Xu CZ. FLSim: an extensible and reusable simulation framework for federated learning. Social-Informatics
and Telecommunications Engineering. 2021;369:350-369. [doi: 10.1007/978-3-030-72792-5_30]
Caldas S, Duddu SMK, Wu P, Li T, Konečný J, McMahan HB, et al. LEAF: a benchmark for federated settings. arXiv.
Preprint posted online on December 9, 2019. [doi: 10.48550/arXiv.1812.01097]
Beutel DJ, Topal T, Mathur A, Qiu X, Fernandez-Marques J, Gao Y, et al. Flower: a friendly federated learning researchframework. arXiv. Preprint posted online on March 5, 2022. [doi: 10.48550/arXiv.2007.14390]
PySyft. OpenMined Blog. URL: https://blog.openmined.org/tag/pysyft/ [accessed 2024-03-18]
Ziller A, Trask A, Lopardo A, Szymkow B, Wagner B, Bluemke E, et al. PySyft: a library for easy federated learning. In:
Rehman MH, Gaber MM, editors. Federated Learning Systems. Studies in Computational Intelligence, vol 965. Cham,
Switzerland. Springer; Jun 12, 2021:111-139.
FATE. FedAI. URL: https://fate.fedai.org/ [accessed 2024-03-18]
FederatedAI/FATE-Community. GitHub. URL: https://github.com/FederatedAI/FATE-Community/blob/master/
FATE_Project_Technical_Charter.pdf [accessed 2024-03-18]
FederatedAI/FATE. GitHub. URL: https://github.com/FederatedAI/FATE [accessed 2024-03-18]
He C, Li S, So J, Zhang M, Zeng X, Wang H, et al. FedML: a research library and benchmark for federated machinelearning. MIT Media Lab. Dec 12, 2020. URL: https://www.media.mit.edu/publications/fedml-a-research-library-and-benchmark-for-federated-machine-learning/ [accessed 2024-03-18]
The production AI platform for federated learning at scale. FedML. URL: https://www.fedml.ai/home [accessed 2024-03-18]
Flower: a friendly federated learning framework. Flower. URL: https://flower.ai/ [accessed 2024-03-18]
alibaba/FederatedScope. GitHub. URL: https://github.com/alibaba/FederatedScope [accessed 2024-04-24]
OpenFL: an open source framework for federated learning. OpenFL. URL: https://openfl.io/ [accessed 2024-03-18]
SMILELab-FL/FedLab. GitHub. URL: https://github.com/SMILELab-FL/FedLab [accessed 2024-03-18]
Stripelis D, Anastasiou C, Toral P, Asghar A, Ambite JL. arXiv. Preprint posted online on November 13, 2023. [doi:10.48550/arXiv.2311.00334]
Roth HR, Cheng Y, Wen Y, Yang I, Xu Z, Hsieh YT, et al. NVIDIA FLARE: federated learning from simulation toreal-world. arXiv. Preprint posted online on April 28, 2023. [doi: 10.48550/arXiv.2210.13291]
PFL: Paddle federated learning. PaddleFL. URL: https://paddlefl.readthedocs.io/en/latest/ [accessed 2024-03-18]
Lai F, Dai Y, Singapuram S, Liu J, Zhu X, Madhyastha H, et al. FedScale: benchmarking model and system performanceof federated learning at scale. arXiv. Preprint posted online on June 18, 2022. [FREE Full text] [doi:10.48550/arXiv.2105.11367]
TL-System/plato. GitHub. URL: https://github.com/TL-System/plato [accessed 2024-03-18]

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et al

JMIR Form Res 2024 | vol. 8 | e55496 | p. 12(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
49.
50.
51.
52.

53.
54.

55.

56.

57.

58.
59.
60.
61.
62.

63.
64.
65.
66.

67.
68.
69.

70.
71.

72.
73.
74.
75.

Zhuang W, Gan X, Wen Y, Zhang S. EasyFL: a low-code federated learning platform for dummies. IEEE Internet Things
J. 2022;9(15):13740-13754. [doi: 10.1109/jiot.2022.3143842]
GalaxyLearning/GFL. GitHub. URL: https://github.com/GalaxyLearning/GFL [accessed 2024-03-18]
facebookresearch/FLSim. GitHub. URL: https://github.com/facebookresearch/FLSim [accessed 2024-03-18]
Crowson MG, Moukheiber D, Arévalo AR, Lam BD, Mantena S, Rana A, et al. A systematic review of federated learning
applications for biomedical data. PLOS Digit Health. May 2022;1(5):e0000033. [FREE Full text] [doi:10.1371/journal.pdig.0000033] [Medline: 36812504]
Rudrapatna VA, Butte AJ. Opportunities and challenges in using real-world data for health care. J Clin Invest.
2020;130(2):565-574. [FREE Full text] [doi: 10.1172/JCI129197] [Medline: 32011317]
Hiramatsu K, Barrett A, Miyata Y, PhRMA Japan Medical Affairs Committee Working Group 1. Current status, challenges,
and future perspectives of real-world data and real-world evidence in Japan. Drugs Real World Outcomes. 2021;8(4):459-480.
[FREE Full text] [doi: 10.1007/s40801-021-00266-3] [Medline: 34148219]
Simpson-Yap S, De Brouwer E, Kalincik T, Rijke N, Hillert JA, Walton C, et al. Associations of disease-modifying therapies
with COVID-19 severity in multiple sclerosis. Neurology. 2021;97(19):e1870-e1885. [doi: 10.1212/WNL.0000000000012753][Medline: 34610987]
Simpson-Yap S, Pirmani A, Kalincik T, De Brouwer E, Geys L, Parciak T, et al. Updated results of the COVID-19 in MS
Global Data Sharing Initiative: anti-CD20 and other risk factors associated with COVID-19 severity. Neurol Neuroimmunol
Neuroinflamm. Aug 29, 2022;9(6):e200021. [FREE Full text] [doi: 10.1212/NXI.0000000000200021] [Medline: 36038263]
Pirmani A, de Brouwer E, Geys L, Parciak T, Moreau Y, Peeters LM. The journey of data within a global data sharing
initiative: a federated 3-layer data analysis pipeline to scale up multiple sclerosis research. JMIR Med Inform. 2023;11:e48030.
[FREE Full text] [doi: 10.2196/48030] [Medline: 37943585]
ASP.NET Core. Microsoft. URL: https://dotnet.microsoft.com/en-us/apps/aspnet [accessed 2024-12-30]
FL4E-working prototype. Azure. URL: https://fl-server.azurewebsites.net/ [accessed 2024-12-30]
Federated learning for everyone. Ashkan Pirmani YouTube page. May 14, 2024. URL: https://www.youtube.com/
watch?v=K8n7kffpvQ0&ab_channel=AshkanPirmani [accessed 2024-05-14]
ashkan-pirmani/FL4E-Analysis. GitHub. URL: https://github.com/ashkan-pirmani/FL4E-Analysis [accessed 2024-12-15]
du Terrail JO, Ayed SS, Cyffers E, Grimberg F, He C, Loeb R, et al. FLamby: datasets and benchmarks for cross-silofederated learning in realistic healthcare settings. arXiv. Preprint posted online on May 5, 2023. [doi:10.48550/arXiv.2210.04620]
Breast Cancer Wisconsin (Prognostic). UC Irvine Machine Learning Repository. URL: https://archive.ics.uci.edu/dataset/
16/breast+cancer+wisconsin+prognostic [accessed 2024-12-15]
Reddi S, Charles Z, Zaheer M, Garrett Z, Rush K, Konečný J, et al. Adaptive federated optimization. arXiv. Preprint posted
online on September 8, 2021. [doi: 10.48550/arXiv.2003.00295]
Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V. Federated optimization in heterogeneous networks. arXiv.
Preprint posted online on April 21, 2020. [doi: 10.48550/arXiv.1812.06127]
Matschinske J, Späth J, Bakhtiari M, Probul N, Kazemi Majdabadi MM, Nasirigerdeh R, et al. The FeatureCloud platform
for federated learning in biomedicine: unified approach. J Med Internet Res. 2023;25:e42621. [FREE Full text] [doi:10.2196/42621] [Medline: 37436815]
Liu X, Shi T, Xie C, Li Q, Hu K, Kim H, et al. UniFed: all-in-one federated learning platform to unify open-source
frameworks. arXiv. Preprint posted online on December 30, 2023. [FREE Full text] [doi: 10.48550/arXiv.2207.10308]
Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Nitin Bhagoji A, et al. Advances and Open Problems in Federated
Learning. Foundations and Trends in Machine Learning. 2021;14(1–2):1-210. [FREE Full text] [doi: 10.1561/2200000083]
Nasirigerdeh R, Torkzadehmahani R, Baumbach J, Blumenthal DB. On the privacy of federated pipelines. 2021. Presented
at: SIGIR 2021 - the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval;
July 11-15, 2021:1975-1979; Virtual event, Canada. [doi: 10.1145/3404835.3462996]
Hartebrodt A, Röttger R, Blumenthal DB. Federated singular value decomposition for high-dimensional data. Data Min
Knowl Disc. 2023;38(3):938-975. [doi: 10.1007/s10618-023-00983-z]
Tolpegin V, Truex S, Gursoy ME, Liu L. Data poisoning attacks against federated learning systems. Berlin, Heidelberg.;
2020. Presented at: 25th European Symposium on Research in Computer Security, ESORICS 2020; September 14-18,
2020:480-501; Guildford, United Kingdom. [doi: 10.1007/978-3-030-58951-6_24]
Bagdasaryan E, Veit A, Hua Y, Estrin D, Shmatikov V. How to backdoor federated learning. arXiv. Preprint posted online
on August 6, 2019. [doi: 10.48550/arXiv.1807.00459]
Dwork C. Differential privacy. 2006. Presented at: 33rd International Colloquium, ICALP 2006; July 10-14, 2006:1-12;
Venice, Italy. [doi: 10.1007/11787006_1]
Park J, Lim H. Privacy-preserving federated learning using homomorphic encryption. Applied Sciences. 2022;12(2):734.
[doi: 10.3390/app12020734]
Wood A, Najarian K, Kahrobaei D. Homomorphic encryption for machine learning in medicine and bioinformatics. ACM
Comput Surv. 2020;53(4):1-35. [doi: 10.1145/3394658]

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

Pirmani et al

JMIR Form Res 2024 | vol. 8 | e55496 | p. 13(page number not for citation purposes)

JMIR FORMATIVE RESEARCH
76.
77.

78.

Pirmani et al

Supriya Y, Gadekallu TR. A survey on soft computing techniques for federated learning- applications, challenges and future
directions. J. Data and Information Quality. 2023;15(2):1-28. [doi: 10.1145/3575810]
Huang Y, Yang X, Guo J, Cheng J, Qu H, Ma J, et al. A high-precision method for 100-day-old classification of chickens
in edge computing scenarios based on federated computing. Animals (Basel). 2022;12(24):3450. [FREE Full text] [doi:10.3390/ani12243450] [Medline: 36552370]
owkin/FLamby. GitHub. URL: https://github.com/owkin/FLamby [accessed 2024-05-14]

Abbreviations
FL: federated learning
FL4E: Federated Learning for Everyone
ROC-AUC: area under the receiver operating characteristic curve
RWD: real-world data

Edited by A Mavragani; submitted 15.12.23; peer-reviewed by D Blumenthal, X Zhang; comments to author 23.02.24; revised version
received 25.04.24; accepted 15.05.24; published 17.07.24
Please cite as:
Pirmani A, Oldenhof M, Peeters LM, De Brouwer E, Moreau Y
Accessible Ecosystem for Clinical Research (Federated Learning for Everyone): Development and Usability StudyJMIR Form Res 2024;8:e55496
URL: https://formative.jmir.org/2024/1/e55496
doi: 10.2196/55496
PMID:©Ashkan Pirmani, Martijn Oldenhof, Liesbet M Peeters, Edward De Brouwer, Yves Moreau. Originally published in JMIR
Formative Research (https://formative.jmir.org), 17.07.2024. This is an open-access article distributed under the terms of theCreative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The
complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright andlicense information must be included.

https://formative.jmir.org/2024/1/e55496

XSL• FO
RenderX

JMIR Form Res 2024 | vol. 8 | e55496 | p. 14(page number not for citation purposes)