Towards a Collective Medical Imaging AI: Enabling Continual Learning from Peers

Authors: guangyao  zheng
vladimir  braverman
jeffrey  leal
steven  rowe
doris  leung
michael a jacobs
vishwa s parekh
v  zheng
j l braverman
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 00:33:30

Abstract: Federated learning is an exciting area within machine learning that allows cross-silo training of large-scale machine learning models on disparate or similar tasks in a privacy-preserving manner. However, conventional federated learning frameworks require a synchronous training schedule and are incapable of lifelong learning. To that end, we propose an asynchronous decentralized federated lifelong learning (ADFLL) method that allows agents in the system to asynchronously and continually learn from their own previous experiences and others', thus overcoming the potential drawbacks of conventional federated learning. We evaluate the ADFLL framework in two experimental setups for deep reinforcement learning (DRL) based landmark localization across different imaging modalities, orientations, and sequences. The ADFLL was compared to central aggregation and conventional lifelong learning for upper-bound comparison and with a conventional DRL model for lower-bound comparison. Across all the experiments, ADFLL demonstrated excellent capability to collaboratively learn all tasks across all the agents compared to the baseline models in in-distribution and out-of-distribution test sets. In conclusion, we provide a flexible, efficient, and robust federated lifelong learning framework that can be deployed in real-world applications.

Main paper: 
 introduction: 
 Federated Learning (FL) has recently emerged as a promising approach enabling multiple agents to train a model collaboratively without sharing their data (Adnan et al., 2022) , which protects data privacy and reduces computational costs at the local agent level by distributing the computation to multiple agents to train the model on their local data and sharing only the model updates with a central server. FL implementations have shown promising results in various medical applications (Roth et al., 2020; Jiang et al., 2022; Yan et al., 2021) . FL frameworks often rely on synchronized learning schedules, meaning all participating agents train simultaneously and synchronously communicate with the server. Additionally, conventional FL approaches cannot perform Lifelong Learning (LL) to dynamically integrate information from newer nodes joining the federation or retain information from older nodes leaving the federation, an essential aspect of machine learning applied to medical imaging with constantly changing medical imaging protocols and emergence of new diseases such as COVID-19 (Karani et al., 2018a; Zheng et al., 2023; Derakhshani et al., 2022; Karani et al., 2018b) . set up for cross-modality 3D localization of spleen. The blue, orange, and green boxes represent different agents in the setup. Each agent sequentially encounters two different imaging modalities for along with experiences shared by the other nodes, enabling them to learn to localize the spleen across all four modalities (as opposed to just the two they encountered).To address these limitations, we propose ADFLL, an asynchronous decentralized federated lifelong learning framework constructed using a network of lifelong learning nodes. We develop and evaluate ADFLL using deep reinforcement learning (DRL) as the base model architecture at each node for the task of 3D landmark localization in medical imaging volumes. DRL with LL capability using selective experience replay (SERIL) has previously been validated for continual landmark localization across different imaging environments using DRL for medical imaging and also forms the basis of LL for each node in ADFLL (Isele and Cosgun, 2018; Rolnick et al., 2019; Zheng et al., 2023) . SERIL techniques allow for LL by simultaneously training from a selected sample of a model's previous experiences in addition to the model's current task. ADFLL expands this capability of SERIL to a collaborative setup by allowing SERIL nodes to not only learn from their own previous experiences but also from experiences shared by collaborator nodes, thereby enabling continual learning from peers, as shown in Figure 1 .In this work, we evaluated ADFLL across two different experimental setups for crossdomain 3D landmark localization in multiparametric brain MRI and multi-modality whole body imaging in-distribution and out-of-distribution datasets using deep reinforcement learning (DRL). In addition, we evaluated the ability of ADFLL framework to scale with increase number of agents as well as integrate and retain information without catastrophic forgetting in different LL scenarios with nodes leaving and joining the framework. 
lifelong deep reinforcement learning: 
 We implemented DRL based on the deep Q-network (DQN) algorithm, as illustrated in Figure 2 . The 3D DQN model implemented in this work was adapted from (Mnih et al., 2013; Alansary et al., 2018; Vlontzos et al., 2019; Parekh et al., 2020) . The environment was depicted using a 3D imaging volume and the agent was represented by a 3D bounding box capable of six distinct actions: a ∈ x++, x--, y++, y--, z++, z--. The state was defined by an agent's current location (or a chain of locations), each represented by a 3D bounding box of 45 × 45 × 11 pixels. The reward is calculated by the change in distance to the target landmark location before and after the agent takes an action. The agent's exploration within the environment generated state-action-reward-resulting state [s, a, r, s ′ ] tuples, which are recorded and sampled in the experience replay buffer (ERB) over multiple episodes.We implemented lifelong learning using selective experience replay (Rolnick et al., 2019; Zheng et al., 2023) , a model-agnostic lifelong learning approach that enables sharing experiences across different models. ERBs produced by DRL agents during the previous training enable us to achieve lifelong learning. To learn a generalized representation of current and past tasks, the model selects a batch of experiences from the ERB of its current task and the ERBs of previous tasks during training. The information in the ERBs is non-sensitive as the state and resulting states are tiny fractions of the total 3D image, roughly 0.3%.We developed the Asynchronous Decentralized Federated Lifelong Learning (ADFLL) by constructing a network of lifelong DRL agents with the additional modification in the train- ing setup that allows each agent to sample experiences from the current dataset ERB, the agent's personal experiences, and the incoming experiences from other agents, as shown in Figure 2 . Each agent shares their database of personal experiences with each other to facilitate learning from each other's experiences. More specifically, once an agent finishes training with a dataset, the resulting experience from the training is shared with the network. As a result, every agent in the network can learn from each other's experiences, thereby integrating federated lifelong learning capability. Furthermore, to address the communication inefficiencies of an all-to-all decentralized federated learning framework, we use a predefined set of hub nodes that act as communication hubs for spatially adjacent nodes in the network, as shown in Figure 2 . Subsequently, every node in the network exclusively and asynchronously communicates with its nearest hub node. By periodically synchronizing their databases with each other, the hub nodes regulate and preserve the experiences in the system in case of communication failures or node dropouts. Our code is available at: https://github.com/vishwaparekh/ADFLL. 
baseline agents: 
 All-knowing agent: Agent X is trained using central aggregation of all the data across all the ADFLL agents in one place. This gives us the baseline "upper bound" performance for comparison to the ADFLL agents.Conventional lifelong DRL agent: Agent M has sequential access to all the datasets and forms an "upper bound" comparison between lifelong learning with and without experience sharing between lifelong learning DRL agents.Conventional DRL agent: Agent Y is a conventional DRL with access to only one dataset, forming our "lower bound" baseline for performance comparison with ADFLL agents.Multiparametric Brain MRI Dataset: We used the multiparametric MRI (mpMRI) brain tumor segmentation (BraTS) dataset (Menze et al., 2014) in this study. This dataset consisted of 285 patients and included pre-contrast T1-weight, post-contrast T1-weighted, T2-weighted, and Fluid Attenuated Inversion Recovery (FLAIR) sequences in the axial orientation. We randomly sampled a subset of 100 patients for this study (60 patients with high-grade glioma (HGG) and 40 patients with low-grade glioma (LGG)). We split the 100 patients into two parts 80:20. 80 were used for training and 20 for evaluation, with the training set consisting of 48 HGG and 32 LGG tumors and the test set consisting of 12 HGG and 8 LGG tumors. We reconstructed the dataset to include all three imaging orientations (coronal, sagittal, and axial). As a result, we obtained twenty-four imaging environments with combinations of two pathologies, 4 imaging sequences, and 3 image orientations. All the experiments were performed for the task of left ventricle localization. A sample of 8 task-environment pairs is shown in Appendix Fig. 4 .Multimodality Whole-Body Imaging Dataset: All studies were performed in accordance with the institutional guidelines for clinical research under a protocol approved by our Institutional Review Board (IRB), and all HIPAA agreements were followed for this retrospective study. Our dataset consisted of 50 images volumes, 10 from each imaging sequence type (PET, CT, MRI-T1, MRI-DIXON-F, and MRI-DIXON-W). The PET/CT dataset was acquired using a Biograph mCT 128-slice PET/CT scanner (Siemens Healthineers). For the 18F-DCFPyL scans, patients were intravenously injected with no more than 333 MBq (9 mCi) of radiotracer approximately 60 min before image acquisition. The field of view was vertex to mid-thigh for 18F-DCFPyL. The T1, DIXON-F, and DIXON-W sequences were sampled from the whole body (WB) multiparametric MRI dataset that was acquired using an imaging protocol that scanned from the shoulders to the lower mid calf and described in (Leung et al., 2020) . Each of the five imaging sequences was annotated with 3D spleen localization for the experiments. In addition thirty-seven imaging volumes were randomly sampled from the same datasets for internal validation (10 PET, 3 CT, 4 MRI-T1, and 10 MRI-DIXON-F and W).Test Set: All the ADFLL agents and the baseline agents were evaluated using external out-of-distribution datasets sourced from The Cancer Imaging Archive (TCIA) and the Medical Segmentation Decathlon (MSD) databases. More specifically, we sampled a total of 10 PET volumes from the NSCLC-Radiogenomics (N=4) and ACRIN-NSCLC-FDG-PET (N=6) collections (Gevaert et al., 2012; Clark et al., 2013; Machtay et al., 2013; Bakr et al., 2017 Bakr et al., , 2018;; Kinahan et al., 2019) , 41 CT volumes from the spleen MSD dataset (Antonelli et al., 2022) , and 10 WB mpMRI volumes from the CMB-CRC (N=3), CPTAC-CCRCC (N=2), CPTAC-PDA (N=3), and CPTAC-UCEC (N=2) (Clark et al., 2013) .Cross-Environment 3D ventricle localization: We initialized this experiment with four agents and a sub-sample of eight brain mpMRI environments (shown in Fig. 4 ). We implemented two agents on an NVIDIA DGX-1, each with an NVIDIA V100, and two on Google Cloud, each with an NVIDIA T4. The two agents on Google Cloud, A1 and A2, have their individual hubs, H1 and H2 whereas the remaining two agents, A3 and A4, on the DGX-1 were connected to the third hub, H3. We trained all four agents for three iterations. At the beginning of each iteration, each agent received a randomly selected training dataset with a different environment. As a result, each agent trained on a total of only three environments (out of eight) for ventricle localization. In addition, the agents shared their ERBs with each other at the end of every iteration to facilitate peer-to-peer experience sharing with lifelong learning.Each of the agents had different training speeds owing to different GPUs used by each agent. Subsequently, we implemented asynchronous learning, meaning when the agent finishes training on a task, it will broadcast its ERBs to the hub and begin training on the next dataset using its previous ERB and any new ERBs available at the hub. As a result, the number of ERBs available from other ADFLL agents when starting a new round of training will significantly vary between the slowest and the fastest agents in the network. This process is continued until all four agents complete three rounds of training. For comparison, Agent M was trained for eight rounds, and Agent X was trained to use the central aggregation of all eight datasets.Cross-Modality 3D Spleen Localization: While the first experiment evaluated the ability of the ADFLL framework to perform asynchronous experience sharing with lifelong learning, the second experiment evaluated the capability of the ADFLL framework to learn from ERBs that encoded different imaging modalities.We initialized this experiment with five agents and five training sets, each consisting of 10 volumes from different imaging sequences (PET, CT, MRI-T1, MRI-DIXON-F, MRI-DIXON-W) for the task of 3D spleen localization. At the beginning of each iteration, each agent encounters one of the five datasets to train a deep reinforcement learning model for localization of spleen on that dataset. Each agent was trained for three rounds i.e., each agent sequentially encountered three datasets (out of five). In addition, the agents shared their ERBs with each other at the end of every iteration to facilitate peer-to-peer experience sharing with lifelong learning. At the end of three rounds, we evaluated each ADFLL node on all five tasks on both the validation and external test sets and compared its performance to Agent X (that had central access to all the datasets), Agent M (that had sequential access to all the datasets), and Agent Y (that had access to only the CT dataset)Evaluation Metric: The performance metric was set as the terminal Euclidean distance in pixels between the agent's prediction and the target landmark. We performed paired t-tests to compare the performance of the ADFLL agents with the traditional lifelong learning framework, all-knowing deep reinforcement learning agent, and partialknowing deep reinforcement learning agent. The p-value for statistical significance was set to p ≤ 0.05. 
results: 
 Cross-Environment 3D Ventricle Localization: As shown in Table 1 , all four ADFLL agents had excellent performance with no significant difference in performance from the allknowing Agent X (p > 0.05). In addition, three agents (A1, A3, and A4) had no significant difference from Agent M (best-case traditional LL agent), and A2 was significantly better than Agent M (p = 0.01), just after three rounds of training, compared to eight rounds of training for Agent M. After three rounds of training, A2 was able to achieve a mean Cross-Modality 3D Spleen Localization: Table 2 summarizes the performance of all five ADFLL agents along with the three baseline agents, Agent X (all-knowing best-case DRL agent), Agent M (traditional lifelong learning agent), and Agent Y (DRL agent trained on just the CT dataset) for the internal validation dataset. Agent 2 was the best performing ADFLL agent with the Euclidean distance error of 8.01±5.47. As shown in Appendix Table 2 , there was no significant difference in the Euclidean distance error between the four (out of five) ADFLL agents and the Agent X (d = 8.01±5.47) for the internal validation dataset. Furthermore, all five agents had no significant difference in Euclidean distance error from the Agent M (d = 11.22 ± 20.09). Furthermore, all five ADFLL agents had a significantly lower Euclidean distance error compared to Agent Y (d = 117.82 ± 70.31) that was trained only the CT dataset. Figure 3 illustrates the performance of the best performing ADFLL agent, Agent 2 as well as the baseline agents across all five imaging sequences.Appendix Tables 3, 4 , and 5 summarize the performance of all ADFLL and baseline agents across the three external datasets. Apart from spleen MSD CT dataset where Agent M significantly (p < 0.05) outperformed three (out of five) ADFLL agents, there was no significant difference between the ADFLL agents and the best-case baseline agents, X and M across all the experiments. 
discussion: 
 In this work, we proposed the asynchronous decentralized federated learning framework to achieve collective intelligence via continual learning from peers with excellent results across all experiments. The ADFLL framework's performance was similar to upper bound baselines: the all-knowing baseline with all the datasets centrally aggregated in one location and the conventional LL baseline, where an agent sequentially trains across all the datasets.The cross-environment ventricle localization experiment allowed us to evaluate the realworld setting where different agents with different compute environments train at different speeds leading to asynchronous communication. Despite training asynchronously and on different sets of shared experiences, all four agents had a similar performance to the conventional lifelong learning model. However, the fastest agent (A1) had the worst performance across all ADFLL agents owing to being trained on only a subset of experiences shared across the network. Similarly, the cross-modality spleen localization experiment allowed us to evaluate the capability of ADFLL agents to learn from peer ERBs encoding different modalities from disparate datasets. Our results that ADFLL agents are not only capable of learning from ERBs that encode significantly different imaging environments, but also demonstrate excellent generalization performance when when evaluated on external datasets with different modalities.Asynchronous federated learning has also been explored in other areas (Chen et al., 2019) . They offer the ability to deal with nodes with different computational power but lack the decentralization that allows the system to be more flexible. Similarly, in (Liu et al., 2022; Huang et al., 2022) , the cost of removing a central node is a quadratic complexity communication scheme in that every node communicates with every node. Our work has certain limitations. This preliminary study focused on a single landmark localization task with deep reinforcement learning. Our future work will expand this framework to multi-agent systems and other medical imaging tasks such as segmentation and classification. In conclusion, we demonstrated a privacy-aware, asynchronous, decentralized federated learning system with robust and efficient system topology. Results: In the two simulation studies, our framework showed scalability of up to 24 agents, robustness against network dropout, and flexibility in system topology. As shown in Fig. 5 , we see that the average Euclidean distance error across all agents and all 24 tasks decreases as more agents are added to the system, with an average Euclidean distance error of 16.89 ± 16.34 at the end of 4 rounds. This demonstrates the capability of ADFLL agents to directly train from peer experiences without pre-training, allowing newer agents joining the system directly learn existing knowledge within the system in one round. For the deletion experiment, the average Euclidean distance error across all agents decreases while half of the agents are deleted every round, resulting in an average Euclidean distance error of 8.55 ± 7.12 after the final remaining agent at the end of 5 rounds (shown in Fig. 5 ). In comparison, the average Euclidean distance error was 8.34 ± 7.26 (p > 0.05) for Agent X and 8.15 ± 5.42 (p > 0.05) for Agent M. This shows that the knowledge agents learned and captured in ERBs are not lost when agents are removed from the system. When agents are added, the new agents can catch up with existing agents in one round. Moreover, the 75% dropout rate applied to every round of both experiments shows the robustness of our framework against network failures, a significant bottleneck for federated learning frameworks. To test the influence of agent count in a system on its performance, we tested systems with 2, 4, 8, 16, or 24 Agents training for two rounds on their performance to localize the top left ventricle in all 24 imaging environments. Since it is prohibitively expensive to experiment on 24 different machines, these systems were simulated on the NVIDIA DGX-1 with a synchronous training protocol.Result: Figure 6 demonstrates the comparison of Agent Rewards (AR) when different ADFLL systems consisting of 2, 4, 8, 16, and 24 agents were trained for two rounds in addition to the best outcome agent. The ADFLL agent systems with ≥ 4 agents learned more than 80% of the environments within just two rounds. Finally, the ADFLL systems with 16 and 24 agents learned greater than 90% of the environments. To further test the scalability limit of our framework, we designed an experiment that involves 72 agents. We will use the 24 agents from the previous experiments and add 48 agents to the system, and the 48 agents will run for two rounds. They will be randomly assigned to an imaging environment each round. In the first round, they will learn from their local data and ERBs of the previous 24 agents. In the second round, they will learn from their local data, ERBs of the previous 24 agents, and ERBs from the first round of the 48 agents. Additionally, the robustness of our framework against communication failure is tested, and 50%, 75%, 90% dropout rate is applied to the number of ERBs each agent receives, meaning only 50%, 25%, 10% ERBs reached the agents.Result: Figure 7 shows the difference in average distance error when using different dropout rates during ERB sharing. 90% dropout rate performs the worst out of the three dropout rates, with higher average distance error and higher deviation in performance. Performance 75% and 50% are very similar, with 50% dropout rates having a slightly lower average distance error. Compared to no sharing in round 1, sharing just a tiny fraction (10%) of the ERB in the system still can have a massive boost in performance. Figure 7 shows the highest dropout rate the new 48 agents can accommodate without losing significant performance.  
 
 Appendix:  


Decision: All reviewers found the proposed method to be novel and the results promising. 3 weak accept and 1 borderline accept recommendations from reviewers.

Reviews: review: paper_summary: paper_summary: The authors propose a privacy-focused, asynchronous federated learning system that allows for efficient addition and removal of agents to and from the system. The approach was evaluated on anatomy localization based on the BraTS dataset.
main_review: main_review: The authors have appropriately addressed all my concerns, and I would like to thank them for actively participating in the rebuttal. Acknowledging the comments of the other reviewers, and the explanations I got from the authors for my review, I am keeping my original score.
strength_weakness: strength_weakness: The results are preliminary, implementing the system for a task with a larger dataset would better describe the effectiveness of the agents learning from each other. The small number of training iterations might also affect the presented results.
questions: questions: - Please address how the number of training rounds were determined. Wouldn't it show a more complete picture of the system if all agents were trained until convergence on a validation dataset?
- Could the authors describe more technical details of what experience (eg. its dimensions) is shared between the agents?
paper_summary: paper_summary: This work proposes Asynchronous Decentralized Federated Lifelong Learning (ADFLL). This method combines the advantages of federated learning and lifelong learning, enabling multiple agents to collaboratively train models without sharing raw data. Compared to traditional federated learning, the ADFLL method does not require a central node, can handle data and agent heterogeneity, and can continuously learn multiple tasks, avoiding catastrophic forgetting.

Deep Reinforcement Learning (DRL) framework: The paper proposes a DRL framework that utilizes the Deep Q-Network (DQN) algorithm for landmark localization tasks. This framework demonstrates flexibility and effectiveness in handling medical imaging data and can adapt to different task environments.

Lifelong learning with selective experience replay: To prevent catastrophic forgetting, the paper employs selective experience replay, focusing on selected experiences from past tasks. This approach enables the model to retain knowledge of previous tasks while learning new ones, thereby enhancing generalization and accuracy.

Distributed database system: To manage communication among agents effectively, the paper adopts a distributed database system that connects agents with central nodes, facilitating asynchronous communication and experience sharing. This system architecture ensures robustness in the event of node or central node failures and manages communication complexity efficiently.

In summary, the paper introduces a novel approach that integrates federated learning, lifelong learning, and deep reinforcement learning, effectively addressing privacy, data heterogeneity, and catastrophic forgetting issues in medical image analysis, thereby offering new solutions to the field.
main_review: main_review: While the method shows promise in analyzing medical images, it appears to be a general solution rather than specifically tailored for medical imaging needs. This lack of specialization could lead to suboptimal performance in certain situations where specific medical requirements are crucial. Moreover, the paper's visuals, such as figures and charts, lack aesthetic appeal, potentially hindering readers' comprehension and engagement with the content. Improving the visual presentation could enhance the paper's readability and attractiveness. Additionally, the absence of detailed comparisons with similar methods makes it challenging for readers to gauge the method's performance advantages. Comparative experiments are vital for validating its superiority and applicability across various scenarios. Furthermore, the method's complexity and computational demands may pose practical challenges, potentially requiring significant computational resources or leading to high costs. Lastly, if the models or algorithms lack interpretability, it could hinder their acceptance and usability in clinical settings, as healthcare professionals may struggle to understand and trust the results.
strength_weakness: strength_weakness: While the method performs well in medical image analysis, it seems to be a general approach rather than specifically designed for medical imaging. This may suggest that it lacks optimization for specific medical field requirements, potentially resulting in suboptimal performance in certain scenarios.

The figures and charts in the paper may lack aesthetic appeal, which could affect readers' comprehension and engagement with the content. Visually appealing illustrations and charts can better showcase methods and results, enhancing the paper's readability and attractiveness. (core drawback)

The paper may lack detailed comparative experiments with other similar methods, making it difficult for readers to assess the method's performance advantages. Comparative experiments can help validate the method's superiority and its applicability in different scenarios.  (core drawback)

The method may involve complex algorithms and computational processes, potentially leading to high computational costs in practical applications or requiring more computational resources support. (small drawback)

If the models or algorithms involved in the method lack sufficient interpretability, it may face challenges in explaining results and credibility in practical applications, limiting its acceptance and application range in clinical settings.  (small drawback)
questions: questions: I recommend authors not to rebuttal, they should consider submitting the paper after improving the effectiveness of experimental analysis and enhancing the aesthetics and visual appeal of the figures and charts.
paper_summary: paper_summary: The paper introduces an Asynchronous Decentralized Federated Lifelong Learning (ADFLL) framework aimed at overcoming the limitations of conventional federated learning in medical imaging, specifically landmark localization. It allows for continual learning from peers without a central node, enabling the system to adapt to new tasks and environments without catastrophic forgetting. The framework was evaluated using the brain tumor segmentation (BraTS) dataset, demonstrating its efficacy in maintaining high performance across various imaging environments and under conditions of significant agent turnover and network dropout. The ADFLL framework showed robustness, scalability, and flexibility, with performances comparable to centralized baselines, making it a promising approach for real-world applications in medical imaging AI.
main_review: main_review: Based on the author's rebuttal and updated paper, this paper has improved, and the author successfully addressed some problems. I think this paper has shown a good work.
If this paper can be accepted, I will suggest the author could improve the writing of this paper, to help people to understand easier.
strength_weakness: strength_weakness: 1. For section 3.3, can you break into sections to clearly explain your experiments? This is the most important part of your article, it is very confusing to read. I spent a lot of time reading this section and still diffucult to read and understand.
a) "As shown in Fig.3, we also see that the average Euclidean distance error across all agents decreases, while half of the agents are deleted every round, resulting in an average Euclidean distance error of 8.55 ± 7.12 after for the final remaining agent at the end of 5 rounds."
This sentence is difficult to understand, why error decreases?
b) also, you need to make your table or your content explicitly clear, what are X, Y, M, which one is the upper and lower bound?  We don't remember, we need to go back to your content to search again.

2. Because there is a page limit for htis submission, you need to reduce the length of section 3.2, section 4 as well as section 1. You need to use more paragraphs to clearly explain your work to authors
3. The plot (Figure 3) and table 1 cannot be screenshots. They are not acceptable. We may question about the authenity of the experiment.
questions: questions: Can you re-explain your experiment, i.e. section 3.3 again?
paper_summary: paper_summary: The paper presents the asynchronous decentralized federated lifelong learning (ADFLL) method as a solution to the limitations of traditional federated learning, enabling lifelong learning in a decentralized, privacy-preserving manner. ADFLL allows agents to continuously learn from both personal and network-wide experiences, efficiently integrating new tasks and nodes. Experiments use the BRATS dataset in simulation and deployment setups, and ADFLL demonstrated remarkable performance in retaining knowledge and adapting to new agents, matching or outperforming centralized and conventional lifelong learning approaches.
main_review: main_review: I am satisfied with the author's response regarding the motivation and the integration of reinforcement learning, addressing my concerns. And I accordingly increase my final score. Additionally, it would be beneficial to provide further clarification on the IID Federated Learning (FL) setting and explain why the current version focuses exclusively on the IID setting.
strength_weakness: strength_weakness: - The motivation behind the study is somewhat unclear; the authors should more explicitly emphasize the significance of the ADFLL framework for medical imaging tasks, particularly outlining the primary motivation for evaluating landmark localization tasks using the BraTS dataset.
- Additionally, the rationale for incorporating deep reinforcement learning within the ADFLL framework is not immediately apparent. From my perspective, this task could potentially be accomplished through a straightforward supervised learning approach.
questions: questions: In the description of section 2.5, the client data partitioning appears to be divided into IID distributions, failing to introduce substantial data heterogeneity across different clients.
paper_summary: paper_summary: An asynchronous decentralized federated lifelong learning (ADFLL) method that allows lifelong learning agents to continually learn from their own previous experiences, as well as experiences from other nodes in the network, was proposed, thus overcoming the potential drawbacks of conventional federated learning. An evaluation of the ADFLL framework in two experimental setups was setup: a simulation setup where we simulate addition and deletion of nodes in a FL setup and a deployment setup where different nodes working on different tasks are initialized on different machines with different compute.
main_review: main_review: The authors lay the foundation of a promising federated learning system that allows for agents to learn from the experience of other agents. They found no significant differences between their system and the baseline model that had access to all training data highlighting ADFLL's effectiveness.

The reviewer would like to recommend to accept this article, I cannot be confident on this as I am not an expert in Reinforcement Learning.
strength_weakness: strength_weakness: 1. The authors claimed that they developed a new framework, but they failed to explain the details.
2. The authors failed to explain the combination of federated lifelong learning and deep reinforcement learning.
3. Experiments are not sufficient. Only one dataset was included, and no baseline comparison.
questions: questions: Please see the weaknesses.
The authors should address the novelty of their method in rebuttal.

Comments: title: Rebuttal to Reviewer 1
Rebuttal to Reviewer 5
Rebuttal to Reviewer 4
Rebuttal to Reviewer 3
Rebuttal to Reviewer 2
comment: Thank you for taking the time to provide valuable insights on our submission. Please see below the point-by-point response to the comments (in bold):

**The reviewer believes the manuscript would benefit greatly with a publicly available code repository describing how to set up ADFLL for the interested reader.**

Thank you for your feedback. The code is publicly available at https://github.com/vishwaparekh/ADFLL. We have added the link to the manuscript.

**The results are preliminary, implementing the system for a task with a larger dataset would better describe the effectiveness of the agents learning from each other. The small number of training iterations might also affect the presented results.**

We agree with the reviewer that the results are preliminary, and we would like to address this with additional experiments we conducted that involve landmark localization of the spleen on five different imaging sequences from three modalities: MRI Dixon water, MRI Dixon fat, MRI T1, CT, and PET. We have also validated our model performance on external datasets from Medical Segmentation Decathlon and The Cancer Imaging Archive **The clinical data and the experimental details are provided in Sections 3.2 and 3.3 in the updated manuscript**.


**Please address how the number of training rounds were determined. Wouldn't it show a more complete picture of the system if all agents were trained until convergence on a validation dataset?**

The reviewer raises an important point. We determined the minimum number of training rounds as the number of training rounds necessary for all the agents in the system to be exposed to all the tasks within the system, whether through direct datasets or ERBs. Thus the minimum training rounds = 1+number of tasks/number of agents which resulted in three rounds for the BRATS dataset experiment. 

The goal of the ADFLL setup was to enable peer-to-peer experience sharing to learn new unseen tasks. As a result, we developed ADFLL with the hypothesis that different agents will only have a partial view of the complete validation set defined by the tasks they encounter and would not have access to the complete validation dataset that captures all the environments being trained in the framework. 

**Could the authors describe more technical details of what experience (eg. its dimensions) is shared between the agents?**

The reviewer brought up a critical question about the clarification of ERBs. The ERB produced after DRL training is integral to our framework. An ERB contains sampled state-action-reward-resulting-state tuples that capture the DRL agents’ explorations of the image environment. The environment is represented by 3D imaging volume, and the agent was represented by a 3D bounding box capable of six distinct actions: moving positive or negative in the x, y, or z direction. The state was defined by an agent’s current location (or a chain of locations), each represented by a 3D bounding box of 45 × 45 × 11 pixels. The reward is calculated by the change in distance to the target landmark location before and after the agent takes action. The resulting state is the state of the agent after it takes an action. Thus the information in the ERBs is non-sensitive as the action and reward are information about the training session, and the state and resulting states are tiny fractions of the total 3D image, roughly 0.3%.
Thank you for taking the time to provide valuable insights on our submission. Please see below the point-by-point response to the comments (in bold):

**1. The authors claimed that they developed a new framework, but they failed to explain the details. The authors failed to explain the combination of federated lifelong learning and deep reinforcement learning.**

We have updated our introduction and methods sections to improve readability and explain the combination of federated lifelong learning and deep reinforcement learning. In addition, we have also added a new Figure 1 to illustrate the combination between peer-to-peer experience sharing, lifelong learning, and deep reinforcement learning. 

We have added the following paragraph to the introduction
"We develop and evaluate ADFLL using deep reinforcement learning (DRL) as the base model architecture at each node for the task of 3D landmark localization in medical imaging volumes. DRL with LL capability using selective experience replay (SERIL) has previously been validated for continual landmark localization across different imaging environments using DRL for medical imaging and also forms the basis of LL for each node in ADFLL"




**2. Experiments are not sufficient. Only one dataset was included.**

We have added a new experiment to address this limitation of our original submission. The new experiment focuses on cross-domain 3D localization of spleen on five different imaging sequences from three modalities: MRI Dixon water, MRI Dixon fat, MRI T1, CT, and PET. In addition to learning from ERBs encoding different modalities, we also evaluated the trained ADFLL agents on multiple external datasets from Medical Segmentation Decathlon (MSD) and The Cancer Imaging Archive (TCIA). **The clinical data and the experimental details are provided in Sections 3.2 and 3.3 in the updated manuscript**

**3. No baseline comparison**

The ADFLL framework proposed in this work evaluates the capability of lifelong learning agents to learn from each others' experiences for cross-domain landmark localization in medical images. Across all our experiments, we have included comparison to two upper-bound and one lower-bound baseline models, detailed below:

Agent X: The "upper bound" best case scenario where all the tasks being shared across the network are centrally available for training. 

Agent Y: The "lower bound" worst case scenario for centralized training, when the agent is only presented with 1 task. 

Agent M: The "upper bound" lifelong learning scenario for centralized training, when tasks and data are available to the agent sequentially (one dataset available after each training round). This provides comparison between training from ones own experiences (traditional LL) and training from each others experiences (ADFLL proposed in this work)
Thank you for taking the time to provide valuable insights on our submission. Please see below the point-by-point response to the comments (in bold):

**1. The motivation behind the study is somewhat unclear; the authors should more explicitly emphasize the significance of the ADFLL framework for medical imaging tasks, particularly outlining the primary motivation for evaluating landmark localization tasks using the BraTS dataset.**

Thank you for your feedback. we have clarified and emphasized the significance of our framework for medical imaging tasks in the abstract and introduction. More specifically, we emphasized the capability of our framework to learn from cross-domain experiences shared by different agents in the network enabling learning in newer unseen domains. To further emphasize the significance of the proposed framework, we have added a new experiment focused on cross-domain 3D localization of spleen on five different imaging sequences from three modalities: MRI Dixon water, MRI Dixon fat, MRI T1, CT, and PET. In addition to learning from ERBs encoding different modalities, we also evaluated the trained ADFLL agents on multiple external datasets from Medical Segmentation Decathlon (MSD) and The Cancer Imaging Archive (TCIA). **The clinical data and the experimental details are provided in Sections 3.2 and 3.3 in the updated manuscript**

**2. Additionally, the rationale for incorporating deep reinforcement learning within the ADFLL framework is not immediately apparent. From my perspective, this task could potentially be accomplished through a straightforward supervised learning approach.**

The proposed ADFLL framework is compatible with any memory based deep lifelong model as the base architecture at different nodes. This work uses deep reinforcement learning (DRL) as an example to demonstrate and evaluate the capabilities of the ADFLL framework. While the task of 2D object detection can be accomplished using a straightforward object detector, DRL has demonstrated excellent performance in tasks involving 3D landmark localization in the literature [1, 2]. Therefore, we evaluated the ADFLL framework for 3D landmark localization in this work. 

**3. In the description of section 2.5, the client data partitioning appears to be divided into IID distributions, failing to introduce substantial data heterogeneity across different clients.**

The reviewer raises an important point. We agree that our initial experiment sampled all the environments from the same imaging dataset leading to IID distributions. Therefore, we added a new experiment focused on cross-domain 3D spleen localization to demonstrate the capability of the proposed ADFLL framework in non-iid settings. Please refer to our response #1. The new experiment samples environments from three different modalities from two distinct datasets acquired using different scanners (PET/CT vs. MRI). Furthermore, we evaluated our models on multiple external datasets sourced from MSD and TCIA. 

References

1. Florin-Cristian Ghesu, Bogdan Georgescu, Yefeng Zheng, Sasa Grbic, Andreas Maier, Joachim Hornegger, and Dorin Comaniciu. Multi-scale deep reinforcement learning for real-time 3d-landmark detection in ct scans. IEEE transactions on pattern analysis and
machine intelligence, 41(1):176–189, 2017.
2. Amir Alansary, Ozan Oktay, Yuanwei Li, Loic Le Folgoc, Benjamin Hou, Ghislain Vaillant, Konstantinos Kamnitsas, Athanasios Vlontzos, Ben Glocker, Bernhard Kainz, and Daniel Rueckert. Evaluating Reinforcement Learning Agents for Anatomical Landmark
Detection. Medical Image Analysis, 2019.
Thank you for taking the time to provide valuable insights on our submission. 

- We have shortened and clarified the abstract
- We have updated Figure 1 (now is Figure 2) so the text is more readable.
- We have thoroughly revised and checked the typos in our paper.
- We have updated experimental details for all our experiments to improve readability. 
- We have updated the table of results and highlighted key results.
- We have updated the results section to explain the table of results better.

We have added an additional experiment to our manuscript to highlight our contribution. The new experiment focuses on cross-domain 3D organ localization of the spleen on five different imaging sequences from three modalities: MRI Dixon water, MRI Dixon fat, MRI T1, CT, and PET. We have also validated our model performance on external datasets: the MSD spleen dataset, PET and MRI datasets from The Cancer Imaging Archive. **The clinical data and the experimental details are provided in Sections 3.2 and 3.3 in the updated manuscript**

We hope this addresses all the concerns you have with our paper. Again, we would like to thank you for your time and feedback!
Thank you for taking the time to provide valuable insights on our submission. Please see below the point-by-point response to the comments (in bold):

**While the method performs well in medical image analysis, it seems to be a general approach rather than specifically designed for medical imaging. This may suggest that it lacks optimization for specific medical field requirements, potentially resulting in suboptimal performance in certain scenarios.**

We agree with the reviewer that our framework is a general approach. However, while this study can be generally applied to any application, we have targeted the specific problem of 3D landmark localization for medical imaging. We have expanded our experiments to include 3D cross-modality organ localization across different modalities (PET, CT, and MRI) to demonstrate the capability of the proposed method in medical imaging. Furthermore, we have included validation on external out-of-distribution datasets as well. **The clinical data and the experimental details are provided in Sections 3.2 and 3.3 in the updated manuscript**


**The figures and charts in the paper may lack aesthetic appeal, which could affect readers' comprehension and engagement with the content. Visually appealing illustrations and charts can better showcase methods and results, enhancing the paper's readability and attractiveness. (core drawback)**

We thank the reviewer for giving us feedback on the representation of our framework using figures and charts. We have improved them and the overall clarity of the manuscript to make it more visually appealing and easy to follow.


**The paper may lack detailed comparative experiments with other similar methods, making it difficult for readers to assess the method's performance advantages. Comparative experiments can help validate the method's superiority and its applicability in different scenarios. (core drawback)**

The ADFLL framework proposed in this work evaluates the capability of lifelong learning agents to learn from each others' experiences for cross-domain landmark localization in medical images. Across all our experiments, we have included comparison to two upper-bound and one lower-bound baseline models, detailed below:   

Agent X: The "upper bound" best case scenario where all the tasks being shared across the network are centrally available for training.
Agent Y: The "lower bound" worst case scenario for centralized training, when the agent is only presented with 1 task.
Agent M: The "upper bound" lifelong learning scenario for centralized training, when tasks and data are available to the agent sequentially (one dataset available after each training round). This provides comparison between training from ones own experiences (traditional LL) and training from each others experiences (ADFLL proposed in this work)

**The method may involve complex algorithms and computational processes, potentially leading to high computational costs in practical applications or requiring more computational resources support. (small drawback)**

The reviewer brings up an excellent point about high computational costs in practical applications. However, the primary goal of this work was to demonstrate a proof of concept in peer-to-peer experience sharing with lifelong learning for medical imaging applications. In this work, we did address the issue of high network bandwidth requirement in an all-to-all communication setup with the hub-node-hub optimized communication setup. In addition, different techniques have been proposed in the literature to reduce the size of ERBs [1] as well as reduce the overall computational complexity of lifelong learning through image compression [2]. In our future work, we plan to integrate different optimization approaches for reduce our framework's overall computational complexity.  

**If the models or algorithms involved in the method lack sufficient interpretability, it may face challenges in explaining results and credibility in practical applications, limiting its acceptance and application range in clinical settings. (small drawback)**

The reviewer brings up another excellent point regarding algorithmic interpretability. This was also a primary motivation behind using memory based lifelong learning technique in our work. The selective experience replay buffers shared between agents in our framework are transparent and interpretable. In case of failures, ERBs shared between different agents can be evaluated and improved in its information content. 

References

[1] Zheng, Guangyao, et al. "Selective experience replay compression using coresets for lifelong deep reinforcement learning in medical imaging." Medical Imaging with Deep Learning. PMLR, 2024.

[2] Zheng, Guangyao, et al. "A framework for dynamically training and adapting deep reinforcement learning models to different, low-compute, and continuously changing radiology deployment environments." arXiv preprint arXiv:2306.05310 (2023).