Distributed Multi-Agent Lifelong Learning
Authors: Prithviraj Tarale
Edward Rietman
Hava T Siegelmann
Venue: Accepted by TMLR ()

Abstract: Lifelong learning (LL) machines are designed to operate safely in dynamic environments by continually updating their knowledge. Conventional LL paradigms often assume that new data come labeled and that each LL machine has to learn independently from its environment. However, human labeling is expensive and impractical in remote conditions where automation is most desired. We introduce the Peer Parallel Lifelong Learning (PEEPLL) framework for distributed Multi-Agent Lifelong Learning, where agents continually learn online by actively requesting assistance from other agents instead of relying on the expensive environment to teach them. Unlike classical distributed AI, where communication scales poorly, lifelong learners need to communicate only on information they have not yet learned. Additionally, agents reply only if they are highly confident: Our TRUE confidence score uses a compute-efficient application of Variational Autoencoder to quantify confidence in prediction without needing data reconstruction. TRUE outperforms traditional Entropy-based confidence scores, reducing communication overhead by 18.05\% on CIFAR-100 and 5.8\% on MiniImageNet. To improve system resilience to low-quality or adversarial responses, our agents selectively accept a subset of received responses using the REFINE algorithm, which results in a 51.99\% increase in the percentage of correct accepted responses on CIFAR-100 and 25.79\% on MiniImageNet. Like traditional LL agents, PEEPLL agents store a subset of previously acquired knowledge as memory to learn alongside new information to prevent forgetting. We propose a Dynamic Memory-Update mechanism for PEEPLL agents that improves QA's classification performance by 44.17\% on CIFAR-100 and 26.8\% on MiniImageNet compared to the baseline Memory-Update mechanism. Our findings demonstrate that a PEEPLL agent can outperform an LL agent even if the latter has environmental supervision available, thus significantly reducing the need for labeling. PEEPLL provides a framework to facilitate research in distributed multi-agent LL, marking a substantial step towards practical, scalable lifelong learning technologies at the edge.

Published in Transactions on Machine Learning Research (01/2025)
Distributed Multi-Agent Lifelong Learning
Prithviraj P. Tarale
plarale@umaass.cmm edu
Deportment of Computer Science
University of Massachusetts Amherst Amherst, MA 01003
Edward Rietman
erietman crietman@umas. edu
Deportment of Computer Science
University of Massachusetls Amherst Amherst, MA 01003
Hava T. Siegelmann
hava Grumass...... edu
Department of Computer Science
Universily of Massachusetts Amherst Amherst, MA 01003
Reviewed OpenReview: https:// o/per//vee...... net/ formm? id= HIVVrHuuus
Abstract
Lifelong learning (LL) machines are designed to operate safely in dynamic environments by continually updating their capabilities. Conventional LL paradigms often assume that new data come labeled and that each L.I. machine has to learn independently from its environ- ment. However, human labeling is expensive and impractical in remote conditions where automation is most desired. We introduce the Peer Parallel Lifelong Learning (PEEPLL) framework for distributed Multi-Agent Lifelong Learning, where agents continually learn online by actively requesting assistance from other agents instead of relying on the expen- sive enviromment to teach them. Unlike classical distributed AI, where communication scales poorly, lifelong learners need communicate only on information they have not yet learned. Additionally, agents reply only if they are highly confident: Our TRUE confidence score rep- resents application of a Variational Classifier to quantify confidence in prediction. TRUE outperforms traditional Entropy-based confidence scores, reducing communication overhead by 18.05% on CIFAR-100 and 5.8% on MinilmageNeet. To improve system resilience to low- quality or adversarial responses, our agents selectively accept a subset of received responses using the REFINE algorithm, which results in a 51.99% increase in the percentage of cor- rect accepted responses on CIFAR-100 and 25. 79% on MinilmageNet. Like traditional LLL agents, PEEPLL agents store subset of previously acquired knowledge as memory to learn alongside new information to prevent forgetting. We propose Dynamic Memory- Update mechanism for PEEPLL agents that improves QA's 's classification performance by 44.17% on CIFAR-100 and 26.8% on MinilmageNet compared to the basclinc Memory-Updas mecha- nism. Additionally, we demonstrate that PEEPLL agent can outperform an LL agent even the latter has environmental supervision available, thus significantly reducing the need for labeling. PEEPLL provides framework facilitate rescarch distributed multi-agent LL, marking a substantial step towards practical, scalable lifelong learning technologies at the edge. Our code is available at httpss//giihbb.ccm//Pttiiiiniii
1 Introduction
Lifelong Learning (LL) is an emerging field of AI that seeks to continually improve system capabilities with runtime experience while preventing catastrophic forgetting of previously learned experiences (Aljundi et al, 2019b a Mai et al. 2020 Chaudhry al.] 2018b 2019). It makes no assumptions about how the data appear in the real environment. To be context-aware LL must include online lifelong learning (OLL), where models learn online from each data point only once. This is in addition to offline global consolidation when

---

Published in Transactions on Machine Learning Research (01/2025)
time permits. OLL is particularly relevant for edge conditions where agents must be highly autonomous and adaptive.
State-of-the-art LL paradigms often assume that all incoming new data are labeled through environmental supervision and that LL agents must learn individually from their environments. This assumption is limiting the applicability of AT. In realistic environments, agents may have varied skills that they learned through their particular experiences. We propose a paradigm in which agents can learn from their peer agents in addition to learning from the environment. This improves the system's 's adaptability and reduces reliance on expensive environmental supervision. This study focuses on presenting and solving distributed malti-agent OLL, which has numerous applications, from self-driving vehicles to drones and more. Unlike classical distributed AI, our agents continually learn and communicate only during unprecedented events, progressively reducing inter-agent commmicattin as such events become precedented; this limits communication among agents. Reducing reliance on environmental supervision is important to make LL applications more seamless for realistic and edge environments.
Recent literature motivates the urgency to incorporate a multi-agent perspective into LL paradigms. The authors who participated in DARPA's Shared Experience Lifelong Learning (ShELL) program argue that "One vision of future artificial intelligence (AI) is where many separate units can learn independently over a lifetime and share their knowledge with each other. The result is a network of agents that can quickly respond to and learn new tasks, that collectively hold more knowledge than a single agent and that can extend current knowledge in more diverse ways than a single agent. We propose that the convergence of such scientific and technological advances will lead the emergence of new types of scalable, resilient and sustainable AT systems." (Soltoggio al. 2024). They suggest, among possible applications, Multi-Agent Active Sensing, Space Exploration, Responsive and Personalized Medicine, and Distributed Cybersecurity Systems.
Multi-Agent Machine Learning (MAML) facilitates agent collaboration either by solving problems jointly during deployment (Foerster et al. 2016 Gupta et al., 2017 Hüttenrauch et al. 2017) or by distributing tasks among agents during training and integrating their efforts during testing (Raja et al. 2022 Kim et al., 2022] Brito al.) 2021). However, the integration of MAML into LL introduces new challenges and necessitates reevaluation of the traditional MAML and LL methods. A primary challenge in MAML scttings is that the communication overhead between agents scales poorly as the number of agents increases. Furthermore, unique limitation arises when integrating MAML into LL: communication protocols in MAML are often based on past interactions, reputations, or a fixed, known task distribution among agents (Das et al. 2018; Jiang & Lu 2018 Hoshen 2017 Das et al.) 2017). This limiting assumption does not hold for LL agents who continuously update their knowledge. Finally, agents must navigate learning from potentially incorrect peer responses. To address these issues, we introduce framework to facilitate multi-agent coordination for LL agents: Peer, Parallel Lifelong Learning (PEEPLL, pronouneed 'People').
1.1 Peer, Parallel Lifelong Learning (PEEPLL)
PEEPLL's mechanism and framework are designed to navigate the dynamic, uncertain environments typical of multi-agent LLI settings; it focuses on judicious peer communication without overwhelming the system with excessive communication. Multi-agent cooperation across domains of fields helps reduce reliance on annotated data (environmental supervision). Regardless of the domain, in any decentralized multi-agent LL system, agents must be self-aware- - able to autonomously identify novel tasks, actively seek help, responsibly assist others, and carcfully evaluate received help.
Learning is also increasingly being Qin et al. 2023). robotics (Lesort Vokinger et al. 2021p. where agents develop sub=domain-exxperuis in their respective environments through continual learning. The principles underlying PEEPLL are applicable across domains. In language modeling, for instance, PEEPLL can allow LLM agents to communicate with peers when handling uncertain prompts, allowing them to continuously learn to generate accurate and contextually relevant responses. The received peer responses can also be used for future fine-tuning. In high-stakes medical AI, PEEPLL can allow agents to seek assistance when
While this study focuses supervised image classification, Lifelong
applied in domains like language modeling (Gururangan et al.
2020
ct al. 2019: Logacjov Aleksej] 2021), and medical AT (Lee & Lee 2020
2

---

Published in Transactions on Machine Learning Research (01/2025)
uncertain and take only confident and well-validated actions. In robotics, PEEPLL facilitate real-time collaboration, enabling robots to responsibly exchange sub-domain-specific expertise and adapt to new tasks (either online or offline, depending on the application). The Framework section below provides domain- agnostic guidelines for developing dedicated algorithms to the components of PEEPLL.
Our solution PEEPLL demonstrates that a PEEPLL agent with no environmental supervision can perform better on novel tasks than an LL agent with complete environmental supervision. This advancement helps minimize disruptions to the user experience and progresses toward realizing seamless ShELL technologies.
When an agent, designated Query Agent (QA), receives query that it cannot confidently answer, it seeks assistance from other peer agents in the network, designated Response Agents (RAs). The RAs respond if they are confident in their answers. The QA then selects a subset of these responses for lifelong learning. Please refer to Figure that illustrates the PEEPLL mechanism in detail.
Mechanism:
Clsseiootion/ Ourpur
Query Packet
Classifier
Send Query
Classifier
Send Query
if QA-C Thresh.
QA-C Thresh.
Query
(
Learn
Confidence (QA-C)
Resoonic
Q
Confidence (RA-C)
Respouse Ir RA-C (QA-C Thresh.)
(QAIC Thresh.)
Momery
Menoey
Readive Resparoxs
Rendie Responsss
Updare
Selective Filtering
Update
Selecrive Filcering
Ellared Rapassos
Prodicted Clas
2. Confidence: R.A
3. C.Caimmd Known-Clloee Classes
Figure 1: PEEPLL's Query-Reponse Mechanism: The agent receiving the original query is designated the Query Agent (QA), while the agents it seeks assistance from termed Response Agents (RAs). All agents are identical and can function interchangeably as either QA RA, depending on the recipient of the query. Upon receiving query, the QA assesses its confidence in providing an accurate response. If underconfident, the QA queries other agents in the network (RAs). RAs then evaluate their confidence in responding to the query. confident, they transmit their responses back to the QA. Finally, the QA applies Selective Filtering to accept only a subset of the received responses. The QA then learns these accepted responses in conjunction with replay samples from its memory (circle with R) while updating its memory with the accepted responses.
Framework: We identify three major components of the PEEPLL mechanism that are essential for effective multi-agent coordination in lifelong learning technologies (1) Confidence-Evvalstiion (2) Selective Filtering, and (3) Lifelong Learning in PEEPLL. In the remainder this section, we detail the components and initial benchmarks of those components.
Sisidencc-Evaluationn This component enables (a) the QA to autonomously recognize novel lasks and request assistance when needed and (b) the RAs evaluate their ability to assist the QA reliably, transmitting only likely correct and relevant responses. The primary objective is (1) restrict communication to essential queries and responses and (2) promote stable learning at the QA.
Given the continual evolution of LL agents' knowledge, Confidence Evaluation must operate beyond reliance on historical performance or preexisting knowledge bases, ensuring assessments are agent- agnostic and contextually relovant to each spocific query. Morcover, the confidence score must retain its efficacy even after the agent has lifelong learned.
Our proposed TRUE confidence score reduces communication overhead by 18.05% on CIFAR-100 and 5.8% Mini- TmageNete compared to the widely accepted and effective 'Entropy' based meth
3

---

Published in Transactions on Machine Learning Research (01/2025)
Yona et al., 2022 Chen et al. 2021 (see Section 4.1). TRUE serves as general contribution to the ML. community for uncertainty quantification.
2. Sclective Response Filter: After receiving responses from all the confident RAs, the QA further accepts only the most reliable responses. This component is important to security in multi-agent systems, assuring that an agent does not simply choose the first or the most confident respouse, which might have been adversarial.
Our most effective method increases the proportion of correct responses in the selected responses by 51.99% on CTFAR100 and 25.79% on NinilmageNett (Section 4.2).
3. Lifelong Learning in PEEPLL: Integrating MAML into LL requires reevaluating traditional LL concepts. Replay-based LLI agents allocate memory to store samples from previous experiences and learn alongside new information to prevent forgetting. In PEEPLL, incorporating incorrect responses from RAs into the QA's 's learning process and memory introduces new challenges:
(a) Hemory-Uppaate Algorithms must handle incorrect responses from RAs when updating QA's memory. These incorrect responses can fill the QA's 's memory budget and disturb learning, but we show that they may also have a beneficial regularization effect on learning (see Section 4.3).
Algorithms must dynamically retrieve appropriate memory samples by con- sidering (a) relevant current queries, (b) the QA's confidence in those queries, and (c) the confidence of the responding RAs. Algorithms can also consider the regularization elfect of noisy responses in selecting the most pertinent memory samples to improve performance.
(b) Memory-Relrieeee
(c) Adapliie-Learmingg Algorithms must modulate the importance of potentially incorrect responses based on their confidence. These algorithms may also consider the regularization effect of noisy responses to improve performance, balancing stability and regularization as a tradc-off.
Our proposed Memory- Update method improves QA's 's local performance on all lifelong learned tasks by 44.17% on CIFAR-100 and 26.8% on MinilmageNet (see Section 4.3).
We present the conceptual (i) and technical contributions (ii, iii) of this paper below:
(i) We introduce PEEPLL, a framework for solving distributed Multi-Agent Lifelong Learning, addressing the unique challenges of integrating multi-agent. coordination with lifelong learning.
(ii) We set initial benchmarks for the components of PEEPLL. Within our solution to PEEPLL, agent with no environmental supervision achieves higher lifelong performance than an LL agent with complete environmental supervision while minimizing communication overhead.
(iii) We introduce an application of a Variational Classifier for superior uncertainty quantification compared to Entropy- based methods.
2 Related Work
Lifelong Learning (LL) seeks to enable AI systems to contimually improve their capabilities through experience beyond the limited-time training phase. Old versions of LLL described systems that updated data structures while leaving the underlying neural networks unchanged (Thrun & Mitchell 1995] Thrun 1998 Ruvolo & Eaton 2013). Following DARPA's L2M program, modern LL emphasizes continual updates to the neural network itself, iee. the underlying computational AT model. This paradigm shift is supported by the Super- Turing computational thcory (Sicgelmann) 1995 1998), an alternative to the Turing computing model. Unlike Turing compulation, which relies on fixed program to process input, Super-Turing like biological systems, adapts its program based on input; this is the basis for LL. Due to its ongoing adaptability, LL, like Super-Turing, is computationally more powerful and produces richer output behaviors than the fixed Turing model or traditional neural networks. PEEPLL extends these principles to a distributed multi-agent setup to make LL more practical and scalable by reducing reliance on environmental supervision.
We next discuss Single-Agent Lifelong Learning, given the literaturo's gap in Multi-Agent Online Lifelong Learning, and Multi-Agent Lifelong Learning for offine implementations. Finally, we include the rationale behind the choice of a fully distributed setup in our study.
4

---

Published in Transactions on Machine Learning Research (01/2025)
Single-Agent Lifelong Learning: Lifelong learning strategies can be classified into three categories: Memory-Based, Architecture-Basee and Regularization-Base Methods. We focus on Memory-Based Life- long Learning strategies within the Incremencellltal setting, where data of unknown classes are presented soquentially in distinct, non-overlapping 'tasks' to the lifelong learning agent. Architecture-based methods are incompatible with this setting (Mai et al.] 2021 hence, we do not locus on these. Regularization--ase methods such as Elastic Weight Consolidation (EWC) (Kirkpatrick et al.) 2016) struggle with this setting (Lesort et al., 2021). EWC prevents forgetting by penalizing changes to parameters important for previously learned tasks, using a Fisher information matrix to measure importance. Memory-based lifelong learning methods mitigate forgetting by storing and replaying (relearning) data. from previously seen classes. This typically involves SemorryUUdatee selecting what to store and (leoory-Retrient
selecting what to replay. Mai et al. (2020) use Shapley values for informed ret.rieva;; Aljundi et al. (2019a) retrieve samples that max- imize learning interference; Aljundi et al. (2019b) focus on gradient diversity while storing in the memory; Chaudhry et al. (2018b) constraint the model's loss on its memory samples to mitigate forgetting; Prabhuu et al. (2020) (GDumb) adopt a simple, balanced sampling approach, retraining the network from scratch at test time. Despite its simplicity, GDumb is more effective than many strategies; however, it is an offline strategy. Finally, Experience Replay (ER in Table 2) (Chaudhry et al.
2019 uses random sampling for memory management (update and retrieval) and often surpasses other complex methods. ER is considered state-of-the-ar for Single- Agent Online Lifelong Learning (van de Ven et al.
2022 Naive strategies fine-tune the model on incoming data without replaying previously learned data. III strategies sample independently and identically from all of the training data.
Multi-Agent Lifelong Learning: Tn Rostami et al. [2017), each agent knows specific distributions of the dataset. They assume that this distribution is fixed and known. Their lifelong learning agent is designed to be aware of all classes and learns only from out-of-distribution data, known as the Domain--Incremental Scenario. Our work differs in two key aspects: Firstly, we do not predefine or know how tasks are distributed among agents, as this is not a fair assumption for lifelong learning agents. Secondly, our agents are strictly unaware of certain classes, focusing instead on tackle calastrophic forgetting in Federated Learning (FT.) using a global generative model for data-Iree knowledge distilation, focusing on privacy but increasing communication overhead and presenting a risk of single point of failure due to server contrality (centralized topology of communication). Instead, our work focuses on a Peer-to-Peer lopology.
Lifelong Learning. [Babakniya et al. (2023)
Multi-Agent Architecture After thoroughly evaluating communicatio topologies Verbracken et al. 2019), we selected a fully distributed peer-to-peen architecture for the PEEPLL system. Although centralized systems offer quick synchronization, they struggle with scalability and single-node failure risks; ral-ntral- ized systems improve scalability but still have communication overhead and node failure risks. Peer-to-peer networks avoid these issues but face challenges in synchronizing agent knowledge. PEEPLL addresses this through Confidence-Evahattinn methods for knowledge alignment and Lifelong Learning strategies for con- tinuous knowledge synchronization.
We compare our solutions to PEEPLL with ER, EWC, GDumb, Naive, and IID. We implement these with our modified data introduction regime (Section 4). Further, we propose that GDumb serves as a multi-agent LL baseline as well. This is because if we assume 100% accurate communication and that all agents in the network share a common memory, an agent can retrain its network using all relevant memory samples when encountering new classes. Although this is not fully distributed setup, it is a strong baseline to beat.
3
PEEPLL: Our Approach
This section presents our solutions to the components of the PEEPLL mechanism discussed in Section [.] (1) Confidence-FFvalattion Strategy, (2) Selective Response Filtering, and (3) Lifelong Learning in PEEPL..
3.1
Sifiidence--vlluatiin Strategy
We introduce a modified compute-cffcient version of Variational Autoencoders for image classification with- out input reconstruction to quantify uncertainty in model prediction. This architecture is similar to the re-
5

---

Published in Transactions on Machine Learning Research (01/2025)
cently proposed Variational Classifier (VC) by Dhuliawala et al 2024 Since our study focuses Memory- Based Lifelong Learning, where subset of experiences is stored in the agent's memory, our uncertainly quantification method is targeted for such agents.
We first discuss our Variational Classifier's Architecture and Training. Then, detail how the model's uncer- tainty prediction (inversely, its confidence prediction) discerned.
Architecture and Training: VC's architecture comprises two primary components:
1. Encoder: Takes the image imput ('x') and outputs ( and 2log var(z). [log_var(()) 2mean (z) and 2log var (z)
are used to define the probability distribution from which a latent code for the image is sampled.
The Encoder is optimized using Kullback Leibler divergence loss to facilitate a continuous latent space representation. The Kullback Leibler divergence loss can be mathenatically expressed as:
K Σ k=1
(L((o(<(<)ll(()))===== - (1+ fog
1
llg var (2) (# (I)) 2 e (fio)
where (z(x) is the approximate posterior distribution of z given x, parameterized by & (encoder), p(z) is the prior distribution over z (Gaussian), K is the dimension of Emeon and Vlog_ver, zhean is k-th entry of the 2mean vector, and varr is k-th entry of the zlog_ vector.
2. Decoder: Takes the latent code as input and outputs a classification layer. This component is trained using Cross-Entrops loss to predict the class of the input image. Note here that we forgo input roconstruction to conserve computational resources; hence, reconstruction loss is used for training.
Iifiidence-vvluattion mechanism: Given query the encoder evaluates = and 2 _oo_var' var and a latent code = is sampled. The decoder then takes in z and yields a prediction Pg for If there are associated memory samples for the label Py. the corresponding .memory and memory for each memory sample is retrieved. "log_ var Using these, we evaluate (1) Semantic Disparity, (2) Dispersion Disparity, and (3) Entropy, which are then averaged to produce the confidence score.
(1) Semantic Disparity captures the disparity in the property of the mean position of the query in the latent space versus the mean position of the model's memory of the predicted label for the query. We define Semantic Disparity using Euclidean distance:
dsemantic (g, memory) = =
Zmean
memory
where HiCHoor
represents N E memory; where N is the total number of retricved memory samples. &mean represents the mean of the latent variable's probability distribution for given input; it where the model predicts the center of the input's position in the latent space would be.
(2) Dispersion Disparity captures the disparity in the property of variance- the variance of the query versus the variance of the model's memory of the predicted label for the query. We define Dispersion Disparily using Manhattan distance:
ddispersion (q, mcmory) cxp(pp/p var) exp( (-og
memoty
memory; where 2log var represents Eig var where N the number of retricved memory samples. 2log is the logarithm of the variance of the distribution of the latent variable. Intuitively, the variance represents the spread or dispersion of the latent representations around zmean A larger variance (higher exp(21og var) indicates the model's higher uncertainty in pinpointing the class for similar images.
memoor
The choice of Euclidean distance for the Semantic Distance is to measure the spatial distance between the query's x_ mean and the memory samples' =_mean in the latent space, as 2_mean represents a meaningful 'physical' location in this space. In contrast, =_var does not correspond a physical spatial location but rather indicates the variance across dimensions. Therefore, we use the Manhattan Distance evaluate the sum of disparities between the query's and memory samples' variance across all dimensions. After
6

---

Published in Transactions on Machine Learning Research (01/2025)
experimenting with various distance measures, find our two chosen distance measures to be the most effective.
Semantic & Dispersion Disparity are transformed into confidence scores via a negative exponential function, exp(-a dsemantie) (denoted Csemantie), exp(-8 ddispersion) (denoted Caispersion). This ensures that the scores are bounded between [0, 1] and are inversely proportional to the assessed distances. a and 8 are scaling factors. The transformed distances are subsequently normalized (denoted Csemantics: and Edispersion respectively).
(3) Entropy (Shannon 1948) is defined as H(p) = - Et-llog,22 +c), where p is the probability distribution of the agent's ' predictions, and (a small number) prevents undefined logarithmic operations.
Finally, the mean of the three calculated to ensure that the resultant Triplet Uncertainty Evaluation (TRUE) confidence score falls within the range [0, 1]:
Esemantics + Cdispersion + Entropy
TRUE
3
Refer to Figure 12 and Algorithm 2 in the Appendix for a visualization and a pseudocode evaluate TRUE.
The QA uses TRUE to decide whether to initiate communication with the RAs,, and the RAs use TRUE to determine whether their responses should be transmitted back to the QA. We mcasure TRUE's effectiveness by the 'Sharing Accuracy' metric: the proportion of responses sent by RAs to the QA that were correct. We demonstrate that TRUE offers a higher Sharing Accuracy than Entropy-based measures. This implies that it is possible to generate meaningful latent representations despite forgoing reconstruction.
3.2 Selective Response Filtering
We propose three approaches to filter out pertinent responses at the QA once the RAs have responded.
Majority Voting: Response agents are grouped by the label they are predicting. The label with the highest number of agent endorsements is selected.
2. Clost-ConfidentGGG Group (MCG): Response agents are grouped by their predicted labels. The group with the highest average confidence selected.
Intelligent Comparative Filtering (ICF): The responses of RAs that predict the label 1 are rejected any other RA that has memory samples of the label does not predict class l.
The effectiveness of a filter lies in accurately identifying and accepting the correct responses (inereasing the proportion of correct responses in the accepted subset).
3.3 Lifelong Learning & Memory-Update
As the QA receives responses from RAs, it stores these responses and their corresponding confidence levels in a memory organized by class. We allocate a budget of 50 samples per class. To dynamize our Memory- Updale mechanism, once the memory reaches 50 samples for a class, it dynamically replaces the least confident. samples with new, higher- confidence responses.
4 Results & Analysis
We start this section outlining the modified traditional LL experimental setup to fairly evaluate solutions to PEEPLL's components and present the results in Sections 4.1-4.3.
Data Distribution and Agent Roles: The data in the training set divided into two parts: (1.1) pretraining data for all agents and (1.2) lifelong learning data for the QA. The pretraining data (1.1) is further divided into (1.1.Q-Pre) and (1...R-Pre). (1.1.Q-Pre) contains data from classes assigned for the QA's
7

---

Published in Transactions on Machine Learning Research (01/2025)
pretraining. (1.1.--Pre) contains data from classes assigned for the pretraining of the RAs. These classes are distributed among RAs, ensuring that 2-3 RAs are assigned to each class to foster diverse responses. Each RA is trained on data from its assigned classes in (1.I.R-Pre). From the lifelong learning data (1.2), the data for classes not in (1.1.Q-Pre) are extracted and referred to as (1.2.LL). (1.2.LL) is segmented into distinct "tasks,' each comprising a unique set of classes to facilitate the TJase-nncremenlal Scenario as described by Alijndii et al. (2019b a): Mai et al. (2020). Task, has classes, Cz, absent in previously seen tasks, ((-1) such that C. C C CC-((t-)) The QA is incrementally introduced to a Task,, which it learns in communication with the other agents (RAs) in the network. Note here that since we differentiated the data into the pretraining data (1.1) and the lifelong learning data. (1.2), we can fairly evaluate the RA's 's responses and the confidence--valuattin strategy that discerns the RA's 's confidence while responding. In evaluation, we follow the single-head evaluation setup (Chaudhry et al.) 2018a) where the QA does not know the lask it is introduced, so it has to choose between all possible classes for classification.
Our experimental sctup considers 20 agents- - QA and 19 RAs that are pre-trained on subsets of 5-10 classes each, achieving 75-85% accuracy on their specific lasks. This simulates scenarios with imperfect, non-teacher-like agents. We evaluate our solutions vision datasets CIFAR-100 and MinilmageNett
4.1 Confidence-
Benchmarks: First, we establish benchmarks for comparing confidencc--vnlnnation strategies.
(I) Baseline; Our baseline strategy for assessing agent confidence employs softmax probabilities defined as (P); = cP This transforms the output of the model vector p into a probability distribution.
This distribution provides insight into the agent's confidence level regarding its response. We include results for this in our Appendix (Figure 13), given its documented inferior performance relative to entropy-based measures (Yona et al. 2022), which constitute the primary focus of our comparative analysis and discussion. (2) Entropy-Based Measure: To enhance our confidence asscssment, we use Entropy, introduced by [Shannon| (9488) and widely effective and accepted metrie (Yona et al. 2022 Chen et al.) 2021) for uncertainly quantification. It quantifies 'chaos' in a probability distribution. Since Entropy 2 O we transform this entropy into a confidence score using negative exponential mapping, e-H(p), This transformation maps higher entropy (chaos) to lower confidence and vice versa. Note that this transformation clips the confidence score from 0 to 1.
Sharing Accuracy Total Responses
% Reduction Communication Overhead while TRUE Entropy
75
TRUE
Entropy
70
20
% Correctt RA respontes
65
23
60
Fower responses
33
20
40
15000
25000
35000
10000
20000
30000
40000
Total RA responses received the QA
Correct Responses received the QA
(a) TRUE vs Entropy: This plot displays the percentage of correct responses sent by RAs to the QA using TRUE and Entropy as confidence scores (high is better). TRUE outperforms Entropy consistently.
(b) A PEEPLL system using TRUE needs 5-20% fewer total RA responses (vs. Entropy) for the QA to receive the same number of correct responses. This contributes to a reduction in system communication overhead.
Figure 2: These plots correspond to evaluation on 100. See Figure 10 for MiniImageNet in Appendix.
TRUE Results: The RAs compare their confidence against a threshold, and if their confidence is greater than the threshold, they send their responses back to the QA. We measure Sharing Accuracy: the percentage
8

---

Published in Transactions on Machine Learning Research (01/2025)
of responses sent by RAs to the QAs that match the ground truth. For standardized analysis, in this section, we report numbers that refer to the thresholds for a 1:1 Sharing Ratio the number of responses the QA receives is close to the number of queries it sends. See Figure 2a for a comparison of Entropy and TRUE across thresholds. TRUE shows a 19.29% improvement in Sharing Accuracy (from 46.3% to 55.2%) on CIFAR- 100 and a 7.03% improvement. (from 68.4% to 73.2%) on MinilmageNet. over using Entropy. An ablation experiment demonstrates that incorporating the Semantic Distance (dzemantie) yiclds an 11.7% improvement (47% to 52%) and the Dispersion Distance (ddispersion) yields 16.75% improvement (46.5% to 54.3%) in Sharing Accuracy CIFAR-100. The efficacy of TRUE also demonstrates that it is possible to derive meaningful latent space in Variational Autoencoders without reconstruction.
Since the TRUE confidence score aligns more closely to response correctness than Entropy, the RAs are better equipped to transmit responses judiciously. As a result, it helps restrict communcationn To get 'x' correct responses at the QA, the RAs need send 5-20% fewer responses on CTFAR100 (Figure 2b) and 5-15% on MinilmageNet (Figure 10b in Appendix) while using TRUE vs Entropy. This critical analysis to assess the reduction in communication overhead.
We find that knowledge sharing among agents scales inversely with the number of agents in the environment (see Figure 7, Appendix). This trend is expected as the likelihood of randomly choosing the correct agent to listen to decreases with an increasing number of agents, making the task progressively harder. However, our results indicate that TRUE consistently outperforms Entropy across varying numbers of agents.
The computational complexity for an agent in PEEPLL, in terms of the scalable symbols Q total queries, N(,, out) the number of RAs QA contacts for query, and K total classes is O(Q x N(., out) K). Refer to Section E of the Appendix for derivation of the complexity.
While TRUE has demonstrated clear advantages over entropy-based methods for uncertainty quantification, we note that its reliance on a variational classifier limits its direct application to pre-trained models and purely discriminative architecturcs. This presents intoresting opportunitios to advance and expand the applicability of TRUE to such models by modifying the architecture and fine-tuning.
4.2 Selective Response Filter:
The QA accepts a subset of responses from all received RA responses. We measure the percentage of correct responses in the accepted subset of responses. We provide structured display of results in Table
Table Efficacy of Filters applied at the QA
Method
Sharing
% Improvement
Sharing Accuracy (%)
% Improvement (t)
Accuracy (%)
(t)
CIFAR- 100
MinilmageNet
TRUE
55.2
73.17
TRUE + Majority
66.2
19.92
83.56
14.02
TRUE + MCG
65.8
19.13
82.04
12.14
TRUE + ICF
82.8
50
91.22
24.67
TRUE + Majority + ICF
83.3
50.9
91.91
25.6
TRUE + REFINE
83.9
51.99
92.04
25.79
The QA employs our proposed filters to select a subset of responses received from all RAs. This table displays the percentage of accepted responses that match the ground truth (Sharing Accuracy). REFINE (ICF with MCG) is the most effective filter. These values correspond to a 1:1 Sharing Ratio (thresholds where the number of accepted responses by the QA is close to the number of queries sent out by the QA). For an equivalent analysis across thresholds, see Figure 3.
As seen in Figure 3b (and 11b in Appendix), the Most-Confidend Confident Group (MCG) approach exhibits enhanced effectiveness at lower thresholds (more RAs respond), suggesting that in scenarios where response criteria are
9

---

Published in Transactions on Machine Learning Research (01/2025)
less strict, prioritizing collective agent confidence yields better results than majority voting. The rationale is that when agents with low confidence are allowed to respond, they cannot be taken at face value, and a further focus on their confidence is required. As the threshold increases, the Majority- Voting method parallels and occasionally surpasses the MCG approach. This trend indicates that with higher confidence thresholds, agents become more reliable, allowing responses with the most agent endorsements to be selected with greater assurance of accuracy. Implementing Intelligent Comparative Filtering (ICF) with the MCG filter retains a consistent edge over Majority Voting with ICF.
ICF consistently outperforms non-ICF methods. ICF with MCG constitutes most effective filter, which we term 'REFINE' filter; see Table and Figure 3.
Effectiveness of filters applied at the QA
Effectiveness of filters applied at the QA
% Correct in Accepted RA Responses
80
% Correct in Accepted Responses
30
88
60
TRUE
TRUE
TRUE HCP
TRUE+ICF
Majority
Majority
Majority HIC
Majoriy+KCF
MCG
MCG
MCG+ICF (REFINE)
ICF (REFINE)
5000
10000
15000
20000
25000
30000
35000
40000
Total RA resporoes received at the QA
TRUE Threshold send responses to
(a)
(b)
Figure 3: The QA applies our proposed filters to select a subset of responses received from RAs. These plots display the percentage of correct responses in the accepted subset (filter efficacy) across different confidence thresholds that the RAs must meet before sending their responses to the QA. Both plots are equivalent: the left plot's x-axis (total responses the QA receives) is controlled by the confidence threshold parameter shown on the right plot's x-axis. In the left figure, we cut off at a threshold of a 1:2 ratio of Queries asked to Responses received for clarity. Our REFINE filter consistently outperforms all other filters. This plot corresponds to the evaluation on CIFAR-100 (for MinilmageNet, see Appendix Figure 11).
4.3 Lifelong Learning
For the sake of initial comparison, we designate one agent as the QA, while the remaining agents act as R.As. After the QA learns all Tasks1-r incrementally in communication with its imperfect peers, we measure its local performance on the complete test set. For standardized testing, we evaluate the QA's 's performance by choosing the communication strategy, which includes the thresholds for TRUE and filter choice, such that there is a 1:1 Sharing Ratio, that is, the nunber of peer responses the QA accepts and learns from is close to the number of queries it seeks assistance on.
We compare our solution to PEEPLL against a Single- that learns with complete supervision the QA learns new lasks using ground truth labels, same as the Experience Replay (ER) (Chaudhrr et al., 2019) strategy. This refers to 'ER', as seen in Table 2. ER has been consistently shown to outperform other LL strategies and is state-of-the-art (van de Ven et al. 2022). Matching ER necessitates advanced strategies in Seffidenne-Evvlluattion Selective Response Filter, and Lifelong Learning in PEEPLL, particularly those that judiciously utilize potentially incorrect responses.
As shown in Table 2, despite learning from noisy peer responses, our solutions to PEEPLL outperform ER, the Supervised Single-Agent, which relies on ground-truth responses (complete environmental supervision). by 2.1% on 100 and 2. % MiniImageNet.
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table The QA's local performance on the complete test set after lifelong learning tasks.
Communication Strategy
Accuracy
Quality
Memory
Accuracy
Quality MinilmageNet
Memory
(%)
(%)
Samples
Samples
CIFAR-100
Offiine Single- Agent (10 epochs)
WID
58.2 = 1.1
100
52.4 1.2
100
Naive (fine-tune)
4.9 + 0.2
100
1.6 + 0.2
100
EWC
3.9 + 0.3
100
3.1 + 0.2
100
GDumb
39.5 + 0.8
100
5k
47 + 1.3
100
5k
Offline PEEPLL (10 epochs)
PEEPLLAREFINE Naive
4.2 + 0.3
83.9
4.1 + 0.3
92.04
Online Single- Agent
36.7 + 0.9
100
34.5
Naive
2.4 + 0.1
100
2.2 + 0.2
100
GDumb
3.6 + 0.5
100
5k
9.2 - 0.7
100
5k
ER
29.2 = 0.5
100
5k
24.8 + 0.47
100
5k
Online PEEPLL
PEEPLL+REFING Naive
2.1 + 0.1
83.9
1.9 0.1
92.04
Online PEEPLL Confidence
Entropy
26.5 1 0.4
46.3
5k
22.35 + 0.3
68.4
5k
TRUE
29.7 + 0.15
55.2
4.5k
25.6 + 0.58
73.17
4.3k
Online PEEPLL w. Confidence + Filter
TRUE + Majority
30.8 = 0.57
66.2
4.7k
26.9 + 0.76
83.56
4.6k
TRUE MCG
28.9 + 0.39
65.8
4.9k
26.7 = 0.76
82.05
5k
TRUE + ICF
31.1 + 0.31
82.8
4.5k
26.9 + 0.92
91.22
1.3k
TRUE + Majority ICF
30.6 + 0.27
83.3
1.5k
25.6 + 0.49
91.91
4.7k
TRUE + REFINE
31.3 + 0.72
83.9
4.5k
26.1 + 1.10
92.04
4.7k
This (able presents the QA's local performance (Accuracy (%)) on the complete test set following lifelong learning of all tasks. Other metrics shown include Quality (%) and Memory Samples. Quality (%) refers to the percentage of correct responses the QA learns from, determined by the communication strategy's Sharing Mougle-Agents Accuracy. Single-A Agents learn from the environment with 100% Quality. PEEPLL agents may not fill their memory buffers they do not receive enough RA responses for certain classes, while Naive strategies do not Keplay/Memory, utilize Replay/ Memory. Note that although GDumb not specifically designed for Lifelong Learning, is used strong baseline for comparison. Our results demonstrate that our PEEPLL agent outperforms ER (the slate-of-the-art Online Single-Agent L.I) despite learning from noisy peer responses.
We find that the noisy responses from which the QA learns, fact, generate a beneficial regularization effect its learning process. To analyze this effect and the QA's ability to effectively learn from potentially incorrect peer responses,
monitor its performance lifelong learning journey (see Figures 5a and 8a in Appendix). Observe that. PEEPLL system using a communication strategy with lower Sharing Accuracy shows better QA performance on future tasks. This suggests that responses deemed 'incorrect" may still hold relevance for queries of another class with similar characteristics. This is also supported by Figure 6a, where the QA's initial confidence in each subsequent task progressively increases throughout
future yet-to-be-introduned tasks throughout its
learning journey. This is because the QA has previously, albeit briefly, learned about these classes from 'incorrect" responses. We hypothesize that briefly learning about future classes while learning similar classes beforehand produces a regularization effect that aids in more effective learning of those future classes when they become relevant. This is supported by the trends in Figure 5b (and 8b, Appendix), where initially, the Single-Agent (ER) and the QA learning from peer responses perform comparably. but the QA outperforms in later stages. This phenomenon is akin to and is further supported by the less challenging Domain- Incremental Learning (Domain-IL) scenario in lifelong learning, where an agent incrementally learns the same classes from different data distributions. Research has shown that LL agents find learning casier in Domain-IL than in Class-IL (van de Ven al.) 2022), explaining the effective learning of QA despite polentially incorrect responses from peers. However, note that the QA's improved performance
11

---

Published in Transactions on Machine Learning Research (01/2025)
future tasks comes at the expense of current task performance. This trade-off has implications for the LL algorithms and communication strategies that will be developed for PEEPLI agents.
Note that the communication strategy influences the subset of responses chosen for learning and the per- centage of correct responses within that subset, affecting the regularization effect and the QA's learning for current and future tasks. While we demonstrate that a regularization effect exists, future work will examine in detail the impact of our proposed communication strategies on the regularization effect and the QA's performance current and future tasks.
Another important observation from Figures 5a (and 8a. in Appendix) is that the QA performs better on future tasks within the PEEPLL system that uses TRUE as the communication strategy compared to Entropy, despite TRUE having higher Sharing Accuracy. This indicates that the TRUE score not only aligns more closely with response correctness but also more effectively captures query similarity, other words, a high TRUE score for an incorrect response suggests that the predicted class is likely similar to the query's actual class. This feature weaker with entropy; high entropy score for an incorrect response indicates that the predicted class is less likely, compared to TRUE, to be similar to the query's original class.
QA Gains Confidence and Reduces Communication After Sufficiently Learning Tasks
0.4
0.4
Change in Likelihood of Communication for Task
Correlation 0.81
0.3
0.3
0.2
0.2
Change Confidence at Task
0.1
0.1
0.0
0.0
-0.1
0.1
-0.2
-0.2
-0.3
-0.3
-0.4
-0.4
2
7
11
12
14
16
17
18
19
Task ID
Figure 4: As the QA learns each task incrementally (x-axis), gains confidence in answering queries relatod to that lask (green) and consequently reduces communication calls (maroon) to the RAs for assistance. A green bar indicates the change in the QA's confidence in answering queries from the start to the end of each task. A maroon bar shows the change in the number of times the QA initiates communication with RAs per query from the start to the end of each task. The correlation between the change in QA's TRUE confidence and the reduction in communication calls is 0.81, with a p-value of 2.98e-05. This plot corresponds to a PEEPLL agent learning MinilmageNet with TRUE: Table 2, Row 12, and performs 25.6% on the test set.
Figure 4 illustrates how the PEEPLL framework reduces system communication overhcad over time. As the QA learns a lask, its confidence in answering the task's 's queries increases. And since the QA only requests assistance when is underconfident in answering a query, the increase in confidence leads to fewer requests for assistance. In Figure 4, the change QA's confidence each task shown green, while the change in the likelihood of initiating communication call shown maroon. We find that the correlation between the change in QA's 's confidence and the change in communication likelihood is strong, 0.81, with
12

---

Published in Transactions on Machine Learning Research (01/2025)
p-value of 2.98e-05, meaning it is statistically significant. Figure 6 in the Supplementary Material shows the unprocessed data of growing confidence and a decrease in communication. This reduction in communication overhead helps make multi-agent systems practical.
Note that even after the QA completes its subsequent lifelong learning tasks, the TRUE score contimues exhibit expected trends with each new task initially, confidence is low, then increases over time (see Figures and 6). This demonstrates TRUE's context-aware adaptability the QA's continual learning capability, which is crucial for lifelong learning applications.
Our Dynamic Memory-Update mechanism improves QA's performance by 44.17% on CIFAR-100's test set (21.71% to 31.3% on CIFAR-100 with PEEPLL using TRUE + REFINE) and 26.8% on MinilmageNet's test set (21.21% to 26.9% on CIFAR-100 with PEEPLL using TRUE + Majority) compared to a static Medory-Update system that does not replace old, densiconf responses. This mechanism mitigates the adverse effects of previously reccived incorrect responses. This mechanism further supports the reality of multi-agent systems: if a QA requires assistance from a response agent that is skilled a certain task yet hard to reach (e.g., offline distant), the QA will now be able to receive the responses once that agent becomes available and replace previous low-quality responses in memory. This stabilizes learning and improves the reliability of the system.
QA's Local Performance on Future Tasks
QA'S Local Performance on Complete Test Set
25
TRUE
TRUE
TRUE+HCF
30
MCG+ICF
Majority
Entropy
20
MaJ
25
Single Agent
MCG+ICF
Accuracy (%)
Entropy
15
Single-Agent
Accuracy
10
10
5
16
200
400
600
800
800
ID/ Time
Task ID/ Time
(a) QA's 's performance on pet-to-be-introdue tasks.
(b) QA's performance on all tasks.
Figure 5: Each vertical blue line marks the introduction of new task to learn for the QA. In (a), observe that QA learning with communication strategy with lower sharing accuracy exhibits higher performance untrained tasks. In (b), observe that QA learning from peer responses performs comparably to the single- agent initially but outperforms in the later stages. This plot corresponds to the evaluation on CIFAR-100. See Figure in Appendix for MiniImageNet.
5 Conclusion
This paper introduces Peer Parallel Lifelong Learning (PEEPLL), the first distributed multi-agent lifelong learning framework. PEEPLL brings forth essential directions for realizing distributed lifelong learning, such as autonomously identifying novel tasks, navigating potentially incorrect responses from peers, and managing communication overhead. This paper lays the foundation for further research by setting initial benchmarks: TRUE for Confidence--valuattion
REFINE for Selective Filtering, and Dynamic Memory-Update for Lifelong Learning in PEEPLL. Our results demonstrate that PEEPLL agents can outperform traditional LL agents with complete supervision, even when learning from potentially incorrect peer responses. By reducing reliance on environmental supervision, PEEPLL marks step toward realizing seamless lifelong learning technologies at the odge. Furthermore, since PEEPLL's self-aware agents respond only when confident and sook peer assistance to learn only when underconfident, this minimizes risks during unprecedented events and reduces forgetting of previously learned knowledge caused by unnecessary learning while already confident. Thus, our
13

---

Published in Transactions on Machine Learning Research (01/2025)
proposed PEEPLL mechanisms significantly improve the safety and adaptability of LL systems in dynamic learning conditions. Future research will investigate the mixed dependence on environmental supervision and peer assistance and explore variations in agent quantity, expertise, and memory budgets. We will also explore communication strategics with contralized support to develop more advanced communication strategies. These investigations will further refine PEEPLL's capabilities, paving the way for more seamless, resilient, and adaptive Lifelong Learning systems. Despite its controlled scope, this study establishes a foundation for research in distributed multi-agent lifelong learning. We encourage the research community to work toward this critical direction in AI, which holds significant implications for autonomous and semi- autonomous systems.
Acknowledgments
We sincerely thank the Air Force Research Laboratory (Distributed Lifelong Learning) and the DARPA ShELL (Shared Experience Lifelong Learning) program for supporting this research through their funding. We are also grateful to several students from the BINDS lab who helped at different stages of this project: Devdhar Patel suggested the cross-thresoold TRUE evaluation incorporated in Figures 2a, 3, 10a, and 11. Adam Kohan contributed to the initial code, including the ER implementation, though the PEEPLL solution introduced in this paper diverged significantly from the preliminary efforts. Arjun Karuvally provided valu- able feedback on writing, and Joshua Russell offered helpful suggestions for improving visual representations.
References
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. CoRR, abs/1612.00410, 2016. URL http://arxiv. http:///arxiv.orr/aaa///11. 00410
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. CoRR, abs/ hts/1998.047422 2019a. URL http:://tttttii brg/abs/1909
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Online continual learning with no task boundaries. CoRR, abs/1903.08671, 2019b. URL http:://axxiv.rr/abbb//////
08671
Sara Babakniya, Zalan Fabian, and et al. Chaoyang He. A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks, 2023.
Bruno Brito, Michael Everett, Jonathan P. How, and Javier Alonso-Mora. Where to go next: Learning a subgoal recommendation policy for navigation in dynamic environments. IEEE Robotics and Automation Letters, 6:4616-4623, 2021.
Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. ArXiv, abs/ abs/1801.10112, 2018a.
Arslan Chaudhry, Mare' Aurelio Ranzato, and et al. Marcus Rohrbach. Efficient lifelong learning with A- 00420
GEM. CoRR, Abs/1181.000400, 2018b. URL
Arslan Chaudhry, Marcus Rohrbach, and et al. Mohamed Elhoseiny. Continual learning with tiny episodic 2019. URL http:///aaxxv..or/abb/////// 10486.
memorics. CoRR,
Zhi Chen, Zi Huang, Jingjing Li, and Zheng Zhang. Entropy-based uncertainty calibration for generalized zero-shot learning, 2021.
Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. CoRR, abs/1703.0655 2017. URL httt::/aaxxvv.orr
0be/1703.06585
Abhishek Das, Théophile Gervet, and et al. Joshna Romoff. Tarmac: Targeted multi-agent communication. CoRR, abs/1810.11187, 2018. URL http://awww..rr/aaa////11..1.1
14

---

Published in Transactions on Machine Learning Research (01/2025)
Shehzaad Dhuliawala, Mrinmaya Sachan, and Carl Allen. Variational classification, 2024. URL https:
7/arxiv. hrrg/abs/2305... 10406
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. CoRR, abs/1605.06011 2016. URL http://arxiv.org
aba/1605. 06676
Jayosh K. Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In AAMAS Workshops, 2017.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models domains and tasks. Tn Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, Online, July 2020. Association for Computational Linguistics. doi: 10. 20.18653/v1/202..ad acl- acl-mmin. main. 740. URL
org/2020. acl- maammmain..00 740
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Yedid Hoshen. VAIN: attentional multi-agent predictive modeling. CoRR, abs/ abs/1706.06122, 2017. URL http://w.rrr/bbb/7/00000022
Maximilian Hüttenrauch, Adrian Sošié, and Gerhard Neumann. Local communication protocols for learning complex swarm behaviors with deep reinforcement learning. In ANTS Conference, 2017.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. CoRR, 015//1805.07733, 2018. URL. http://arxiv.orr wwwpexiiv.oog/aaao wrgg/bs//100...... 07733
Benjamin Keel, Aaron Quyn, David Jayne, and Samuel D. Relton. Variational autoencoders for feature
exploration and malignancy prediction of lung lesions. In British Machine Vision Conference, 2023. URL
https://api.senann www.sti best 265456413
Dong-Ki Kim, Matthew Riemer, and et al. Miao Liu. Influencing long-term behavior multiagent rein-
forcement learning. ArXiv, 0tt//2203.03555 2022.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/ 1612.00796, 2016. URL
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Cecilia Lee and Aaron Y Lee. Clinical applications of continual learning machine learning. The Lancel Digital Health, 2(6))e279-e288 2020.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz Rodríguez. Continual learning for robotics. CoRR, abs/1907.00182, 2019. URL http:://axxxii org/abs///////////
1907.00182
Timothée Lesort, Andrei Stoian, and David Filliat. Regularization shortcomings for continual learning, 2021.
Warmter Stefan Logacjov Alcksoj, Kerzel Matthias. Learning then, learning now, and every second in between: Lifelong learning with sinulated humanoid robot. Fronliers in Newrorobotics, 2021.
Zheda Mai, Dongsub Shim, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Adversarial
shapley value experience replay for task-free continual learning. CoRR, abs/2009.00090 2020. URL
https://arxiv. orooorg//oo. hb////b///// wttpaabss//000//// 200.:200000000000
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. CoRR, abs/ 22101.11122,, 2021. URL https://arxxiv
org/ atsp//bb/2101...0 10423
15

---

Published in Transactions on Machine Learning Research (01/2025)
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions progress in continual learning. In Computer Vision-F.CCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, 524 540. Springer, 2020.
Vignesh Prasad, Dipanjan Das, and Brojeshwar Bhowmick. Variational clustering: Leveraging variational autocncoders for image clustering. CoRR, abs/2005.04613, 2020. URL sttp:://axxd.orr//aa///0000 http://w...rga///000......
04613
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Recyclable tuning for continual pre-training. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (ods.); Findings of the Association for Computational Linguisties: ACL 2023, pp. 11403- 11426, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/202. acl.ng URL https::/aclanthooloe htt.:///w...........o.o..nna-eee aiidinngs-ac..77
Sharan Raja, Golnaz Habibi, and Jonathan P. How. Communication-wware consensus-based decentralized
task allocation in communication constrained environments. IEEE Access, 10:19753-191 -97667, 2022. doi:
10.1109/ MAL1109/ACEESS ACCESSS202..311000 2021.313887..
Mohammad Rostami, Soheil Kolouri, Kyungnam Kim, and Eric Eaton. Multi-agent distributed lifelong
learning for collective knowledge acquisition. ColR,, abs/ abs/1709.05412, 2017. URL htttp///axxiv.org
abs/1709.0541
Paul Ruvolo and Eric Eaton. Ella: An efficient lifelong learning algorithm. In International conference on machine learning, pp. 505-515. PMIRR, 2013.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379 403,423, 1948. doi: 10.1002/j. 1011115387775.1.1 1bb.cc1.........
Hava Siegelmann. Neural Networks and Analog Compulation: Beyond the Turing Limit. Springer Science & Business Media, 1998.
Hava T Siegelmann. Computation beyond the turing limit. Science, 268(5210):545-545 1995.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Andrea Soltoggio, Eseoghene Ben-Twhiwhu, Vladimir Braverman, Eric Eaton, Benjamin Epstein, Yunhao Ge, Lucy Halperin, Jonathan How, Laurent Itti, Michael A Jacobs, al. A collective ai via lifelong learning and sharing at the edge. Nature Machine Intelligence, 6(3):251 2221-264,,21 2024.
Sebastian Thrun. Lifelong learning algorithms. In Learning learn, pp. 181-209. Springer, 1998.
Scbastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2); 25-46, 1995.
Gido M. van de Ven, Tinne Tnytelaars, and Andreas S. Tolias. Three types of incremental learning. Nature URL
Machine Intelligence, 1(12):1185-1197 2022. ISSN 2522-5839. doi: 0001111//22202222220000200 org https://Aoo.oogg///111118//2226
Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S.
Rellermeyer. A survey on distributed machine learning. CoRR, abs/1912.09789,, 2019. URL http:
//arxiv.org. abs/1912.
Kerstin N Vokinger, Stefan Feuerriegel, and Aaron Kesselheim. Continual learning in medical devices:
Fda's action plan and beyond. The Lancet Digital Health, 363):3337-3335 2021.
Gal Yona, Amir Feder, and Itay Laish. Useful confidence measures: Beyond the max score, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A
Appendix
QA Confidence Over Answeriigg Quecies Each Taak
Likelthood OA Initisting
Sufficiently Lieehood OA inbsttig communccation
Canfienno
0.6
Uielloos of OA Imbbitinn Communies commmmnccttionn
m
0R Confrdence
0.3
100 200
350
400
2.00
400
Task IDViime
Task ID/Time
Figure 6: QA Gains Confidence & Reduces Communication: Each vertical blue line marks the introduction of new task in the QA's lifelong learning journey on MinilmageNet. (a) Initially, the QA exhibits low confidence in queries related to the new task, but as it learns through communication with RAs, its confidence increases. (b) This increased confidence leads to self-reliance, roducing QA's number of calls to the network for responses and diminishing the system's 's communication overhead.
B Methodology
Here, we briefly discuss how we conduct experimental evaluations of our solutions to the different elements of the PEEPLL framework.
B.1 Confidence- Evaluation
Ionliuence-Evaluatiion Strategies:
Selup: During lifelong learning, the QA is introduced to queries from 1.3. The QA queries the RAs for inputs when their confidence is low. The RAs (pre-trained on 1.2) answer and evaluate their confidence the received queries. Responses with confidence higher than a certain confidence threshold are sent back to the QA.
Experimental Evaluation:
1. We measure the proportion of responses that the RAs sent back to the QAs that matched the ground truth at various confidence thresholds (see Figure 2a). 2. We assess the reduction in total data sent from the RAs to the QAs to achieve the same number of correct responses compared to the entropy-based approach (see Figures 2b and 8). This metric vital for reducing communication overhead in multi-agent settings.
3. We track the QA's growing confidence within specific tasks (Figure 4a). This helps us diminish communi- cation needs the QA becomes more adept at new tasks (Figure 4b). These demonstrations will showcase the confidence metric's adaptability to the QA's continual learning capability lifelong learning settings.
essential property in
4. We also conduct specific analysis at 1:1 sharing ratio, where the number of responses received by the QA from the RAs equals the total number of queries processed (corresponding to the 1.3 data segment). This controlled setting provides a standardized basis for comparison under balanced data exchange conditions.
B.2 Selective Response Filter:
Selup: Upon receipt of responses from RAs, the QA implements an additional selective response filter. This filtering mechanism leverages the broader response pool from multiple RAs, thereby facilitating more informed decision-making.
Experimental Evaluation: The efficacy of the Selective Response Filter is measured by its ability to increase the proportion of the
17

---

Published in Transactions on Machine Learning Research (01/2025)
Sharing Accuracy vs Total Responssss
Sharing Accuracy vs Total RA Responses
TRUE
TRUE
0.70
croupy
0.73
0.85
Corfect respetes
Conoect responoes
0.70
0.50
0.30
0.45
0.50
5000
10000
15000
20000
25000
30000
35003
40000
5000
23000
15000
20000
25000
30000
33000
40000
Total RA resporses recelved at the QA
Total RA responses received et the GA
(a) 10 Agents
(b) 20 Agents
Sharing Accuracy vs Total RA Responses
Sharing Accuracy vs Total RA Responses
0.50
TAUS
ertrpy
0.070
% Correct 04 respontes
Correct (A resporees
8.000
0.40
0.055
0.000
0.30
0045
5000
30000
35000
20000
25000
30000
35000 40000
5000
10000
15000
20000
25000
30000
35000
40000
responses recelived QA
Total RA responses received at the QA
(c) 40 Agents
(d) 100 Agents
Figure 7: Sharing Accuracy Scaling with Number of Agents. As the mumber of agents increases, expoctedly discerning the correct agent to listen to becomes increasingly challenging. Baseline probabilities for randomly selecting the correct agent are as follows: 0.1 for 10 agents, 0.05 for 20 agents, 0.025 for 40 agents, and 0.001 for 100 agents. For fair comparisons against these baselines, refer to when 20k RA responses are received at the QA, as the experiment considers 20k queries. While our current performance indicates progress, there remains significant room for improvement in developing more effective communication strategies for higher number of agents. Notably, TRUE consistently outperforms Entropy, though their performance becomes nearly comparable with 100 agents.
correct responses in the accepted set (Figures 3 and 10, Table 1). For instance, if the filter receives 10,000 samples with 3,000 correct responses (proportion: 30%) and subsequently filters out 5,000 samples, retaining 2,500 correct responses (proportion: 50%), this indicates an increase in the proportion of correct responses (by 20%). Such analysis underscores the filter's capacity to effectively identify and preserve the most relevant and accurate responses. Morcover, conduct specific 1:1 Sharing Analyses for these processes well.
B.3 Lifelong Learning:
Sctup: After receiving responses from RAs, the QA further employs selective filter to accept only the most pertinent answers. The QA then learns from this accepted set of responses.
Experimental Evaluation:
1. QA's Local Performance on the Complete Test: We assess the QA's final test performance when learning responses received using different commiiccation strategies (Table 2 and Figure 6).
2. QA's Local Performance on Untrained Tasks: The QA's effectiveness on tasks it is not explicitly trained
evaluated to understand the regularization effect, as discussed in Section 4.3, and the quality of 'bad'
18

---

Published in Transactions on Machine Learning Research (01/2025)
QA'S Local Performance on Future Tasks
QA'S Local Performance on Complete Test Set
14
TRUE+ICF
Majority
Majority
25
Entropy
Majority4ICS ICF
Single-Agent
10
MCG+ICF
Entropy
Accuracy
Single Agent
Accuracy
15
600
200
600
800
Task ID Time
Task ID/ Time
(a) QA's performance on yel-toobee-ntroduced tasks.
(b) QA's 's performance on all tasks.
Figure 8: Each vertical blue line marks the introduction of a new task to learn for the QA. In (a), observe that QA learning with communication strategy with lower sharing accuracy exhibits higher performance on untrained tasks. In (b), observe that QA learning from peer responses performs comparably to the single- agent initially but outperforms in the later stages. This plot corresponds to the evaluation Minilmage
Algorithm The PEEPLL Algorithm
Input to QA: Sample x, Maximum confidence Crux; Confidence threshold Cthreshold
At QA: Get QA's label on z, 12A
At QA: Get QAs confidence
2r, cQA
At QA: If Q < Cmax, send (x, <Q4) to RAs
5: At RA: Get RA's label on RA
6: At RA: Get RA confidence on 2; RAA > QA send RRA to QA
7: At RA: If c > Cthreshold and c RA
8: At QA: Collect all responses by RAs
At QA: Selectively filter which responses accept
10: At QA: Update Memory with accepted responses
11: At QA: Learn Accepted Responses + Replay memory
responses (Figure 5).
C Implementation Details
We use VGG16 (Simonyan & Zisserman 2015)
the backbone for our agents under PFEPLL We deviate from the current lifelong learning research by doing so that currently uses ResNet18 (He et al. 2015). Our methodology utilizes VGG16 due to its lack of skip connections, enabling the isolation our lifelong learning strategies' impact. ResNet skip connections, known to mitigate vanishing gradients, could introduce confounding factors.
PEEPLL models employ VGG16 as the encoder with small MLP decoder for task-specific outputs. The total parameters of our PEEPLL model were 15417124.
We use the same Optimizer hyperparameters for pretraining and lifelong learning, it would not be practical to tune those parameters further for online learning.
We implemented conventional singlo-agent LIL strategics using Avalanche. We used maximum memory as 5k, the batch size of memory 5k, and the ClassBalancedBBfferr style of memory with no adaptive size. This is the same for implementations of our PEEPLL strategies.
19

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 2 Evaluating TRUE Confidence
1: Input to Agent 'A': Sample x
2: Get latent representations of z, Amean: Fugvar
3: Prediction on x; Pr
4: Get label on z; 12 = argmax(pa)
Get memory samples with label 4,, (z.)
6: Get latent representation of retrieved memory samples, Memory Amayy (m..))
and Memory (x.1z)
Slogyar
dsemantic mean Mean Mean(Memory(nag) Bmae)))
8: ddispersion = 1 Pogvar
(w.le) lll2
entropy E; Po; logg(Ps; +e)
10: Transform into Confidence Csemantie Ascmantic
11: Transform into Confidence Calspersion datuparston
12: Transform into Confidence Ceutropy = e dantropy
13: Normalize scores
14: TRUE = (Ceemantic + Cdispersion + Centropy)/3
15: return TRUE
D Additional Results
Results for Comparison of our TRUE Confidence with Max-of-Softtan are illustrated in Figure 13.
Responses
0.75
Sharing Accuracy vs Total RA Responses
Sharing Accurscy: Correct Rerporses/ Resgonses
Sharing Accuracy vs Total RA Responses
Max of Softmax
IRUE
0.825
Hax- Sortmax
TRUE
0.70
0.800
0.65
Sharing Acouracy Resporrses
0.775
0.00
0.750
0.725
0.700
0.600
10000
35000
40000
3000
10000
25000
20000
25000
30000
35000
40000
Responses
Total RA Responsess
CIFAR100 results
(b) MinilmageNet results
confidence. The plot shows the proportions of the total responses (from all RAs) that match the ground truth (higher is better).
Figure 9: The TRUE score highly outperforms
20

---

Published in Transactions on Machine Learning Research (01/2025)
Sharing Accuracy Total Responses
% Reduction Communication Overhead while using TRUE vs Entropy
TRUE Entropy
90
% fewer RA responses sent to the QA
25
% Correct RA resporces
85
30
70
10000
20000
25000
30000
35000
10000
30000
RA responses
Correct Responsess received at the OA
(a) TRUE vs Entropy: This plot displays the percentage of correct responses in the responses sent by RAs to the QA using TRUE and Entropy confidence scores (high is better). TRUE outperforms Entropy consistently.
(b) A PEEPLL system using TRUE needs 5-20% fewer total RA responses (vs. Entropy) for the QA to receive the same number of correct responses. This contributes to a reduction in system communicttion overhead.
Figure 10: These plots correspond to evaluation on MinilmageNett.
Effectiveness of filters applied the QA
Comparison of Communication Protocols
% Correct in Accepted RA Resporses
% Correct in Accepted RA Responses
80
80
80
80
90
TRUE
TRUE HCC
TRUE+ HCF
Majority
Majority
70
MajorityICC1
MCG
Majorityiic
MCG+iCF (REFINE)
TRUE + REFINE
5000
10000
Total RA responses received at the QA
15000
20000
25000
30000
35000
40000
0.0
0.2
Total RA Responses Received at the QA
0.4
0.6
0.8
Figure 11: MinilmageNet: Each filter seleetively accepts a subset of the responses received at the QA from the responses dispatched by the RAs utilizing TRUE. The figure depicts the ratio of correct responses within the selected subset. In the left figure, we cut off at threshold of 1:2 ratio of Queries asked to Responses received for clarity.
E Computation Complexity & Scalability
E.1
Symbols
M: Memory size, K: Total number of classes in the system, K(i, t): Number of classes in agent i at time 4, N(g, out): Number of agents queried for query q, N(q, back): Number of agents responding back for query q, N: Number of agents, M(i, k, t): Number of samples in memory of class k at time t for agent i, Dimension of latent vector, C: Cost of communication, Fy: Forward pass of the entire network (Encoder + Decoder), Fg: Forward pass of the Encoder, By: Backward pass of the entire network (Encoder + Decoder).
21

---

Published in Transactions on Machine Learning Research (01/2025)
C
Mean
Normalize
Normalize
Transform ex
Transform e*
Transform e
Entropy
Co00a
E
M
Encoder
Decoder
Query-
Query -2
Classification
Query (Q)
Mear
Expanencal Mian
logyar
Pxpawetal
VAE- Classifier
Predicned Clas Lates Code Meass (P- 1P-Mem-z z nan'
Predicted Clan
Prodicted Class Memory (P-Mem)
Laten Code Mouus
(P-Mem-z logar
Figure 12: TRUE: Confidence Evaluation mechanism for agent. The circle with E denotes the operation that returns the Euclidean distance wetween two vectors, and M the Manhattan Distance. 'Transform e --=; takes in value and maps it using the e function.
Note N(g, back) S N(q, out) < N. The number of agents chosen for querying is an important research question. This selection should not be based on their previous interactions. Instead, it could be determined by factors such proximity or other metadata.
E.2
Computational Complexity Analysis
E.3
Query Agent Complexity
Confidence Calculation
Get the prediction class k: OFFx)
Find memory samples belonging to le O(1)
Evaluate latent codes for those memory somples: O(Fg M(i,k,t)
Calculate Dispersion and Semanlic Distance: Old x M(i, k)))
Get the mean of Dispersion and Semantic Distance: 0(1)
Transform Mean Dispersion and Semantic Distance: O(1)
Get Entropy of Prediction: O(d)
Transform Entropy: O(1)
22

---

Published in Transactions on Machine Learning Research (01/2025)
Average three scores: O(1)
Final Complexity: O(Fx + (Fr + d) x M(i,k,))
Seeking Assistance
Check confidence againsl Uhreshold: O(1)
Send query and confidence other agent: O(N oul) x C)
Final Complexity: O(N(q, out) C)
Selective Filtering
Put agents into groups by their predicted class: O (g. back))
Calculate each group's 's confidence (MCG); O(1) (keep track while sorting)
Pick group with highest confidence (MCG): O(1) (keep track while sorting)
Pick group with highest members (Majorily): O(1) (keep track while sorting)
Reject conflicting responses (ICF): (g back) x K R or O(N back) x N(q, back))**
Final Complexity: Minimum(O([((q, back) K), (Q, back) x (g, back)) when ICF used.
Elsc, O(N).
*For query, make a dictionary with keys as classes and values as the prodictions of the agents who know that class. Maintain this dictionary as you receive the answers O(N back) K). Now, go through each response, and if the predicted class of that response has any value the dictionary that is not the same as the key (predicted class), disregard that response O(N(q, back)). Hence, the complexity would be (N, back) x K).
**Go through each response and then check each agent see if they know the response's predicted class, and if they do, compare their predictions with the prediction of the response. If they do not match, reject the response. The complexity is ( back) x Σ, k(i,t)) N N(g, back) x K), but if known classes in agents are indexed by the class id in a dictionary, we have (q, back) x E, 1) and the algorithm becomes (NN(g back) x N (g. back)).
O(N back)
Learning Update
Backward pass: O(BN)
Final QA Complexity:
Minimum of the following
O(FN (Fg + x M(ikk,t) + out, C. N(9, back) x K + Br)
O(Fx (Fg + d) x M(i, k,t) N(q, C. + N(g, back) N(q, back) Bx)
E.4 Response Agent Complexity
Confidence Evaluation: O(Fx + (Fg + d) x M(ikktt)
Sending Response Back: OCC)
Final RA complexity: O(Fx (FF + x M(i,k,t) +C)
23

---

Published in Transactions on Machine Learning Research (01/2025)
E.5 Scalability Analysis
Total Complexity for Agents to exist in PEEPLL For Q queries: For agent i, of the Q queries, Q1 (i) is the number of times agent i serves as QA, and Q2 (i) is the number of times it serves as R.A. Note that Q1(1) + Q2(1) Σ Q. Therefore, the total complexity is
(1))) +Fx+ (,, out) xC+N((+ x K+ Bx) (0
Generalizing the above by considering the three terms that scale, Q, N, and K, we have complexity of O(Qx((,, out)xK), where N(., out) is the number of agents that it reaches out to for any query.
F
Design Decisions
Confidence scores derived from Semantic and Dispersion distances alone outperform Entropy by 21.2% on CIFAR100 and 6.5% on MinilmageNee. This is comparable to TRUE (Semantic + Dispersion + Entropy) vs Entropy. Entropy provides an additional stream of information that may be beneficial for some applications.
1. Entropy in TRUE:
2. Semantic and Dispersion Distances: Dhuliawala et al. (2024): Prasad et al. (2020): ; Alemi et al.) (2016); Keel al. (2023) show that the position of data in the latent space holds significance to the class it belongs to. To further support this, plot the latent space and demonstrate the formation of clusters according lo classes, also recently shown by Prasad et al. (2020); see Figure 13. In TRUE, measure the divergence in the mean and variance that define datapoint's position in the latent space. Intuitively, TRUE representative of how close data point would be to the cluster predicted by the model in the latent space. Their inclusion is supported by ablation studics (Section 4.1, TRUE Results, para 1).
TIME Latert
TINE Latert Space
-60
15
(a) Lighter dots represent the training data, and '4' denotes the test data. The colors of the dots and '4' represent its true labels. The agent has an average TRUE confidence of 0.75 for predicting the '4' data.
(b) 'x' represents data of class 5, which the model is not trained on. The color 'x' indicates the model's prediction. The agent has an average TRUE confi- dence of 0.34 for predicting the 'x' data.
Figure 13: An agent's t-SNE visualization of its learned latent distribution of classes it trained on CIFAR100. The agent trained on classes displayed in Figure Observe that the 'x' data is generally toward the edges of the clusters, and the 'x' latent vectors not clustered themselves.
24


Reviews:
[{"review_id": "u94zmAqSu7", "reviewer": "Reviewer_HNSr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strengths]**  \nThe paper is well-written and easy to follow. The proposed lifelong learning framework efficiently utilizes other agents without relying on human annotations. The TRUE score effectively improves lifelong learning performance compared to simple entropy-based confidence evaluation.\n\n**[Weaknesses]**  \nWhile the authors claim that the principles of PEEPLL are applicable across various domains in multi-agent lifelong learning, it is not clear how this would work in areas such as large language models (LLMs). Since all experiments are conducted on supervised image classification tasks (CIFAR-100 and MiniImageNet), the scalability of the framework to other domains is not sufficiently demonstrated. The applications of lifelong learning framework need to be clearly explained.\n\nA key limitation of the TRUE score is the reliance on a specific architecture (i.e., variational autoencoder). Given that many advanced perception tasks use discriminative architectures, adding a probabilistic regularization term might harm performance. Moreover, the TRUE score cannot be applied directly to pre-trained foundation models since it requires training with a probabilistic regularization term, which is typically not used during the training of widely used foundation models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a framework called Peer Parallel Lifelong Learning (PEEPLL), where agents continuously learn online by seeking assistance from other confident agents, rather than relying on human annotators. PEEPLL consists of three key components: 1) confidence evaluation, 2) selective filtering, and 3) lifelong learning. Notably, the confidence evaluation is based on the newly introduced TRUE score, proposed by the authors. The TRUE score leverages a variational autoencoder to enhance the confidence assessment of other agents. The proposed framework demonstrates significant performance improvements on CIFAR-100 and MiniImageNet, and the ablation study clearly highlights the effectiveness of each component.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**  \nSince the primary goal of lifelong learning is to enable models, such as LLMs, to acquire new information that arises over time, illustrating the application of this broad framework could significantly strengthen the paper. Additionally, while the TRUE score shows substantial improvements over evaluation techniques using the entropy, it is highly dependent on the model architecture. Therefore, the authors should discuss these limitations in the paper.", "broader_impact_concerns": "No significant ethical concerns are identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oAdOKVeF8r", "reviewer": "Reviewer_rvfH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed system uses many novel components and each of these components is carefully evaluated. It's clear the authors put a lot of thought into the design of the system, and this comes through in the presentation of the paper. \n\nThe paper is well-written overall, and all components of the algorithm are well motivated. The experimental setup is carefully described with relevant details. Plots are nice with error bars and good color choices.\n\n**Weaknessess**\n\nIn my view, the main weakness of the paper is the lack of effective baselines. The main baselines used by the authors are a single agent, and different versions of the proposed framework. First of all, the single agent underperforms the proposed method due to lack of regularization in the single agent. I would encourage the authors to regularize the single agent until it outperforms the proposed method to have a fair comparison. \n\nMore importantly, there are no other continual learning/lifelong learning/multiagent methods against which the method is compared. I understand that the authors may be designing an algorithm for a setting that is not considered in prior literature; nevertheless, it is important to include results from additional methods even if they are not applicable to the setting. For instance, how would a single agent with experience replay perform? How about multiple agents with a shared experience replay pool? It's ok if these methods outperform the proposed PEEPLL system, but it's useful to get a sense of how well the system performs in comparison to stronger baselines.\n\nAlso, the proposed system is fairly complex. The authors test many components of their system in isolation, but given the large number of design choices, it may be infeasible to empirically evaluate each one. Unfortunately, without empirical isolation, these design choices can appear as arbitrary heuristics rather than the careful choices they are. I encourage the authors to include a comprehensive design appendix that details the motivation and pros/cons of each of the design choices made by the authors (for example, why is the form of dispersion disparity in terms of exp(log var) instead of log var directly, why is entropy chosen as a component of TRUE, etc.).\n\nThe authors state that they use a VAE in the confidence evaluation strategy. However, the proposed method doesn't use a reconstruction loss, so it's by definition not an autoencoder. It might be more accurately described as a classifier with a particularly constrained latent representation. It's not at all clear to me how the authors would have considered this to be a VAE.\n\nThe ICF description in section 3.2 could be expanded upon a little bit more.\n\nThere are some formatting issues in the paper as well: in certain places, parenthetical citations should be used instead of in-line citations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new system to perform multi-agent lifelong learning. Roughly speaking, each agent makes predictions on inputs, but also receives inputs from other agents to refine their predictions. The system consists of three components: 1) agents evaluate confidence of predictions, 2) agents filter responses from other agents, 3) agents use a memory bank to enable continual learning. The paper evaluates each of these components in isolation and together and demonstrate that the components and the system overall outperform baselines on image classification tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical for acceptance**\n\nCompare with additional baselines in continual learning and multiagent learning\nPlease fix citations (change /cite to /citep when appropriate)\n\n**Would strengthen**\n\nInclude a design appendix detailing each design decision made in the paper", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n3LTbAqLs7", "reviewer": "Reviewer_sb6p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The authors propose a multi-agent lifelong learning framework called Peer, Parallel Lifelong Learning, which allows agents to improve learning performance without environmental supervision. \n\n- The authors establish some benchmark components of the proposed framework and demonstrate that it can outperform environmental supervision. \n\nWeaknesses\n\n- The proposed framework intuitively generalizes lifelong learning to multiple agents. Although the framework appears to be a new concept, specific technical developments are not entirely clear. \n\n- The computational complexity and the scalability of the proposed method are not characterized. \n\n- The proposed method lacks a theoretical foundation, leaving performance questionable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies lifelong learning in a multi-agent environment. The authors propose a distributed lifelong learning framework in which agents can continually learn online with other agents via query-reponse mechanism. They show that the communication among agents scales well in terms of computational efficiency. In experiments, the authors demonstrate the resilience of communication and the efficiency of memory usage.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See them in Weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, I think this paper focuses on the experimental results of a new framework and provides enough evidence to support the effectiveness of the design. The authors provide the source code for the experiments in this paper. 

In the minor revision, I suggest the authors to fix the typos in this paper and improve the readability and comprehensiveness of the source code.

---

All reviewers acknowledge that the design of this framework is novel, and the paper is well-written. 

Reviewer sb6p questioned about 'specific technical developments' and 'computational complexity and the scalability' which are answered by the authors in the paper. Reviewer sb6p also cares about the theoretical foundation of this paper which is still a weakness of this paper.

Reviewer rvfH questioned about 'effective baselines' and 'evaluation of each component in this complex system', which are answered by the authors in the paper.

Reviewer HNSr questioned about the application of this framework in other domains such as large language models. This is still a weakness of this paper (based on the current experiments) although the authors have updated the manuscript to explain the applicability.
Decision: Accept (poster)