Lightweight Learner for Shared Knowledge Lifelong Learning
Authors: Yunhao Ge
Yuecheng Li
Di Wu
Ao Xu
Adam M. Jones
Amanda Sofie Rios
Iordanis Fostiropoulos
shixian wen
Po-Hsuan Huang
Zachary William Murdock
Gozde Sahin
Shuo Ni
Kiran Lekkala
Sumedh Anand Sontakke
Laurent Itti
Venue: Accepted by TMLR ()

Abstract: In Lifelong Learning (LL), agents continually learn as they encounter new conditions and tasks. Most current LL is limited to a single agent that learns tasks sequentially. Dedicated LL machinery is then deployed to mitigate the forgetting of old tasks as new tasks are learned. This is inherently slow. We propose a new Shared Knowledge Lifelong Learning (SKILL) challenge, which deploys a decentralized population of LL agents that each sequentially learn different tasks, with all agents operating independently and in parallel. After learning their respective tasks, agents share and consolidate their knowledge over a decentralized communication network, so that, in the end, all agents can master all tasks. We present one solution to SKILL which uses Lightweight Lifelong Learning (LLL) agents, where the goal is to facilitate efficient sharing by minimizing the fraction of the agent that is specialized for any given task. Each LLL agent thus consists of a common task-agnostic immutable part, where most parameters are, and individual task-specific modules that contain fewer parameters but are adapted to each task. Agents share their task-specific modules, plus summary information ("task anchors") representing their tasks in the common task-agnostic latent space of all agents. Receiving agents register each received task-specific module using the corresponding anchor. Thus, every agent improves its ability to solve new tasks each time new task-specific modules and anchors are received. If all agents can communicate with all others, eventually all agents become identical and can solve all tasks. On a new, very challenging SKILL-102 dataset with 102 image classification tasks (5,033 classes in total, 2,041,225 training, 243,464 validation, and 243,464 test images), we achieve much higher (and SOTA) accuracy over 8 LL baselines, while also achieving near perfect parallelization. Code and data can be found at https://github.com/gyhandy/Shared-Knowledge-Lifelong-Learning

Published in Transactions on Machine Learning Research (05/2023)
Lightweight Learner for Shared Knowledge Lifelong Learning
Yunhao Ge
yunhaoge Guscc edu liyueche dlusc. edu dam9 29830ussc edu aoru Busc. edu adammjj @usc. edu amanda. rios Gintel. com fostirop Trusc. edu sa. wen wenwennsia Dsall uc. on pohsuanh Gusc. edu zmurdock Gusc. edu gsahin Buscc edu shuoni Trusc. edu klekkala Cusc. edu ssontakk Crusc. eduu itti edu
Yuecheng Li
Di Wu
Ao Xu
Adam M. Jones?
Amanda Sofie Rios"
Tordanis Fostiropoulos
Shixian Wen"
Po-Hsuan Huang
Zachary William Murdock2
Gozde Sahin
Shuo Ni
Kiran Lekkala
Sumedh Anand Sontakke!
Laurent Itt
Thomas Lord Department of Computer Science, University of Southern California Neuroscience Graduate Program, University of Southern California Intel Labs
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
Dornsife Department of Psychology, University of Southern California
Equal contribution as second author
Reviewed on OpenReview: https: // openreview. net/ forum? id= id=
Abstract
In Lifelong Learning (LL), agents continually learn as they encounter new conditions and lasks. Most current LL is limited to a single agent that learns tasks sequentially. Dedi- cated LL machinery is then deployed to mitigate the forgetting of old tasks as new tasks are learned. This is inherently slow. We propose a new Shared Knowlodge Lifelong Learning (SKILL) challenge, which deploys a recentralized population of LL agents that each sequen- tially learn different tasks, with all agents operating independently and in parallel. After learning their respective tasks, agents share and consolidate their knowledge over a decentral- ized communication network, so that, in the end, all agents can master all tasks. We present one solution to SKILL which uses Lightweight. Lifelong Learning (L.II.) agents, where the goal is to facilitate efficient sharing by minimizing the fraction of the agent that specialized for any given task. Each LLL agent thus consists of common lask-agnostic inmutable part, where most parameters are, and individual task-specific modules that contain fewer param- eters but are adapted to each task. Agents share their task-specific modules, plus summary information ("task anchors") representing their tasks in the commnn task-agnostic latent space of all agents. Receiving agents register each received task-specific module using the cor- responding anchor. Thus, every agent improves its ability to solve new tasks each time new task-specific modules and anchors are received. all agents can communicato with all others, eventually agents become identical and can solve all tasks. On a new, very challenging SKILL-102 dataset with 102 image classification tasks (5,033 classes in total, 2,041,223 225 training, 243,464 validation, and 243,464 test images), we achieve much higher (and SOTA) accuracy over 8 LL baselines, while also achieving near perfect parallelization Code and data. can be found at https: : //g1 thub. com/ gyhandy/ SeainddySShareee ShareddKnowledgeee Kwwwwedowwwwweeeeeeeeeeeeee www.eeeddee-iieeeee----eeerreeee

---

Published in Transactions on Machine Learning Research (05/2023)
1 Introduction
Lifelong Learning (LL) is a relatively new area of machine learning (ML) research, in which agents continually learn as they encounter new lasks, acquiring novel lask knowledge while avoiding forgetting of previous tasks (Parisi et al. 2019 This differs from standard then--epe-pply ML, which cannot incrementally learn without catastrophic interference across successive tasks (French) 1999).
Most current LL research assumes a single agent that sequentially learns from its own actions and sur- roundings, which, by design, is not parallelizable over time and/or physical locations. In the real world, tasks may happen in different places; for instance, we may need agents that can operate in deserts, forests, and snow, as well as recognize birds in the sky and fish in the deep ocean. The possibility of parallel task learning and sharing among multiple agents to speed up lifelong learning has traditionally been overlooked. To solve the above challenges, we propose a new Lifelong Learning challenge scenario, Shared Knowledge Lifelong Learning (SKILL): A population of originally identical LL agents is deployed to a number of distinct physical locations. Each agent learns sequence of tasks in its location. Agents share knowledge over a decentralized network, so that, in the end, all agents can master all tasks. SKILL promises the following benefits: speedup of learning through parallelization; ability to simultaneously learn from distinct locations; resilience to failures as no central server is used; possible synergies among agents, whereby what is learned by one agent may facilitate future learning by other agents.
Application scenarios for SKILL include: 1) Users each take pictures of landmark places and buildings in their own city, then provide annotations for those. After learning and sharing, all users can identify all landmarks while traveling to any city. This could also apply to recognizing products in stores or markets in various countries, or foods at restaurants worldwide. Thus, even though each teacher only learns at one or a few locations (or lasks), eventually all users may be interested in knowledge from all locations, as it will be useful during travel. 2) Agents in remote outposts worldwide with limited connectivity are taught to rocognize symptoms of new emerging discases, then share their knowledge to allow any agent to quickly identify all diseases. 3) Explorers are dispatched to various remote locations and each learns about plant or animal species they encounter, then later shares with other agents who may encounter similar or different species. 4) Each time a criminal of some sorts is apprehended (e.g., shoplifter, insurgent, spy, robber, sex offender, etc), the local authorities take several hundred pictures to learn to identify that person. Then all local authorities share their knowledge so that any criminal can later be identified anywhere.
However, to solve SKILL, one must address the following challenges:
Chal-1 Distributed, decentralized learning of multiple tasks. A solution to SKILL should support a population of agents deployed over several physical locations and each learning one or more sequential lasks. For resilience reasons, the population should not rely on a single central server.
Chal-2 Lifelong learning ability: Each agent must be capable of lifelong learning, i.., learning a sequence of tasks with minimal interference and access to previous data as each new task is lcarned.
Chal-3 Shareable knowledge representation: The knowledge representation should easily be shared and understood among agents. Agents must be able to consolidate knowledge from other agents in a decentralized, distributed fashion.
Chal-4 Speedup through parallelization: Shared knowledge should be sufficiently compact, so that the benefits from using multiple parallel agents are not erased by communications costs. Adding more agents should result in greater speedup compared to single agent. We measure speedup as the the ratio of time it takes for one agent to learn all tasks compared to N agents (larger is better). As a goal for our work, we strive for speedup of at least 0.5 x N with N agents, where perfect speedup would be 1.0 x N if there was no parallelization and communications overhead.
Chal-5 Ability to harness possible synergies among tasks: When possible, learning some tasks may improve learning speed or performance at other, related tasks.
To address the SKILL challenge, we take inspiration from neuroscience. Many approaches to LL involve at least partially retraining the core network that performs tasks (feature extraction backbone plus classification head), as every new task learned. But transmitting and then merging these networks across multiple agents would incur very high communications and computation costs. With the exception of perceptual learning, where human visual cortex may indeed be altered when learning specific visual discrimination tasks for days
2

---

Published in Transactions on Machine Learning Research (05/2023)
weeks (Goldstone 1998 Dosher & Lu 2017), there is little evidence that our entire visual cortex - from early stage filters in primary visual cortex to complex features in inferotemporal cortex - is significantly altered when we just learn, e.g., about a new class of flowers from a few exemplars. Instead, the perirhinal cortex (and more generally the medial temporal lobe) may be learning new representations for now objects by drawing upon and combining existing visual features and representations from visual cortex (Deshmukh el al. 2012). This may give rise to specialized "grandmother colls' (Bowers) 2017] (or Jennifer Aniston neurons; Quiroga al. (2005); Quiroga (2017) that can be trained on top of an otherwise rather immutable visual cortex backbonn. While the grandmother cell hypothesis remains debated in neuroscience (vs. distributed representations; Valdez et al. (2015)), here, it motivates us to explore the viability of new light lifelong learning scheme, where the feature extraction backbone and the latent representation are fixed, and each new object class learned is represented by a single new neuron that draws from this representation.
From this inspiration, we propose a simple but effective solution to SKILL based on new lightweight lifelong learning (LLL) agents. Each LLL agent uses a common frozen backbone built-in at initialization, so that only the last layer (head) plus some small adjustments to the backbone (beneficial biases) are learned for each task. To eliminate the need for task oracle, LLL agents also learn and share summary statistics about their training datascts, or share few training images, to help other agents assign test samples to the correct head (task mapper). On a new, very challenging dataset with 102 image classification tasks (5,033 classes in total, 2,041,225 training, 243,464 validation, and 243,464 test images), we achieve higher accuracy compared to 8 LL baselines, and also near-perfect parallelization speedup.
Our main contributions are: (1) We formulate a new Lifelong learning challenge, Shared Knowledge Lifelong Learning (SKILL), which focuses on parallel (sped up) task learning and knowledge sharing among agents. We frame SKILL and contrast it with multi-task learning, sequential LL, and federated learning (Sec. B (2) A new LL benchmark dataset: SKILL-102, with 102 complex image classification tasks. To the best of our knowledge, it is the most challenging benchmark to evaluate LL and SKILL algorithms in the image classification domain, with the largest number of tasks, classes, and inter-task variance (Sec. 4p. (3) A solution to the SKILL problem: Lightweight Lifelong Learning (LLL) for efficient knowledge sharing among agents, using fixed shared core plus task-specific shareable modules. The need for a task oracle eliminated by using a task mapper, which can automatically determine the task at inference time from just input image (Sec. 5p. (4) Our SKILL algorithm achieves SOTA performance task accuracy (less catastrophic forgetting), low shared (communication ion) resources, and high speedup ratio (Sec. 6 (5) The proposed Lightweight Lifelong Learner shows promising forward knowledge transfer, which reuses the accumulated knowledge for faster and more accurate learning of new tasks.
three main metrics: High LL
2 Related Works
2.1 Lifelong Learning
Lifelong Learning (LL) aims to develop Al systems that can continuously learn address new tasks from new data, while preserving knowledge learned from previously learned tasks (Masana et al. 2022). It also refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experiences (Parisi et al. 2019 LL is challenging because it is usually assumed that the training data from previous tasks is not available anymore while learning new tasks; hence one cannot just accumulate training data over time and then learn from all the data collected so far. Instead, new approaches have been proposed, which fall under three main branches (De Lange et al., 2021): (1) Regularization methods add an auxiliary loss term to the primary task objective to constrain weight updates, so as to minimally disturb previously learned knowledge while learning new tasks. The extra loss can be penalty on the parameters (EWC (Kirkpatrick et al.) 2017). MAS (Aljundi et al. 2018) and S1 ZZenk et al. 2017
on the feature- space (FDR [Benjamin et al. 2018)) ), such as using Knowledge Distillation (LwF [Li & Hoiem 2017). DMC (Zhang el al. 20200). (2) Parumeler-Isolation melhods assign a fixed set. of model parameters to a lask and avoid over-writing them when new tasks are learned (SUPSUP (Wortsman et al. 2020)). PSP (Cheung ctt al. 2019) and BPN (Wen et al.) 2021). (3) Rehearsal methods use a buffer containing sampled training data from previous tasks, as an auxihary to new task's training set. The buffer can be used either at the end of the task training (iCaRL, ER (Rebuffi et al. 2017b Robins 1995)) or during training (GSS, AGEM,
3

---

Published in Transactions on Machine Learning Research (05/2023)
legend
TTsk
Physical Region (R)
Agont
Agent with task abiltee
Share knowledge jommuniccaann
Leaming
iie
Multi-task Learning
b) Sequential Lifelong Learning
Federated Learning
R2
Acnterr
3mo
Rs
Ne
Shared Knowledge Lifelong Learning (SKILL)
Communicate agents
Learning
mull
bostenn
x) Midl-teiic
X
X
Learning
Serpartiod
Linkong
X
V
X
X
Learning
o) Federated
X
X
V
V
1-1
di Shared Knowlodge
v
V
V
Leerning (SKILL))
Figure 1: SKILL vs. related learning paradigms. a) Multi-task learning (Caruana 1997): one agent learns all tasks at the same time in the same physical location. b) Sequential Lifelong Learning (S-LL) (Li & Hoiem 2017): one agent learns all tasks sequentially one location, deploying LT-specific machinery avoid task interference, c) Federated learning (McMahan et al., (017):: multiple agents learn the same task in different physical locations, then sharing learned knowledge (parameters) with center agent. d) Our SKILL: different S-LL agents in different physical regions each learn tasks, and learned knowledge is shared among all agents, such that finally all agents can solve all tasks. Bottom-right table: Strengths & weaknesses of each approach.
AGEM-R, GSS, DER, DERPP Lopez-Paz & Ranzato, 2017 Chaudhry et al., 2018 Aljundi et al. 2019 Buzzega et al. 2020 However, most. traditional LL a algorithms cannot satisfy the requirement of SKILL: parallel learning for speeding up, and sharing knowledge among agents.
2.2 Multi-task Learning
Multi-Task Learning (MTL) aims to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks (Zhang & Yang. 2021 Crawshaw 2020 Ruder 2017 The main difference between MTL and SKILL is that. MTL assumees that all lasks are located in the same physical region, and that one can access the datasets of all tasks the same time (Zhang & Yang 2021)
While MTL learns multiple tasks together, SKILL assumes that different knowledge sources separated in different physical regions and different agents should learn them in parallel.
2.3 Federated Learning
Federated learning (FL) is machine learning setting where many clients (e.g., mobile devices, networked whole organizations) collaboratively train model under the orchestration of central server, while keeping the training data decentralized Kairouz et al. (2021); Li et al. (2020); Bonawitz et al. (2019). As shown in Fig. [ compared with SKILL: (1) FL agents usually learn the same lask from multiple partial datasets in different locations, relying on the central server to accumulate and consolidate the partial knowledge provided by each agent. In contrast, SKILL agents solve different tasks, and share knowledge over decentralized network. (2) Each SKILL agent may learn multiple tasks in sequence, and hence must solve the L.I. problem of accumulating new knowledge while not forgetting old knowledge. Sequences of tasks
computers,
4

---

Published in Transactions on Machine Learning Research (05/2023)
and the LL problem are usually not primary focus of FL, with few exceptions Yoon et al. (021 which still do not directly apply to SKILL, as they focus on single task for all agents and use a central server. Because federated learning relies on a central server, it is susceptible to complete failure if that server is destroyed; in contrast, in SKILL, as long as not all agents are destroyed, the surviving agents can still share and master some of the lasks.
2.4 Other methods that may help solve SKILL
One related direction is to share a compact representation dataset: Dataset distillation Wang et al. (2018) combines all training exemplars into a small number of super-cxemplars which, when Iearned from using gradient descent, would generale the same gradients as the larger, original training set. However, the distillation cost is very high, which could not satisfy the 0.5NN speedup requirement. Another related direction is to reuse shared parameters for different tasks: Adversarial reprogramming Elsayed et al. (0188) computes a single noise pattern for cach task. This pattern is then added to inputs for a new task and fed through the original network. The original netw processes the combined input + noise and generates an output, which is then remapped onto the desired output domain. However, the cost of the reprogramming training process is high, which could not satisfy the 0.5.V specdup requirement. Another related direction is use a generative adversarial network (GAN Goodfellow et al. (2020)) to learn the distribution of different datasets and generate for replay. Closed-loop GAN (CloGAN Rios & Itti (2018) ) could be continuously trained with new data, while at the same time generating data from previously learned tasks for interleaved training, However, the GAN needs more parameters to transmit, and the high training time does not satisfy the 0.5.V speedup requirement.
3 Shared knowledge in lifelong learning (SKILL)
The chief motivation for SKILL is to enable the next generation of highly-effciient, parallclizable, and resilient lifelong learning.
Assumptions: (1) A population N agents wants to learn a total of T different tasks separated into N physical regions. (2) Each agent i asynchronously Iearns 1 T; S T tasks, in sequence, from the distinct inputs and operating conditions it encounters. As in standard LL, training data from previous tasks is not available anymore while learning the next task. (3) Each agent performs
a "teacher" for its T; tasks, by sharing what it has learned with the other N - 1 agents; at the same time,, each agent also performs a "student" by receiving knowledge from the other N - 1 agents. In the end, every agent has the knowledge to solve all T' tasks. Fig. [ contrasts SKILL with other learning paradigms. Note how here we use "teacher" and "student" to distinguish the two roles that every agent will perform; this is differont from and not to be confused with other uses of student/teacher lerminology, for example in knowledge distillation.
(4) There is a perfect task oracle training time, i.., each agent is told which tasks it should learn. (5) There is clear scparation between tasks, and between training and test phases.
Evaluation metrics: (1) CPU/computation expenditure. This metric is important to gauge the efficacy of an approach and its ability to scale up with more agents operating in parallel. Wall-clock time is the main metric of interest, so that speedup can be achieved through parallelism. Thus, if N agents learn for I unit of time, wall-clock time would be 1, which is an N-fold speedup over a single sequential agent. In practice, speedup < N expected because of overhead for sharing, communications, and knowledge consolidation. Because wall clock time assumes a given CPU or GPU speed, we instead report the number of multiply- accumulate (MAC) operations.
(2) Network/ communication expenditure. Sharing knowledge over a network is costly and hence should be minimized. To relate communications to computation, and hence allow tradc-offs, we assumm a factor = 1, 000 MACs / byte transmitted. It is hyperparancter in our results that can be easily changed to adapt lo different network lypes (e.g., wired vs. wireless). (3) Performance: After the population of N agents has collectively learned all T tasks, we report aggregated (averaged) performance over all T tasks (correct classification rate over all tasks). Note how here we assume that there no task oracle at test time. After training, agents should be able to handle any input from any task without being told which task that input corresponds to. SKILL does not assume a free task oracle because transmitting
5

---

Published in Transactions on Machine Learning Research (05/2023)
Skill-102 Stat
100
Comparison
SKILL-102
Visual decatrion
C/ar100
F-CelebA
Fine grained lasks
y Tasks
102
20
10
level difficulty
7,059
100
Images
5,162,149
1,889,142
60,000
1,180
1,440,086
Max A
60-
Picure Reality
apsuracyúillial ty)
99.60%
97 97.5%%
27.45%
36.30%
49. 48.50%
Mix
Nature
Diteerent classilication target
5
1
Task Object
object recognition
imagg slyle (raaturelari (aad)
20
style classification
X
scene cas ficction
counting number medical diagnosis
Mix super/line-
classification
10!
10a
10a
# of classes for each task
Fine Grained 6 tasks Stat
F-CelobA Stat
Cifar1 00 Stat
Visual Domain Decathlon Stat
100
100
100
100
level of difficuity
level of difficuity
level of difficuity
level of difficulty
80
80
60
60
40
40
000000
20
20
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
of classes for each task
of classes for each task
of classes for each task
classes for each task
Figure 2: (a) SKILL-102 dataset visualization. Task difficulty (y-axis) was estimated as the error rate of a R8 trained from scratch on each lask for a fixed number epochs. Circle size reflects dataset size (number of images). (b) Comparison with other benchmark datasets including Visual Domain Decathlon
Rebuffi al.] 70). Cifar-100 (Krizhevsky et al.) 2009). F-CelebA (Ke et al. 2020), Fino-grained 6 tasks
Russakovsky et al.
2014) (Wah et al.] 2011), (Nilsback & Zisserman 2008b), (Krause et al. 2013 (Saeeh
& Elgammal 2015).
Eitz et al. 2012 c) Qualitative visualization of other datasets, using the same legend
and format as in a..
training data across agents potentially very expensive. Thus, agents must also share information that will allow receiving agents to know when new test input relates to each received task.
Open questions: What knowledge should be shared? SKILL agents must share knowledge that is use- Tul to other agents and avoid sharing local or specialized knowledge that may be misleading, in conflict with, or inappropriate to other agents. The shared knowledge may include model parameters, model structure, patpraaiiaattonssspeiaaaaatttttt
input data, specific contextual information, etc. There are also size/memory cenemory/coominn /cmmmuication constraints for the shared knowledge. When and how to share? Different com- munication network topologies and sharing frequencies likely would lead to different results. Here, we will sidestep this problem and assume fully connected communication network, and broadcast sharing from each agent to all others each time new lask has been learned.
4 SKILL-102 dataset
We use image classification as the basic task framework and propose a novel LI. benchmark dataset: SKILL- 102 (Fig. 2 SKILL-102 consists of 102 image classification datascts. Each one supports one complex classification task, and the corresponding dataset was obtained from previously published sources (e.g., task 1: classify flowers into 102 classes, such as lily, rose, petunia, etc using 8,185 wrain/val/test images (Nilsback) & Zisserman 2008a task 2: classify 67 types of scenes, such as kitchen, bedroom, gas station, library, etc Quattoni & Torralba 2009;; full dataset sequence and details in Suppl. Fig. S5.
using 15,524 images
In total, SKILL-102 is a subset all datasets/taste stasks and images in DCT, and comprises 102 tasks, 5,033 classes and 2,041,225 2,5 training images (Suppl. Sec. and Suppl. Fig. S5). After training, the algorithm is presented 243,464 test images and docides, for each image, which of the 5,033 classes it belongs to (no task oracle). To the best of our knowledge, SKIL.I-102 is the most challenging completely real (not synthesized
6

---

Published in Transactions on Machine Learning Research (05/2023)
permuted) image classification benchmark for LL and SKILL algorithms, with the largest number of tasks, number of classes, and inter-task variance.
5 Lightweight Lifelong Learner for SKILL
To satisfy the requirements of SKILL (see Introduction), we design Lightweight Lifelong Learning (I.LL.) agents. The design motivation is as follows: We propose to decompose agents into a generic, pretrained, common representation backbone endowed into all agents at manufacturing time, and small task-specific decision modules. This enables distributed, decentralized learning as agents can learn their own tasks independently (Chal-T). It also enables lifelong learning (Chal-2) cach agent by creating a new task-spocific module for each new task. Because the shared modules are all operating in the common representation of the backbone, this approach also satisfies ( (Chal-3). Using compact task-specific modules also aims to maximize speedup through parallelizatio (Chal-4). Finally, we show a few examples where knowledge from previously learned tasks may both accelerate the learning and improve the performance on new tasks (Chal-5).
Fig. g shows the overall pipeline and 4 roles for each agent. Agents use a common frozen backbone and only a compact task-dependent "head' module is trained per agent and task, and then shared among agents. This makes the cost of both training and sharing very low. Head modules simply consist of (1) a classification layer that operates on top of the frozen backbone, and (2) set of beneficial biases that provide weight task-specific re-tuning of the backbone, to address potentially large domain gaps between the task-agnosti backbone and the data distribution of each new task. To eliminate the need for task oracle, LLL agents also learn and share task anchors, in the form of summary statistics about their training datasets, or share few training images, to help other agents assign test samples to the correct head at test time (task mapper). Two representations for task anchors, and the corresponding task mapping mechanismss, are explored: Gaussian Mixture Model Classifier (GMMC) and Mahalanobis distance classifier (MAHA). Receiving agents simply accumulate received heads and task anchors in banks, and the anchors for all tasks received so far by an agent are combined to form a task mapper within that agent. We currently assume a fully connected communication network among all agents, and every agent, afler learning a new lask, broadcasts its head and task anchor to all other agents. Hence, all agents become identical after all tasks have been learned and shared, and they all can master all tasks. At test time, using one of all identical agents, we first run input data through the task mapper to recover the task, and then invoke the corresponding head to obtain the final system output. The task mapper eliminates the need for task oracle at test time. The combination of using a pre-trained backbone, task-specific head and BB, and task mapper enables lifelong learning in every agent with minimal forgetting each agent learns sequence of tasks (see results).
Pretrained backbone: We use the xception (Chollet, 2017) pretrained on ImageNet (Deng et al. 2009). as it provides a good balance between model complexity and expressivity of the embedding. The backbone is embedded in every agent at manufacturigg time and is frozen. It processes 299 x 299 RGB input images, and outputs a 2048D) feature vector. Any other backbone could be used, depending on available resources.
Beneficial Biases: To address potentially large domain shifts between ImageNet and future tasks (e.g.. line-drawing datasets, medical imaging datasets, astronomy datasets), we designed beneficial biases (BB). Inspired by the Beneficial Perturbation Network (BPN) of Wen et al. (2021), BB provides set of task- dependent, out-of-network bias units which are activated per task. These units take no input. Their constant outputs add to the biases of the neurons already present in the backbone network; thus, they provide one bias value per neuron in the core network. This is quite lightweight, as there are far fewer neurons than weights in the backbone (22.9M parameters but only 22k neurons in xception). Different from BPN, which works best in conjunction with an LL method like EWC (Kirkpatrick et al.] (017)) or PSP (Cheung et all, 2019), and only works on fully-connected layers, BB does not require EWC or PSP, and can perform as add-on module on both convolutional layers (Conv) and fully-connected layers (FC). Specifically, for each Conv layer, we have
(1) with input feature x E Rwwhhe, output fcature y E w''''''o', b IRC' is the original frozen bias of the backbone, and B Rdd is our learnable beneficial bias. The size of B equal to the number of kernels (c') in this Conv layer. (w. h and w', h², d' denote the width, height and channels of the input and output feature
= Conv(x) + +b+ B
7

---

Published in Transactions on Machine Learning Research (05/2023)
R
wew
Fll Caredyen 2021443 mager
Agent
Agents
R
23 Flower
Fastion
images
e4000 saaees
R
Agant
vo clacic
inagee
Ra
ELEYP
More Agents In more reglons
Agent
Region,
Training
Share knowledge after each fask learning with other agents
Fixed backbone
Mora tasks
Learrable
12
Time
T5
Learnalle
Boneficial Blas
images
Task enchor
Receive Knowledge
agents
Testing
mi
Knowledge consolidate
Task
Task
mapper
Figure 3: Algorithm design. Top: overall pipeline, where agents are deployed in different regions to learn their own tasks. Subsequently, learned knowledge is shared among all agents. Bottom: Zoom into the details of each agent, with main roles: 1) Training: agents use common pre-trained and frozen backbone, stored in ROM memory at manufacturing time (gray trapezoid with lock symbol). The backbone allows the agent to extract compact representations from inputs (c.g., with xception backbone, the representation is latent vector of 2048 dimensions, and inputs are 299 x 299 RGB images). Each agent learns a lask-specific head (red triangle) for each new task. A head consists of the last fully-connected layer of the network plus our proposed LL beneficial biasing units (BB) that provide task-dependent tuning biases to all neurons in the network (one float number per neuron). During training, each agent also learns a GMMC or Mahalanobis task anchor which will form task mapper. 2) Share knowledge with other agents: each agent shares the learned task-specific head, Beneficial Bias (BB), and GMMC module (or training images for Mahalanobis) with all other agents. 3) Receive knowledge from other agents: each agent receives different heads and GMMC/Mett MaMhaaaabbonobs task mapper anchors from other agents. All heads are stored in head bank and all task anchors are consolidated to form task mapper. 4) Testing: At test time, an input is first processed through the task mapper. This outputs a task ID, used to load up the corresponding head (last layer + beneficial biases) from the bank. The network is then equipped with the correct head and is run on the input to produce an output.
sdie respectively.) For FC layers,
y FC(x) +b+
(2)
8

---

Published in Transactions on Machine Learning Research (05/2023)
with Z E R y E be R' and B E R", The size of B (beneficial bias) is equal to the number of hidden units (1) in this FC layer.
GMMC task mapper: To recover task at test time, each agent also learns Gaussian Mixture clusters
(GMMC) RRoos& & Itti 2020) that best encompass each of its tasks' data, and shares the cluster parameters (means + diagonal covariances). This is also very fast to learn and very compact to share. As shown Fig. 3 bottom right), during training, each agent clusters its entire training set into Gaussian clusters:
A f(z) = = E.),
Σ - =1
(3)
We use k = 25 clusters for every task (ablation studies in Appendix). In sharing knowledge, each agent performs a 'teacher' role on its learned task and shares the mean and diagonal covariance of its clusters with all other agents (students). In receiving knowledge, each agent performs a "student" role and just aggregates all received clusters in a bank to form task mapper with KT clusters, keeping track of which task any given cluster comes from: mapp() (1) : 1,--, (NKT, ΦKT) : At test. time, image 2y is evaluated against all clusters received so far, and the task associated with the cluster closest to the test image is chosen: Task Dmap((M ((Nm; @m)), where m arg max, (P(m,x))) The probability of image I; belonging to the m Gaussian cluster is given by:
p(m, = (arlir Em) En= (nrlar En)
(4)
Mahalanobis task mapper: To perform as a task mapper, the Mahalanobis distance (MAHA) method (Lee et al. 2018) learns C class-conditional Gaussian distributions Ma Ε) 1,2, ... C, where C is the total number of classes of all T tasks and Σ is tied covariance computed from samples from all classes. The class mean vectors and covariance matrix of MAHA are estimated as: Me N. Eiz; 4; (No: number of images in each class) and Σ 1 Ein (e)(z; - pc) (N: total number of images shared to the student agent). In training, each Leacher agent computes the mean of each class within its task and randomly samples a variable number m of images per class. In our experiments, we use m = images/clas /class for every task. During sharing knowledge, each agent shares the sample class means along with the saved images with all other agents. The shared images received by the student agents are used to compute the tied covariance. Similar to GMMC, the student agents also maintain task mapper to keep track of which task any given class comes from. For test image I, MAHA computes the Mahalanobis distance for all classes received so far and assigns the test image to the task associated with the smallest Mahalanobis distance, defined as:
N'
argmin(a - 1 -Me)
(5)
System implementation details: (1) Frozen xception backbone (Chollet) 2017), with 2048D latent rep- resentation. (2) Each agent learns one 'head' per task, which consists of one fully-connected layer with 2048 inputs from the backbone and e outputs for classification task with classes (e-g., task is to clas- sify c == 102 types of flowers), and BB biases that allow us to fine-tune the backbone without changing its weights, to mitigate large domain shifts. (3) Each agent also fits k = 25 Gaussian clusters in the 2048D latent space to its training data. (4) At test time, a test image is presented and processed forward through the xception backbone. The GMMC classifier then determines the task from the nearest Gaussian cluster. The corresponding head is loaded and it produces the final classification result: which image class (among 5,033 total) the image belongs to. (5) The workflow is slightly different with the Mahalanobis task mapper: while GMMC clusters are learned separately at each teacher for each task the task is learned, the Mahalanobis classifier is trained by students after sharing, using 5 images/clase shared among agents. (6) Agents are implemented in py Torch and run on desktop-grade GPUs (e.g., nVidia 3090, nVidia 1080).
6 Experiments and results
Each LLL agent in our approach is a sequential lifelong learner, capable of learning several tasks in its physical region, one after the other. Hence, before we show full results on the SKIL. challenge, first
9

---

Published in Transactions on Machine Learning Research (05/2023)
compare how well LLL can learn multiple tasks sequentially in single agent, compared to baselines LL algorithms. This is the standard LLL scenario where tasks are learned one after the other and data from previous tasks is not available while learning new tasks.
Baselines: We implemented 8 baselines from the literature. For those that require a task oracle, we (unfairly to us) grant them a perfect task oracle (while our approach uses imperfect GMMC or Mahalanobis task mappers). When possible, we re-implement the baselines to use the same pretrained xception backbone our approach. This ensures a fair comparison where every approach is granted the same amount of pre-training knowledge and the same feature processing ability. The two exceptions are PSP [Sheung et al. (2019) that uses ResNet-18, and SUPSUP Wortsman et al. (2020) that uses RosNet-50.
Our baselines fall in the following 3 categories (De Lange et al. 2021 (1) Regularizalion methods add an auxiliary loss term to the primary task objective to constraint weight updates. The extra loss can be a penalty on the parameters (EWC (Kirkpatrick et al.] 2017), MAS (Aljundi et al.) 2018} and SI [Zenke jet al. 2017)) or on the feature-space (FDR (Benjamin et al. 2018)), ) such as using Knowledge Distillation (DMC Zhang et al. 2020). ). We use EWC as the representative of this category: one agent learns all 102 tasks in sequence, using EWC machinery to constrain the weights when new task is learned, to attempt to not destroy performance on previously learned tasks. We also use SI, MAS, LwF, and Online-EWC as baselines of this type. (2) Isoaeler-Tsolation melhods assign a fixed set of model parameters to a lask and avoid over-writing them when new tasks are learned (SUPSUP (Wortsman et al. 2020 PSP (Cheung et al. 2 We use PSP as the representative of this category: one agent learns all 102 tasks in sequence, generating a new PSP key for each task. The keys help segregate the tasks within the network in an attempt to minimize interference. We used the original PSP implementation, which uses different backbone than ours. PSP accuracy overall hence may be lower because of this, and thus we focus on trends (decline in accuracy as more tasks are added) as opposed to only absolute accuracy figures. We also used SUPSUP as baseline of this type. (3) Rehearsal methods use a buffer containing sampled training data from previous tasks, as an auxiliary to a new task's training set. The buffer can be used either at the end of the task training (iCaRL, ER (Rebulli el al. 2017b Robins 1995)) or during training (GSS, AGEM, AGEM-R, GSS, DER, DERPP (Lopez-Paz Paz & Ranzato 2017] Chaudhry et al. 2018 Aljundi et al.) 2019 Buzzega et al, 2020 We use ER and as the reprosentative of this category: One agent learns all 102 tasks in sequence. After learning each task, it keeps a memory buffer with 10 images/class (size hence keeps increasing when new tasks are learned) that will later be used to rehearse old tasks. When learning a new task, the agent learns from all the data for that task, plus rehearses old tasks using the memory buffer.
Accuracy on first task: To gauge how well our approach is achieving lifelong learning, we plot the accuracy on the first task as we learn from 1 to 102 tasks, in Fig. + There is nothing special in our dataset about the first task, except that it is the first one. A good LL system is expected to maintain its accuracy on task 1 even more subsequent tasks are learned; conversely, catastrophic interference across tasks would rapidly decrease task 1 accuracy with more learned tasks. Overall, our approach maintains the highest accuracy task 1 over time, and virtually all of the accuracy degradation over time is due to increasing confusion in the task mapper (e-g., curves for Mahalanobis task mapper alone and LLL w/BB w w/MAHA are nearly shifted versions of each other). Indeed, once the task is guessed correctly, the corresponding head always performs exactly the same, no matter how many tasks have been learned.
Normalized accuracy on first 10 tasks: We compare our method to the baselines on the first 10 tasks, when up to 20 subsequent tasks are learned. A good LLL system should be able to maintain accuracy on the first 10 tasks, while at the same time learning new tasks. Because in SKILL-102 different tasks have different levels of difficulty, we normalize accuracy here to focus on degradation with an increasing number of new tasks. For example, the accuracy of our method (L.LI w/o BB) when learning a single task is 92.02% for task 1, but only 52.64% for task 6, which is much harder. Here, we define a normalized accuracy as the accuracy divided by the initial accuracy just after a given task was learned (which is also the best ever accuracy obtained for that task). This way, normalized accuracy starts at 100% for all tasks. If it remains near 100% as subsequent tasks are learned, then the approach is doing a good job at minimizing interference across tasks. Conversely, rapidly dropping normalized accuracy with an increasing number of subsequent tasks learned indicates that catastrophic interference is happening.
10

---

Published in Transactions on Machine Learning Research (05/2023)
a)
1st Task Acc Across Task Training
LLL(Ours) (MAHA, no BB)
1.000
LLL(Ours) (MAHA, BB)
GMMC
0.950
Accuracy
LINEAR
(LL((urs) (GMMC, BB)
0.900
LLL((urs) (GMMC, no BB)
0.875
Mahanalobis
0.850
BB
0.0
0
20
40
60
80
100
b)
# Tasks Already Been Trained
1st Task Acc Across Task Training
1.0
LLL(Ours) (MAHA, no BB)
LLL(OUrs) (MAHA, BB)
PSP
Online_EVv EWC
Accuracy
0.6
LWF
EWC
0.4
SI
ER
0.2
SUPSUP
MAS
20
40
60
80
100
Tasks Already Been Trained
Figure 4: Accuracy task 1 (learning to classify 102 types of flowers) a function of the number of tasks learned. a) Comparison between methods. b) Comparison between our best and other baselines. Our approach is able to maintain accuracy on task 1 much better than the basclines as more and more tasks are learned: while our approach does suffer some interference lask accuracy remains to within 90% of its initial best even after learning 101 new tasks (for the 4 LLL variants, BB=beneficial biases, MAHA=Mahalanohhi Distance task mapper, GMMC-GMMC task mapper). In contrast, the accuracy EWC, PSP, and several other baselines on task catastrophically degrades to nearly zero after learning just new tasks, even though granted these methods perfect task oracle. The best performing baseline, ER, is of the episodic buffer type (a fraction of the training set of each task is retained for later rehearsing while learning new tasks), with an un-bounded buffer that grows by 10 images/ /class. This methods does incur higher (and increasing) training costs because of the rehearsing (Suppl. Sec. D.) Note how SUPSUP does not experience any degradation on task 1, which a desirable feature this approach. However, a drawback that SUPSUP is not able, even from the beginning, to learn task as well as other methods (50.64% accuracy vs. over 90% for most other approaches). We attribute this to SUPSUP's limited expressivity and capacity learn using masks over random backbone, especially for tasks with many classes, Indeed, SUPSUP can perform very well on some other tasks, usually with smaller number of classes (c.g., 91.93% correct SVHN, 93.188. 18% Brazillian Coins, 99.11% on UMNIST Face Dataset; see Supp. Fig. S6).
Our results in Fig. 5 show that, although not perfect, our approach largely surpasses the baselines in its ability maintain the accuracy of previously learned tasks, with the exception of SUPSUP, which suffers degradation (see caption of Fig. 5 for why).
Task mapper accuracy after learning 1 to 102 tasks: To investigate how well our approach is expected to scale with more tasks, we computed task mapper accuracy on all tasks learned so far, after learning 1, 2, evaluate degradation with more tasks that is due to incrcasing confusion
3, ... 102 tasks. This allows the task mapper, as opposed to being due classification difficulty of newly added tasks. Results are shown in Fig. Task mapping accuracy starts at 100% after learning 1 task (all test samples correctly assigned
11

---

Published in Transactions on Machine Learning Research (05/2023)
task 0
Ours LLL(Ours) (MAHA, no BB)
task
LLL(Ours) (MAHA, BB)
1.0
1.0
task
Normalized Accuracy
Normalized Accuracy
task
6.8
task 4
0.6
6.6
task
task
task
0.2
task
task 9
0.
6000
# Subsequent task learned
of Subsequent task learned
EWC
PSP
ER
SUPSUP
1.0
Normalized Accuracy
Normalized Accuracy
Normalized Accuracy
Normalized Accuracy
0.0
c.a
0.8
6.0
0.6
0.6
6.6
0.6
0.2
62
02
0.2
0000
13 15 # of Subsequent task learned
6.0
15 17 # of Subsequent task learned
01
3
7
51 00
7 13 15 17 # of Subsequent task learned
0.0
5
13 17
# of Subsequent task learned
LWF
SI
MAS
Online EWC
1.0
Normalized Accuracy
Normalized Accuracy
Normalized Accuracy
Normalized Accuracy
0.8
0.8
an
0.
0.6
8.6
9.6
0.8
0.4
6.4
0.
64
0.2
6.2
92
0.2
00
6.001
20.1
Subsequent task leameed
Subsequent task learned
Subsequent task learned
Subsequent task leamed
Figure 5: Normalized accuracy the first 10 tasks (one per curve color) as up to 20 additional tasks are learned. Our LLL approach is able to maintain high normalized accuracy on the first 10 tasks, while all other baselines except SUPSUP sulfer much stronger catastrophic interference. SUPSUP is special case as there is no interference among successive tasks when a perfect task oracle available. Hence normalized accuracy for all tasks remains at 100%. However, we will see below that the absolute accuracy of SUPSUP is as good.
to that task by Mahalanobis or GMMC), then decreases more tasks are learned, eventually still achieving 87.1% correct after 102 lasks for MAHA, and 84.94% correct for GMMC. It is important to note that in our approach, any loss in accuracy with more tasks only comes from the task mapper: once the correct head is selected for a test sample, the accuracy of that head remains the same no matter how many heads have been added to the system. In contrast, other baseline methods may suffer from catastrophic forgetting for both the task mapper and the classification model when more tasks are learned, as further examined below.
When using GMMC task mapping, the regression line is y 00.0012 + 0.952, which intercepts zero for T = 800 tasks. Thus, with the distribution of tasks in dataset, extrapolate that T 500 is realistic as is. Since task interference in our system only comes from GMMC, pushing beyond T 500 might require more than A: = 25 GMMC clusters per task, which would increase CPU and communications expenditure. When using Mahalanobis task mapping, the results are similar with an intercept at. T = 978, though this approach incurs a slightly higher communications cost (discussed below).
A bsooute accuracy: The normalized accuracy figures reported so far were designed factor out variations in individual task difficulty, to focus degradation due to interference among tasks. However, they
12

---

Published in Transactions on Machine Learning Research (05/2023)
Average accuracy MAHA
Average accuracy GMMC
100%
100%
Average accuracy
80%
90%
y==0.001x+0.952
Average accuracy
90%
y==0.0012x+0.952
0.77
R 0.82
60%
50%
50%
40%
40%
30%
30%
20%
16 11
21 26
36 41 46 51 56 61 66 71 75 81 86
91 96
102
11 16 21 26
36 41 46 51 56 61 66 76 81 86 91 96 102 Number of tasks learned
31
Number of tasks learned
Figure 6: Task mapper accuracy on all tasks learned so far, as function of the number of tasks Icarned, when using Mahalanobis (left) or GMMC (right) task mappers. Our approach is able to maintain good task mapping accuracy as the number of tasks increases.
Task Mapper accuracy for each task
1.0
MA-A
0.8
Accuracy
0.6
0.4
0.2
0.0
0
10
20
30
40
50
60
70
80
90
100
Task-ID
Xception Head accuracy for each task
1.0
w/o 00
0.8
Accuracy
0.6
0.4
0.2
0.0
0
10
20
30
40
50
60
70
80
90
100
Task-ID
Figure 7: Absolute accuracy per task after learning 102 tasks. (Top) Absolute accuracy of the GMMC and Mahalanobis task mappers alonc shows quitc bit of variability, indicating various degrees of overlap among lasks. (Bottom) Absolute accuracy of the main xception+head network alone (with or without BB, assuming perfect task mapper) also shows significant variability, indicating various degrees of difficulty per task. The accuracy with BB is overall slightly higher than without BB (orange bars higher than corresponding blue bars in the bottom panel), as further explored in the next figure.
also factor out the potential benefits of BB in raising absolute task accuracy, and they obfuscate the absolute performance of baselines. Hence, we here also study absolute task accuracy.
We first plot the absolute accuracy of our LLL approach, soparately for each task, in Fig. scparating whether BB used or not, and which task mapper used. This shows that our SKILL-102 dataset provides range of difficulty levels for the various tasks, and quite hard overall. BB improves accuracy nearly
13

---

Published in Transactions on Machine Learning Research (05/2023)
a)
Average Acc Across Task Training
1.0
LLL(Ours) (MAHA, no BB)
LLL(Ours) (MAHA, BB)
0.8
Average Accuracy
GMMC
LINEAR
0.6
LLL(Ours) (GMMC, BB)
LLL(Ours) (GMMC, no BB)
Mahanalobis
BB
0.0
20
40
60
80
100
b)
# of Tasks Already Been Trained
Average Acc Across Task Training
1.0
LLL(OUrs) (MAHA, no BB)
LLL(Ours) (MAHA, BB)
Average Accuracy
PSP
Online_EWe EWC
0.6
LWF
EWC
0.4
SI
0.2
ER
SUPSUP
0.0
MAS
20
40
60
100
# of Tasks Already Been Trained
Figure 8: Average absolute accuracy on all tasks learned so far, as function of the number of tasks learned. Our LLL approach able to maintain higher average accuracy than all baselines. BB provides a small but reliable performance boost (LLL w /BB vs. LLL w/o BB). The sharp decrease in early tasks carries no special meaning except for the fact that tasks 4,8,10 are significantly harder than the other tasks in the 0-10 range, given the particular numbering of tasks in SKILL-102. Note how again SUPSUP has a low accuracy for the very first fask. This is because of the nature of its design; indeed, SUPSUP is able to learn some other tasks in our sequence with high accuracy (Suppl. Fig. S5).
all datasets, at an extra computation cost, detailed below. As promisod, BB improves accuracy quitc dramatically on some datasets which have large domain gap compared ImageNet used pretrain the backbone (e-g., 31.98 percent point improvement. with BB on deepvp that contains dashcam images, 24.92 percent point improvement on CLEVR, 24.5 percent point improvement on Aircraft, 20.75 percent point improvement SVHN; full details in Suppl. Fig. S5).
We then plot the absolute accuracy averaged over all tasks learned so far in Fig. 8 The absolute accuracy for GMMC and Mahalanobis is the same as before. However, now the absolute accuracies for the full LLL models and for the basclines conflate two components: 1) how much interference exists among tasks and 2) the absolute difficulty of each of the tasks learned so far.
Computation and communication costs, SKILL metrics: The baselines are sequential in nature, trying to implement them using multiple agents does not make sense it would only add communication costs but not alleviate the sequential nature of these LL approaches. For example, for the EWC baseline, one could learn task 1 on agent A then communicate the whole xeeption weights to agent B (22.9 M parameters = 91.6 MBytes) plus the diagonal of the Fisher Information matrix (another 22.9 M parameters), then agent B would learn task 2 and communicate its resulting weights and Fisher matrix to agent C, etc. Agent B cannot start learning task before it has received the trained weights and Fisher matrix from agent A because EWC does not provide mechanism to consolidate across agents. Thus, we first consider one agent that learns all 102 tasks sequentially, with no communication costs.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1: Analysis of computation expenditures and accuracy for our approach and the baselines, to learn all 102 tasks (with total of 5,033 classes, 2,041,225 training images) in single agent. Here we select LLL, no BB, MAHA as reference (1x CPU usage) since it is the fastest approach, yet still has higher accuracy than all baselines. For our approach, MAHA leads to slightly higher accuracy than GMMC, at roughly the same computation cost. All baselines perform worse that our approach, even though they also requires more computation than our approaches that do not use BB. BB adds significatly to our computation cost, but also leads to the best accuracy when used with MAHA.
Training (MACs)
CPU usage
Average accuracy
VS. Ours, no BB, MAHA
after learning 102 tasks
(LL((Orr)--Singe) Agent, BB, GMMC
1673E+16
-Ix
67.43%
(LLLOls))-S Agent, BB, GMMC (LL((Ous)-Singe) Agent, no BB, Mahalanobis
+8
-90.7x
70.58%
1.73E+16
1x (reference)
68.87%
LLL Single Agent, BB, Mahalanobis EWC
1-56++18
-90.7x
12.1%
1.75E+18
-1013xx
8.86%
PSP
6,28F++17
-36.4x
25.49%
ER SUPSUP EWC-ONLINE LWF SI MAS
4.63E+18
-262.8x
35.32%
101E+18
-8.
56.22%
+8
-00.1x
7.77%
1556+18
-90.5x
8.41%
2.07E 18
~120.1x
13.80%
2.06E
-119.6x
20.54%
Table. 1 shows the computation expenditures (training time in terms of the number of mulliply-accummulae (MAC) operations needed to learn all 102 datasets) for our approach and the baselines. Our approach overall has by far the lowest computation burden when BB is not used, yet all 4 variants of our approach perform better than all baselines. BB increases accuracy but at a significant computation cost: This is because, to compute BB biases, one needs to compute gradients through the entire frozen backbone, even though those gradients will only be used to update biases while the weights remain frozen in the backbone.
Our approach presents the advantage that it can also be parallelized over multiple agents that each learn their own tasks in their own physical region. All agents then learn their assigned tasks in parallel. Fach agent is the 'teacher' for its assigned tasks, and "student" for the other tasks. Then all agents broadcast their shared knowledge to all other agents. As they receive shared knowledge, the students just accumulate it in banks, and update their task mapper. After sharing, all agents know all tasks (and are all identical). As mentioned above, the main source of performance degradation in our approach is in the task mapper, which gets increasingly confused at increases.
For our baselines, we are not aware of a way to parallelize their operation, except that we were able to create modified version of SUPSUP that works several parallel processors. In our modified SUPSUP, cach agent learns mask for each of its tasks, then commuicattes its masks to all other agents. At test time, we (unfairly to us) grant it a perfect task oracle, as our GPUs did not have enough memory to use the built-in task mapping approach of SUPSUP, given our 102 tasks and 5,033 classes (this would theoretically require 1.02 TB of GPU memory).
Table. 2 shows the computation and networking expenditures for our approach and our modified SUPSUP to learn all tasks in the SKILL-102 dataset. Because some algorithms run on GPU (e.g., xception backbone) but others on CPU (e.g., GMMC training), and because our tasks use datasets of different sizes, we measure everything in terms of MACss tion on most hardware). To measure MACs for cach component of our framework, we used a combination of empirically measured, framework-provideer (e.g,, py Torch can compute MACs from the specification of all layers in a network), or sniffed (installing a hook in some algorithm that increments a counter each time a MAC is exocuted). To translate communication costs to MACs, we assume a nominal cost of a = 1, 000 MACs to transmit one byte of data. This is a hyperparameter in our results that can be changed based deployment characteristies (e.g., wireless vs. wired network). The amount of data shared per task for
operations, which are implemented as one atomic instruc-
15

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: Analysis of computation and network expenditures for our parallelized LLL approach and our parallelized SUPSUP, to learn all T 102 tasks. Our approach supports any number of agents N such that 1 ISN& T. Maximum speedup is expected when N = ' and each agent learns one task, then shares with all others. Here, we report numbers for T = 102, N = 51, and each agent learns 2 tasks in sequence. Note that in our approach, accuracy is not affected by N, only the amount of parallelization speedup increases with N. Note how in this table we still report MACs but taking parallelization into account (e.g., teacher CPU for N agents is single-agent CPU divided by N). Teacher CPU: Time learn lasks from their training datasets, plus to possibly prepare data for sharing (e.g., compute GMMC clusters). Communications: Our LLL agents communicate either GMMC clusters Mahalanobis training images, while our modified SUPSUP communicates masks. Here assume that there is communication bottleneck at the receiver (student)) the shared data from 100 tasks needs to be received serially, over single networking interface for each student. Hence our communication figures are for all the shared data from all other tasks apart from those an agent learned itself. We convert communication costs to equivalent MACs by assuming 1,000 MACs per byte transmitted. BB adds a small extra communication cost, to transmit the biases. Student CPU: For GMMC, students do not do any extra work (hence, student CPU 0); for Mahalanobis, students compute covariance matrix for all 102 tasks. Speedup factor: is just total MACs for single agent divided by total MACs for parallel agents and by N. All approaches here achieve near perfect parallelization (> 0.99N, where 1.0N is perfect). Accuracy: In addition to being faster when BB is not used, our LLL variants still all outperform the parallel SUPSUP in accuracy, by a large margin (> 10%).
Communi- caations (bytes)
Student CPU (MACs)
CPU usage vs. Durs-SKILLL
Average accuracy after learning tasks
Teacher CPU (MACs)
Total (MACs)
Parallelization efficiency (xN)
no BB, MAHA
LLL:(Ous)- Agents, no BB, GMMC LLL(Ouxs)- Ageuts, BB, GMMC (LL(Our)--Multipp Agents, BB, Mahalanobis
1.69 14
8.22E 07
0.00E 00
169E 14
0,93898519
-0.98x
67.43%
1.53++16
1.03E+08
0,00E+000
1,33E+16
0 0.909999334
-87.2x
70.58%
1.69E+14
6.72E+009
5.00E++09
1.76E+14
0.996630551
1x (reference)
68.87%
(.L[(Orrs-Muultiph Agents, BB, Mahalanobis
1,53++16
6.74E+000
5:00F+000
1,33E+16
0,098962712
-87.3x
72.1%
Parallel SUPSUP,
Perfect Tack Oracle
9.918+15
3.03E+188
0,00E+100
9.91E+15
0.999999617
-36.4x
56.22%
our approach is quite small (details in Suppl. Sec. G): 813 KBytes/task for LLL with GMMC, no BB; 883 KBytes/tan task for LLL with GMMC, BB; 1.74 MBytes for LLL with MAHA, no BB; and 1.81 MBytes for LLL with MAHA, BB (on average, given that our tasks have 49.34 classes each on average; see Suppl. Fig. S5).
Our results in Table. 2 show:
Our approach has very low paralleliaation overhead, which leads to almost perfect speedup 0.99N for all variants. Indeed, teachers just learn their task normally, plus a small overhead to train GMMC their own tasks, when GMMC is used. Communcattions are less than 2 MBytes per task (Suppl. Sec. G). Students either do nothing (just accumulate received knowledge in a bank) or update their Mahalanobis task mapper.
The basclines have comparatively much higher training cost, yet their performance is poor. Performance of episodic bulfer / rehearsing methods might be improved further by increasing buffer size, but note that in the limit (keeping all training data for future rehearsing), this gives rise to a 5, 000x increase in training time (Suppl. Sec. D).
7 Shared Knowledge Accumulation, Reuse and Boost
As our system learns many tasks, may occur that some tasks overlap with others, i.c., they may share similar images and/or class labels. Here, we explore two approaches to handle such overlap.
16

---

Published in Transactions on Machine Learning Research (05/2023)
7.1 Corrective approach to task overlap/synere
Since our LLL learners can learn a large number of lasks while solving SKILL problems, some synergy occur across tasks if one is able to detect when two classes from two different tasks are equivalent, as shown in Fig. g We implemented a method to compare the semantic distance of the predicted class name and the actual class name. Originally, after the GMMC infers the task that a test image may come from, we would immediately consider the image as misclassified if the predicted task wrong. With the consideration of semantic similarity between class names, we will now always load the prediction head corresponding to the predicted task given by GMMC and use it to infer the class name. If the class guessed incorrectly, but, in the end, the final class name is equivalent to the correct one, then can declare success that particular test image (Fig. To obtain the pairwise similarity, we constructed similarity matrix that stores the semantic distance, measured by the cosine similarity of word embeddings, for all the class names. Those embeddings were obtained from CLIP's (Radford et al. (2021) text encoder based on GPT-2. If the similarity between the predicted class name and the actual class name greater than a threshold (empirically chosen for now), then we declare correct prediction.
Dining
chicken (ood-101)
Bathroom Jiuno Roor Isagri
boat sbe)
like Stantord Online, Producto
cherros [Fond2059)
Blouse Reom Image Datarit)
Sitnnford Online Products
lowtbau classhcatioon
shoes Dutant)
Blcycle (Shashua
churro
JuTTinddo Staree
(NWPU RESESCAS)
(Food
Figure Left: similar classes with a cosine similarity the CLIP embedding greater than 0.90. Right: similar classes with cosine similarity greater than 0.95. This can help correct spurious errors where, for example, a test image from class "bike" from the Stanford Online Products dataset could also be considered correctly classified if the system output were "bicycle" from the Sketches dataset.
As our 102-task dataset contains 5,033 object classes, the full similarity matrix is triangular 5,033 x 5,033 (too large to display here).
The approach yields small but consistent improvement in accuracy (Fig. 10 This is one way in which we can handle possible overlap between tasks, which may inevitably arise when large numbers of tasks are learned.
7.2 Learning approach to task overlap/synergy
The ability to reuse learned knowledge from old tasks to boast the learning speed and accuracy of new tasks is a potential desirable feature of LT, algorithms. Here, this might be achieved if, when learning a new task, an LLL agent could "borrow" the knowledge of old tasks, from not only itself but also the shared knowledge from any other agents.
One important design feature of our LLL agents is that they share partial heads across tasks: Our heads are a single layer with 2,048 inputs (from the xception backbonc) and e outputs for task with classes. Thus, each of the output neurons connected to the 2,048 inputs, but there are no lateral connections. This means that we can consider each of the e output neurons individually as evidence provider for their associated class (remember the analogy "grandmother cells" in Introduction). We can then cherry-pick sets of 2,048 weights corresponding individual classes of previously learned tasks, and use them as initialization for some similar classes in new tasks to be learned. As we show below, this greatly accelerates learning of the
17

---

Published in Transactions on Machine Learning Research (05/2023)
Original Accuracy
Threshold-0.95
Threshold=0.9
0.8
0.4
0.2
0
MIT
Sketches
WikiArt Dataset
Office_
Office_
Office_ Home_
Art. Images
House Room
UIUC
Land
IFood2019
100
Standford NWPU
Home_
Home_
101
Type
Sports
Use
Sports
Online
RESISCAS
Scene
Clipart
Product
Classificatio
Image
Event
Products
Dataset
Classif
Classif
lcation
Figure 10: Correcting spurious errors by realizing when two distinct classes from two tasks actually are the same thing. The approach provides a small but consistent improvement in accuracy over bascline (which declares failure soon as task mapping failed), here shown 15 datasets that have some overlap.
similar classes, compared to starting with randomized initial weights, and also yields higher accuracy these new classes.
1) New task is combination of old tasks: To validate our idea, a new learning paradigm is proposed to use previously learned weights when a new task contains classes that were already learned previously. This experiment considers two datasets and two sets of weights representing the old knowledge, and new dataset that contains all classes from both datasets. Simply normalizing and concatenating the linear weights leads poor performance. Hence, instead, we normalize the weights during training by their p-norm, and concatenate the normalized weights as the new task's weights. The experiment combinations of 2 datasets chosen from 20 datasets, and the average results show that there is very small accuracy loss initially (epoch 0). After a few extra training epochs, we reach a higher accuracy than training new weights from scratch (random initialization; Fig. I1
conducted over 190
The mathematical version. During training, a constraint is added. Let W1 be the full set of 2, 048 x mo
weights of the first dataset, and W2 be the weights of the second dataset, where W1 (" and W2 =
...T.. Normal Linear Layer training's forward path Wx. Define W
where w w:/p
...
norm (w)) Hence, during training, Linear Layer training's forward path 9 = W'x. And concatenate(Wi',
Wa') was used as the weights for the new combined task.
Since the weights are normalized class-wise and not task-wise, this method can be used on any combination of previously leaned classes. For example, for 10 tasks containing 100 classes e₁ C100 and new task containing C1, C3, C10. Co0, can simply find the corresponding w/, w/g, wio w/y and concatenate them together.
Choice of ps We find that using the 2-norm causes the classifier to converge to state where all weight vectors have the same magnitude, which causes an accuracy drop for the old task. Hence, we choose to use the infinity norm, which is still modulated by the weight magnitudes, and is still easy to transfer.
2) New task is different but similar to old tasks. In the previous setting, we assumed the new task classes combination of different old task classes. In more general situation, the task classes
18

---

Published in Transactions on Machine Learning Research (05/2023)
average_accuragy vs. epochs
0.64
0.62
accuracy
0.60
0.58
pretrained
0.56
scratch
0.54
10
15
20
25
epochs
Figure 11: Learning speed for a given object class when the corresponding weights are initialized randomly (orange) vs. from previously learned weights of a similar object class found in one the previously learned tasks (blue), averaged for 190 combinations two previously learned tasks. this example, best accuracy is already reached after just 2 training epochs in the blue curve. In contrast, it takes up to 30 epochs train from random initialization, with st.il final accuracy in the orange curve that is lower than the blue curve. This approach hence leads to significant learning speedup when tasks contain some similar classes.
new classes that all other agents never learned before, but we could still borrow some learned knowledge from similar learned classes. For instance, shown in Fig. 9 the knowledge of classes shown on top of the figure may be helpful to learn the new classes shown at the bottom.
We conduct four different experiments (for 4 pairs of datasets that share some related classes) to show the knowledge boost when we learn a new task. We first check if a learned old task shares similar knowledge with the new one. For instance, before we learn the MIT indoor scenes dataset, we find that the House Room Image Dataset contains classes that are similar to the new classes, in the CLIP embedding space. So we match each class from MTT indoor scenes dalaset come from the House Room Image Dataset. If the class similarity is larger than a threshold, we treat it matched class, then we the similar old class weights to initialize the weights of the new class. If a new class was not matched with old classes, use random initialization. We also conduct corresponding controlled experiment by using random initialization for all new classes. The results of all 4 experiments are shown in Table 3
the previously learned classes, which in this case
Datasets
initialization All
shot
shot
3 shot
Old task
House Room Image Datasel
random
0.86
0.77
0.73
0.52
New task
MIT Indoor Scenes
ours
0.89
0.83
0.8
0.71
Old task
Standford Online Products
random
0.78
0.61
0.61
0.6
New task
Office Home Product.
ours
0.8
0.62
0.59
0.6
Old task
100 Sports
random
1
0.98
0.92
0.92
New task
UIUC Sports Event Dataset
ours
1
0.99
0.97
0.97
Old task
FFod2019
random
0.61
0.42
0.35
0.3
New task
Food-101
ours 0.64
0.5
0.46
0.43
Table 3: Boosted LLL learning when previously learned weights from similar classes can be used to initialize
learning of new classes. We repeat. the experiment with her learning from all images the training set of the new task, or only 10, 5, or 3 images per class. Overall, re-using previously learned weights of similar classes boosts accuracy, usually (but not always) more when the task is only learned from few exemplars (which is much faster than learning from scratch from the whole dataset).
In a more general learning scenario, the new task classes may correspond to similar but not necessarily identical classes in different old tasks. For example, having learned about SUVs may help learn about vans more quickly. Here we conduct two new experiments (Fig. [2 In EXP-1, the new task is sketch image classification, and the classification weights of each class are initialized from learned old class that
19

---

Published in Transactions on Machine Learning Research (05/2023)
is harvested from different learned tasks with the help of CLIP-based word matching. For instance, the classification weights of van are initialized with the weights of SUV from the previously learned Stanford- CARs dataset; the classification weights of Topwear are initialized with the weights of t-shirt from the learned Pashion-Product datasct, etc (total: 5 pairs of classes). The results show that our initialization leads better performance (0.98 v.s. 0.93) and faster convergence (3 v.s. epochs) compared with random initialization. This shows that our method can reuse the shared knowledge from other agents to improve the learning of new tasks. Similar performances shown in EXP-2, which identical to EXP-1 except that it uses different pairs of classes.
EXP-1 Setting
Stanford Cars: AM Goneral Hummer SUY 2000
Fashion, Product: Topwear
OlficaHHmme, Cipprt:
Beb 80m: plana
CORa50
EXP-1 Results
Breekkers
light bulh
Weight Initialization
Accuracy
Converge Epochs
closses
Random
0.93
lighbbub
Ours
0.98
New task: Skatch dassificatin
EXP-2 Setting
Stanford_ Cars: Land Rtaee Rtnne Rover SUIV 2012
Pokemon: Dodrla
Office Harce-Homm_Paaee
Product: Homme_Prucc__
Product Product:
Backpack
Mug
EXP-2 Results
Learned
Weight Initialization
Accuracy
Converge
Epoch
Iuck
blyycee
badkpask
Random
0.90
Ours
0.95
New task: Sketch cassiiaation
Figure 12: Two experiments where the weights from previously learned similar but not identical classes are successful in boosting learning of new classes. Left: pairs of similar classes (according to CLIP). Right: accuracy achieved with weight transfer vs. random initialization.
7.3 Further boost with Head2Toe
A possibly complementary approach to BB to address domain gaps is Head2'Toe (Evci et al. 2022), where the last layer can now directly draw from potentially any of the previous layers in the backbone. This has been shown help alleviate domain gaps, as some of the lower-level features the backbone may be useful solve tasks with big gap, even though the top-level backbone features may not. However, Head2Toe has very high computation cost to select which layers should connect to the head, which is why we have not used it in our main results. Here, explore how that cost of selection of the most appropriate layers to connect to the head for a given task can be eliminated by re-using the computations already expended for BB: Intuitively, layers which have large absolute BB magnitude may also be the most useful to connect to the head.
Compared to the conventional Head2Toe (Evei et al. 2022) with two-stage training (first, select which layers will connect the head, then train those connections), our new BB+H2T uses the biases that have been previously trained and stored in the BB network for feature selection. Specifically, all the biases in the BB network and selected the top 1% largest biases. Then, picked the feature maps corresponding to the selected indices, average pooled them and flattened them to 8,192-dimensional vectors. After that, we concalenated all flattened feature vectors along with the logits of the last layer (after pooling layer, before softmax) in the BB network. Finally, we trained the concatenated vector with Adam optimizer, 0.001 lcarning rate, and 100 epochs. This approach, when combined with BB and MAHA, improved performance averaged over all tasks by 0.78% (when perfect task mapper is available; or by 0.56% when using MAHA).
first concatenated
20

---

Published in Transactions on Machine Learning Research (05/2023)
8 Discussion and Future Works
We have proposed new lightweight approach to lifelong learning that outperforms all baselines tested, and also can benefit. almost perfectly from parallelization. We tested the approach on a new SKILL-102 benchmark dataset, which we believe is the largest non-synthetized lifelong learning challonge datasct to date. While many previous efforts have employed many lasks, those were usually synthesized, e.g., as permutations over a single base dataset (e.g., 50 permuted-MNISS' tasks in Cheung et al.] (2019)). SKILL- 102 contains novel real data in each task, with large inter-task variance, which is a better represcntative of realistic test scenarios. Our proposed lightweight LL points to new direction in LL research, as we find that one can simply use lightweight task-specific weights (head) combined with maximizing the leverage obtained from task-agnostic knowledge that is rapidly adapted by a compact BB module to handle each new task. Our results show how this lightweight design is better able to handle large scale lifelong learning tasks, and also solves our SKILL challenge very well.
We credit our good performance on both sequential lifelong learning and the SKILL challenge to our par- ticular lightweight lifelong learner design: fired backbone, which represent task-agnostic knowledge shared among agents, to minimize the complexity of task-specific knowledge parameters (the head); Beneficial Bi- ases, which on-demand shift the backbone to solve possibly large domain gaps for cach new task, with very compact parameters; a GMMC/MAHH global lask anchor for learned lasks, representing the (asks in the common task-agnostic latent space of all agents, which is easy to share and consolidate, and which eliminates the need for a task oracle at test time. Our results show that combination of this three components help our LLL work well.
Our approach uses a pretrained backbone to represent task-agnostic knowledge, which our results show is very effective strategy. For fair comparison, we also use the same pretrained backbone as the initialization for the baselines (except PSP and SUPSUP; see above). However, our fixed backbone design often cannot handle large domain gaps between new tasks and ImageNet. This is the reason why we proposed BB to relieve the domain gap by shifting the fixed parameters towards each new task with compact biases. Similar to other parameter-isolation methods, our model structure is growing on demand (though slowly) with the number of tasks (we add a new head per task, while they add new masks, keys, etc). Rehearsal- based basclines (c.g., ER) also grow, by accumulating more rehearsing exemplars over time. While some basclines do not grow (e.g., EWC), they also perform very poorly our challenging SKILL-102 dataset.
To further speed up our method with BB, we could use BB on only partial layers. For instance, if we use BB only the last half of the layers in the backbone, we will use only half of the current time to train the model. In future experiments, we will test whether this still gives rise to a significant accuracy benefit.
Currently, we use the CLIP embedding space to match new class with learned old classes, which uses only language knowledge (class labels). As future work, we will use GMMC as a class matching mechanism to utilize the visual semantic information for matching. Specifically, when a agent learns new class, the agent will collect a few shots (e.g., 10 images) of the new class and then use the GMMC mapper (trained on all previous tasks) to decide whether these images belong to a learned task or not, with a threshold. If most of the images are matched to a learned task, we can then summon the shared head of that task to classify them, now to obtain a possible match for individual previously learned classes. If most of the images are classified into one previously Iearned specific class, we can use the weights of that class to initialize the new class weights, similar to what we have done in Sec.
A good task mapper is essential in our approach if one is to forego the use of a task oracle. Thankfully, our results show that task mapping can be achieved with high accuracy even with simple model like GMMC (over 84% correct for 102 (asks). Indeed, in our SKILL challenge, the task mapper is only solving a 102-way classification problem at the granularity of tasks, vs. a 5,033-way full SKILL classification challenge. Here, we focused on GMMC and MAHA, but many other alternatives could be investigated in future work. Our choice of GMMC was based on previous research that compared it to several other techniques, including self-organizing maps and multilayer perceptrons (Rios& & Itti 2020
21

---

Published in Transactions on Machine Learning Research (05/2023)
9 Conclusions
We have proposed new framework for shared-knowledge, parallelized LL. On new, very challenging SKILL-102 dataset, we find that this approach works much better than previously SOTA baselines, and is much faster. Scaling to > 500 difficult tasks like the ones in our new SKILII-102 dataset seems achievable with the current implementation.
Broader impacts statement: We believe that LLL will spur a new generation of distributed LL systems, it makes LL. more accessible to edge systems and more parallelizable. Thus, broader impacts expected to be positive in enabling more lightweight devices to learn at the edge and to share what they have learned.
Acknowledgement: This work was supported by DARPA 1219011,,,,,,4,, C-BRIC (one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA), and the Army Research Office (W911NF2020053). The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government or any agency thereof.
22

---

Published in Transactions on Machine Learning Research (05/2023)
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytclaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Compuler Vision (ECCV), pp. 139-154 2018.
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in newral information processing systems, 32, 2019.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arxiv:1805 08289, 2018.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzoechi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of machine learning and systems, 1:374 388, 2019.
Jeffrey S Bowers. Grandmother cells and localist representations: a review of current thinking, 2017.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in newral information processing systems, 33:15920-15980 15930, 2020.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75) 1997.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXie preprint arXiv:1812.00499, 2018.
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint frxiv:1902 1902.05220, 2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Michael Crawshaw. Multi-task learning with deep neural networks: A survey.
ar preprint
arXiv:2009.09796 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. 1EEE transactions on pattern analysis and machine intelligence, 44(7):3366-33)) 13336-33355,, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Sachin S Deshmukh, Jeremy I, Johnson, and James J Knierim. Perirhinal cortex represents nonspatial, but not spatial, information in rats foraging in the presence of objects: comparison with lateral entorhinal cortex. Hippocampus, 22(10):0045-0553 2012.
Barbara Dosher and Zhong-Lin Lu. Visual perceptual learning and models. Annual review of vision science, 3:343-366 2017.
Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM Transactions grophics (TOG), 31 (4):1-10, 2012.
Gamaleldin Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. Xiv preprint wrXie:1806.11146 2018.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe: Utilizing intermediate representations for better transfer learning. Xix preprint or 2rXim:2001.03520 2022.
23

---

Published in Transactions on Machine Learning Research (05/2023)
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):
228, 1999.
Robert Goldstone. Perceptual learning. Annual review of psychology. 49(1) 612, 1998.
lan Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):
139 144, 2020.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trendso in Machine Learning, 14(1 11:1 2200, 2021.
Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of mixed sequence of similar and dissimilar tasks. Advances in Neural Informalion Processing Systems, 13:18493- 1194-18850444 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114((3):3521133) 3526, 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained catego- rization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554 561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Tian Li, Anit Kunar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):00-60, 2020.
Zhizhong and Derck Hoiem. Learning without forgetting. IEEE transactions on pottern analysis and machine intelligence, 40(12))2935 2947, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-660 6476, 2017.
Mare Masana, Xialei Liu, Barthomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Brendan McMahan, Eider Moore, Danicl Ramage, Scth Hampson, and Blaise Aguera y Arcas. Communicationn-llleeeni learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017.
Maria-Elemt Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008a.
Maria- Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. 2008 Siath Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008b.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 413 413-420, 2009. doi: 10.100//CVPR.2205555655
24

---

Published in Transactions on Machine Learning Research (05/2023)
R. Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual repre- sentation by single neurons in the human brain. Nature, 435(7045):1102-1100 2005.
Rodrigo Quian Quiroga. The forgetting machine: Memory, perception, and the Jennifer Aniston neuron. BenBella Books, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PML.R, 2021.
Sylvestro-Alvse Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. volume 30, 20.
Sylvestre Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017b.
arviv preprinl Amanda Rios and Laurent Itti. Closed-loop memory gan for continual learning. arXiv:1811.01140 2018
Amanda Rios and Laurent Ttti. Lifelong learning without task oracle. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 255 263. IEEE 2020.
Anthony Robins. Catastrophic forgetting, rehearsal and pscudorchearsal. Connection Science, 7(2):123-145) 1995.
Sebastian Ruder. An overview of multi-takk learning in deep neural networks. arXiv:1706.0500 05098, 2017.
arXiv preprint
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge (2014). Xiv preprinl arxi: 10t:::409.055 007.0 2(3), 2014.
Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. arxiv preprint 1rX5v 00855, 2015.
André B Valdez, Megan Н Papesh, David M Treiman, Kris A Smith, Stephen D Goldinger, and Peter N Steinmetz. Distributed representation of visual objects by single neurons in the human brain. Journal of Newroscience, 35(13):5180 5186, 2015.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. 2011.
Tongzhou Wang, Jun-Yar Yan Zhu, Antonio Torralba, and Alexci A Efros. Datasct distillation. arXiv preprint arXiw:1811.10959 2018.
Shixian Wen, Amanda Rios, Yunhao Ge, and Laurent Itti. Beneficial perturbation network for designing general adaptive artificial intelligence systems. IEFE Transactions on Newral Networks and Learning Systems, 2021.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 08:15173-15188 2020.
Jachong Yoon, Wonyong Jeong, Giwoong Loe, Eunho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In International Conference Machine Learning, pp. 12073- 12086. PMLRR 2021.
25

---

Published in Transactions on Machine Learning Research (05/2023)
Friedemann Zenke, Ben Poole, and Surya Ganguli. Contimual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3990, PMTR, 2017.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C- C Jay Kuo. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE/CVP Winter Conference Applications of Computer Vision, pp. 1131-1140, 2020.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering. 34(12):5366-5609 2021.
26


Reviews:
[{"review_id": "hO4qeuDjlo", "reviewer": "Reviewer_4LNS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength：\n\n1.\tThe paper proposes a new benchmark, SKILL, which fills a gap in the existing benchmarks for lifelong learning and federated learning.\n\n2.\tThe paper presents comprehensive experimental results to evaluate the proposed method and demonstrate its effectiveness.\n\nWeakness：\n\n1.\tThis work mainly applies existing methods to a novel setting that combines federated learning with lifelong learning. However, there is a lack of significant new designs proposed to address the challenges of lifelong learning or federated learning. Therefore, the novelty of this work is limited.\n\n2.\tThe author proposes five challenges that SKILL needs to address, but it seems that the challenges proposed have not been well addressed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a distributed lifelong learning approach, where multiple physically distributed resources (datasets) are available, each with multiple tasks. Therefore, a Lifelong Learning agent (LLL) is assigned to each resource, and each LLL agent is a sequential lifelong learner capable of learning multiple tasks in its region one after the other. Each agent consists of a shared backbone and a specific head, and communication is enabled among agents to share knowledge, allowing each agent to solve all tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Question:\n\n1. The proposed setting is not commonly seen. Since each location only has data for some tasks, why is it necessary to test on tasks from other locations? Under what circumstances, in reality, is the proposed framework necessary? Could you give some examples?\n\nIt seems that the challenges proposed in the paper have not been well addressed:\n\n2. Challenge 1 did not use a global agent, it assumed that all distributed agents can communicate globally, which seems no different. \n\n3. Challenges 2. Which specific module contributed to addressing the lifelong learning challenges? Can you explain if you have designed mechanisms to prevent forgetting previously learned knowledge in lifelong learning? This is not clearly reflected in the paper.\n\n4. Challenge 3. I understand that the knowledge sharing mentioned in the paper refers to the ability of different agents to share their task-specific heads and GMMC through communication. However, in reality, the shared representation is the fixed backbone network. If each agent could extract generalizable knowledge from its own task and update the backbone network, would it achieve better knowledge sharing? \n\n5. Challenge 5. Can synergies occur between different agents? In my understanding, when an agent needs to perform tasks of other agents, it uses the corresponding task-specific head, so this is just an adaptation issue. Can we say that synergies only occur among tasks from one agent? Even so, this synergy mainly comes from the ability of the neural network itself, could you explain which specific module design of this work contributed to harnessing possible synergies among tasks?\n", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SxOgFBPuX7", "reviewer": "Reviewer_UV1p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The paper is fairly well written, and I only needed a few passes to get the key ideas.\n* The LLL approach is simple and well motivated, and the authors address a key test-time issue of not having an oracle.\n* The authors do a good job of clearly showing the computational and communication costs incurred by their method and baselines, and the baselines shown appear rigorous and fairly evaluated.\n* I found the experiments, especially the approach outlined to corrective and forward transfer, quite creative and definitely believe they would be interesting to the wider community.\n\n### Weaknesses\n* The SKILL problem setting itself seems a bit contrived; the motivation provided (e.g., object recognition in the sky and the bottom of the ocean) doesn't feel particularly obvious to me. There are motivations to do with resilience, but these don't seem that well explored in the paper.\n* I have some concerns over the performance of the baselines. For instance, EWC seems to be remarkably poor, despite being a fairly strong baseline in prior work.\n* I'm not entirely sure about the teacher-student nomenclature; teacher-student usually refers to a system whereby we have one neural network distilling knowledge into another, but in this instance, it just seems like this refers to the sharing of statistics for use in test-time task inference, which feels conceptually different. This might be confusing to readers more familiar with the former, especially given Teacher-Student approaches are used already to solve lifelong learning problems [1].\n* There are quite a few formatting/presentation issues that should be addressed.\n    * In Figure 3, \"Knowledge consolidate\" is covered by the white space of the \"Task mapper\" shape (bottom left).\n    * The graphs should be vectorized and their presentation improved; Figure 2c is blurry and hard to read. As another example, in Figure 4a., it is very hard to distinguish between the curves as there is so much whitespace between 0 and the worst performing method; the authors should clip the axis range.\n    * Figure 4 uses 1th in the title, should use '1st'\n    * Similarly, the tables in Figure 9 and 10 should be formatted correctly. At the moment they are just images, so are referred to as figures; however these should be properly formatted LaTeX tables, with a caption above said tables.\n    * Figure 12, legend and y-axis labels are squashed.\n    * Page 2: \"Each LLL agent uses a common frozen backbone built-in at ~manufacturing time~ *initialization*\"\n\n[1] Lifelong Teacher-Student Network Learning, Ye and Bors, IEEE Transactions on Pattern Analysis and Machine Intelligence 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors present the SKILL challenge, whereby distributed agents must learn in a lifelong manner in parallel across a set of sequential tasks. To solve this problem, the authors present a novel method called lightweight lifelong learning (LLL), which obviates issues regarding storage, lack of knowledge transfer, and test-time oracles.\n\nThe authors show that existing lifelong learning/continual learning approaches fail to perform well in the SKILL paradigm, and that LLL is able to maintain strong performance on prior tasks, while incurring minimal compute and communication costs. The authors then show evidence that the learned lightweight learners can be combined to facilitate forward transfer.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Can the authors provide a better motivation for the SKILL problem setting?\n* Can the authors explain why prior baselines perform so poorly in the SKILL-102 benchmark?\n* Could the authors provide clarity on the use of teacher-student in their LLL approach v.s. the canonical meaning in deep learning?\n* Could the authors update the presentation, as outlined by in the Weaknesses section?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FJZvp2l3Qb", "reviewer": "Reviewer_vikF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The setting of \"lifelong learning\" is a relevant problem for the audience of TMLR (and the broader field)\n- New benchmarks like SKILL-102 are valuable for the field\n- Figure 1 provides a good overview of the approach\n- It is laudable that the authors also evaluate their approach in terms of compute and communication overhead\n- The concept of incrementally task mappers is novel (to my knowledge) and works surprisingly well\n- The authors conduct extensive experiments on the SKILL-102 benchmark, comparing also to prior work (partly reimplemented)\n\nWeaknesses:\n- The considered setting comes with strong assumptions such as the ones listed below. These should be added in the \"Assumptions\" part in Section 3. The relevancy to the broader audience could be increased by considering a more general setting. Assumptions:\n    * all tasks being image classification tasks\n    * perfect task oracle at training time\n    * no two agents ever face the same task during training\n    * clear separation into training and testing phase (unrealistic in lifelong learning) \n    * trustworthy, reliable communication between all agents \n- The proposed SKILL-102 dataset is derived from another dataset (DCT), a paper on this dataset being currently under review. It is unclear if SKILL-102 can be seen as an independent contribution of this paper as it is a subset of DCT. Moreover, the dataset is not publicly available (at least this is note clear from the paper). \n- Evaluation is done essentially on a proprietary in-house benchmark only; this does not allow reproducibility and I don't see these experiments as \"convincing evidence of the papers claims\" as required by TMLR.\n- The presentation of the paper could be improved, overall I don't think 21 pages are required for exposition of the methods and evaluations conducted in this work.\n- The number of parameters of LLL grows linearly with the number of tasks. That is a disadvantage compared to some of the prior work\n- Section 7 feels unrelated to the core content of the paper and also somewhat unfinished (some parts of the exposition read more like a preliminary blogpost \"It is tricky to choose value of p.\", \"On a small sample of classes tested so far\" etc.). I think the paper might be improved by removing this section or moving it to the appendix (alternatively, a major revision of this part is required) . I am also a bit sceptical about the part on \"similarity between the predicted class name and the actual class name is greater than a threshold\": if the threshold goes to -infinity, then all classes are mapped onto a single class and accuracy goes trivially to 100%, if I understand correctly what the authors are doing. That would clearly be problematic.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new class of lifelong learners dubbed \"Lightweight Lifelong Learning\" (LLL). LLL is based on a pretrained and (mostly) frozen feature extraction backbone (Xception) and learns task-specific heads and updates biases in the backbone. Moreover, a task mapping is learned. LLL learns those task-specific parameters in a distributed manner on multiple agents that communicate learned parameters with each other. LLL is evaluated on the newly proposed SKILL-102 benchmark, where it outperforms other approaches in terms of performance, compute requirements and communication overhead.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Conducting additional experiments on established and publicly available benchmarks would be required in my opinion for acceptance.\n- As an additional baseline, it would be helpful to understand how a zero-shot CLIP based classifier would perform on the proposed SKILL-102 benchmark (with queries of the form \"an image of a <class>\" with the respective class added).\n- The SKILL-102 dataset needs to be released publicly, to ensure reproducibility and to be considered a contribution of this paper. Moreover, is it ensured that the pretraining data of Xception (ImageNet1k?) is disjoint from the data contained in SKILL-102?\n- It is somewhat surprising that the simple task mappers considered in the paper that operate in image space have relatively high accuracy. It would be good to understand how they can accomplish this and to release code and data to be able to conduct an independent reproduction of the results. \n- Figure 4a (and some later ones): if all methods have an accuracy > 85%, the range of the y-axis should not start with 0%, which makes the figure hard to read. Also, when possible, error-bars/confidence intervals would be desirable.\n- Minor: In caption of Figure 3, please also mention that BBs fall under the shared \"knowledge\"\n- Minor: The linear regression line in Figure 6 does not make sense, as the trend is clearly non-linear.\n\n- Minor: I assume statements like \"31.98% improvement with BB\" rather mean \"31.98 percent point improvement with BB\"?\n\n- Minor: Figure 6 would be more readable as a scatter plot (w BB vs. w/o BB)\n\n- Minor: Figure 2c \"other datasets' plots\" is not a descriptive Figure caption, please add more details here.\n", "broader_impact_concerns": "The reviewer does not have broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I'll be brief since there is not much need to elaborate, given the reviews: the reviewer consensus is that this paper makes a timely and meaningful contribution to the lifelong learning literature, and is of an acceptable standard for publication at TMLR. The reviewers all give fairly clear constructive feedback, which I see has been responded to and incorporated into an updated draft by the authors. I see no reason to further withhold acceptance.
Decision: Accept (poster)