Continual Learning: Applications and the Road Forward
Authors: Eli Verwimp
Rahaf Aljundi
Shai Ben-David
Matthias Bethge
Andrea Cossu
Alexander Gepperth
Tyler L. Hayes
Eyke Hüllermeier
Christopher Kanan
Dhireesha Kudithipudi
Christoph H. Lampert
Martin Mundt
Razvan Pascanu
Adrian Popescu
Andreas S. Tolias
Joost van de Weijer
Bing Liu
Vincenzo Lomonaco
Tinne Tuytelaars
Gido M van de Ven
Venue: Accepted by TMLR ()

Abstract: Continual learning is a subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting what was learned in the past. In this work, we take a step back, and ask: "Why should one care about continual learning in the first place?". We set the stage by examining recent continual learning papers published at four major machine learning conferences, and show that memory-constrained settings dominate the field. Then, we discuss five open problems in machine learning, and even though they might seem unrelated to continual learning at first sight, we show that continual learning will inevitably be part of their solution. These problems are model editing, personalization and specialization, on-device learning, faster (re-)training and reinforcement learning. Finally, by comparing the desiderata from these unsolved problems and the current assumptions in continual learning, we highlight and discuss four future directions for continual learning research. We hope that this work offers an interesting perspective on the future of continual learning, while displaying its potential value and the paths we have to pursue in order to make it successful. This work is the result of the many discussions the authors had at the Dagstuhl seminar on Deep Continual Learning, in March 2023.

Published in Transactions on Machine Learning Research Mlarch/2024/
Continual Learning: Applications and the Road Forward
Eli Verwimp"
KU Lewven, Belgium
Rahaf Aljundi
Toyota Motor Europe, Belgium University of Waterloo, and Vector Institute, Ontario, Canada University of Tübingen, Germany
Shai Ben-David
Matthias Bethge
Andrea Cossu
University of Pisa, Italy
Alexander Gepperth
University of Applied Sciences Fulda, Germany NAVER LABS Europe, Fronce University of Munich (LMU), Germany University of Rochester, Rochester, NY, USA
Tyler L. Hayes
Eyke Hüllermeier
Christopher Kanan
Dhireesha Kudithipudi
University of Texas at San Antonio, TX, USA Institute of Science and Technology Austria (ISTA) TU Darmstadt & hessian. Germany Google Deep UK
Christoph H. Lampert
Martin Mundt
Razvan Pascanu
Adrian Popescu
Université Paris-Saclay, CEA, LIST, France Baylor College of Medicine, Houston, TX, USA Computer Vision Center, UAB, Barcelona, Spoän University of Illinois Chicago, USA University of Pisa, Italy KU Leuven, Belgium KU Leuven, Belgium
Andreas S. Tolias
Joost van de Weijer
Bing Liu
Vincenzo Lomonaco
Tinne Tuytelaars
Gido M. van de Ven
Reviewed on OpenReview:
Abstract
Continual learning is subfield of machine learning, which aims to allow machine learning models to continuously learn on new data, by accumulating knowledge without forgetting what was learned in the past. In this work, we take a step back, and ask: "Why should one care about continual learning the first place?". We the stage by examining recent continual learning papers published at four major machine learning conferences, and show that contary-onntrrained settings dominate the field. Then, discuss five open problems in machine learning, and even though they might seem unrelated to continual lcarning at first sight, we show that continual learning will inevitably be part of their solution. These problems are model editing, personalization and specialization, on-device learning, faster (re-)training and reinforcement learning. Finally, by comparing unsolved problems and the current assumptions in continual learning, we highlight and discuss four future directions for continual learning research. We hope that this work offers an interesting perspective on the future of continual learning, while displaying its potential value and the paths have to pursue in order to make it successful. This work is the result of the many discussions the authors had at the Dagstuhl seminar on Deep Continual Learning, in March 2023.
desiderata from these
"Corresponding author: eli. aei.verrinppluuleeeenn.

---

Published in Transactions on Machine Learning Research Alarch/2024)
1 Introduction
Continual learning, sometimes referred to as lifelong learning or incremental learning, is a subficld of machine learning that. focuses on the challenging problem of incrementally training models on stream of data with the aim of accumulating knowledge over time. This setting calls for algorithms that can learn new skills with minimal forgetting of what they had learned previously, transfer knowledge across tasks, and smoothly adapt to new circumstances when needed. This is in contrast with the traditional setting of machine learning, which typically builds on the premise that all data, both for training and testing, are sampled i.i.d. (independent and identically distributed) from a single, stationary data distribution.
Deep learning models in particular are in need of continual learning capabilities. A first reason for this is their strong dependence on data. When trained on stream of data whose underlying distribution changes over time, deep learning models tend to adapt to the most recent data, thereby "catastrophically" forgetting the information that had been learned earlier (French) 1999). Secondly, continual learning capabilities could reduce the very long training times of deep learning models. When new data are available, current industry practice is to retrain a model fully from scratch on all, past and new, data (sce Example 3.4). Such re- training is time inefficient, sub-optimal and unsustainable, with recent. large models exceeding 10000 GPU days of training (Radford et al., 2021). Simple solutions, like freezing feature extractor layers, are often not an option as the power of deep learning hinges on the representations learned by those layers (Bengio et al. 2013). To work well in challenging applications in e.g. computer vision and natural language processing, they often need to be changed.
The paragraph above describes two naive approaches to the continual learning problem. The first one, incrementally training - or finetuning - a model only on the new data, usually suffers from suboptimal performance when models adapt too strongly to the new data. The second approach, repeatedly retraining a model on all data used so far, is undesirable due to its high computational and memory costs. The goal of continual learning is to find approaches that have better trade-off between performance and efficiency (e.g. compute and memory) than these two naive ones. In the contemporary continual learning literature, this tradc-off typically manifests itself by limiting memory capacity and optimizing performance under this con- straint. Computational costs are not often considered in the current continual learning literature, although this is challenged in some recent works, which we discuss in Sections 2 and
In this article, we highlight several practical problems in which there is an inevitable continual Icarning component, often because there is some form of new data available for model to train on. We discuss how these problems require continual learning, and how in these problems that what is constrained and that what is optimized differs. Constraints are hard limits set by the environment of the problem (e.g. small devices have limited memory), under which other aspects, such as computational cost and performance, need to be optimized. Progress in the problems we discuss goes hand in hand with progress in continual learning, and we hope that they serve as a motivation to continue working on continual lcarning, and offer an alternative way to look at it and its benefits. Similarly, they can offer an opportunity to align currently common assumptions that stem from the benchmarks we use, with those derived from the problems we aim to solve. Section 3 describes some of these problems and in Section 4 we discuss some exciting future rescarch directions in continual learning, by comparing the desiderata of the discussed problems and contemporary continual learning methods. The aim of this paper is to offer perspective on continual learning and its future. We do not aim to cover the latest technical progress in the development of continual learning algorithms, for this we refer to c.g. Masana et al. 2022 Zhou et al. (023) Wang ct al. 2023
2 Current continual learning
Before exploring different problem settings in which we foresce continual Icarning as uscful tool, we first wish to understand the current landscape. Our aim is to paint a clear picture of how memory and computat ional ional cost are generally approached in current continual learning papers. To achieve this, we examined continual learning papers accepted at four top machine learning conferences (ECCV 222, NeurIPS *22, CVPR '23 and ICML 223 to have a representative sample of the current field. We considered all papers with either "incremental', 'continual", 'Forgelling', 'ifellong' 'calastrophic' in their titles, disregarding false positives.
2

---

Published in Transactions on Machine Learning Research Mlarch/2024/
Opennizad-
compitational
Contratnci
Dosfrrrace
Compooeed
Neantthi
Nrne
41%
1-0%
5-10%
10-99%
100%
stored
Figure 1: Most papers strongly restrict data storage and do not discuss computational cost. Each dot represents one paper, illustrating what percentage of data their methods store (horizontal axis) and how computational cost is handled (vertical axis). The majority of these papers are in the lower-left corner: those that strongly restrict memory use and do not quantitatively approach computational cost. (i.e. it is at most discussed). For more details, see Appendix.
See Appendix for the methodology. For our final set of 77 papers, we investigated how they balance the memory and compute cost trade-offs. We discern five categories:
Not discussed: No clear mention of the impact of the proposed method/analysi. the cost.
Discussed: Cost discussed in text, but not quantitatively compared between methods.
Compared: Cost is qualitatively compared to other methods.
Construined: Methods are compared using the same limited cost.
Optimized: Cost is among the optimized metrics.
Many continual learning papers memory in variety of ways, most often in the form of storing samples, but regularly model copies (e.g. for distillation) or class means and their variances are stored well. We focus on the amount of stored data, as this is the most common use of memory, but discuss other memory costs in the Appendix. Of the examined papers, all but three constrain the amount of stored samples. So rather than reporting the category, in Figure [ we report how strongly data storage is constrained, using the percentage of all data that is stored. It apparent that the majority of these papers do not store any (raw) samples and many store only small fraction. Three notable exceptions that store all the raw data are a paper continual reinforcement learning (RL) Fuu ct al. 2022 something which is not uncommon in RL, see Section 3.5 The second one, by Prabhu et al. (2023a) studies common CL algorithms under a restricted computational cost. Finally, Kishore et al. (023)
study incremental document, retrieval, practical problem
involving continual learning.
While memory costs (for raw samples) are almost always constrained, computational costs are much less so. Sometimes simply discussing that there (almost) no additional computational cost can suffice, especially settings where memory is the crucial limiting factor. Yet it is remarkable that in more than 50% of the papers there is no mention of the computational cost at all. When it is compared, it is often done in the appendix. There are few notable exceptions among the analyzed papers that focus explicitly on the influence of the computational cost, either by constraining (Prabhu ot al. 2023a Kumari et al. 2022 Ghunaim et al. 2023) or optimizing it [Wang et al. 2022b Kishore el al. 2023 For a more elaborate discussion of measuring the computational cost, see Section 4.1 Together, these results show that many continual learning methods are the computational cost. They are
developed with a low memory constraint, and with limited attention two among other relevant dimensions of continual learning in biological systems (Kudithipudi et al. (0022) and artificial variants (Mundt et. al. 2022), yet with the naive solutions of the introduction in mind, they
3

---

Published in Transactions on Machine Learning Research MMarch/20244
are two crucial components of any continual learning algorithm. In the next section, we introduce some problems for which continual learning is inevitable. They illustrate that methods with a low computational cost are, just like methods that work in memory restricted settings, an important setting, yet they have not received the same level of attention.
3 Continual learning is not a choice
To solve the problems described in this section, continual learning is necessary and not just a tool that one could use. We argue that in all of them, the problem can, at least partly, be recast as a continual learning problem. This means that the need for continual learning algorithms arises from the nature of the problem itself, and not just from the choice of a specific way for solving it. We start these subsections by explaining what the problem is and why fundamentally requires continual learning. Next, we briefly discuss current solutions and how they relate to established continual learning algorithms. We conclude cach part by laying down what the constraints are and what metrics should be optimized.
3.1 Model editing
It is often necessary to correct wrongly learned predictions from past data. Real world practice shows us that models often imperfect, e.g. models frequently learn various forms of decision shortcuts (Lapuschkin) et al. 2019), , sometimes the original training data become outdated and are no longer aligned with current facts (e.g. a change in government leaders). Additionally, strictly accumulating knowledge may not always be compliant with present legal regulations and social desiderata. Overcoming existing biases, more accurately reflecting fairness criteria, or adhering to privacy protection regulations (c.g. the right to be forgotten of the GDPR in Europe (European Union 2016)), ) represent a second facet of this editing problem.
When mistakes are exposed, it is desirable to selectively edit the model without forgetting other relevant knowledge and without re-training from scratch. Such edits should thus only change the output of model to inputs in the neighborhood of the effected input-oulpul mapping, while keeping all others constant. The model editing pipeline (Mitchell et al.) 2022) first identifies corner cases and failures, then prompts data collection over those cases, and subsequently retrains/ retrains/updates updates the model. Recently proposed methods are able to locally change models, yet this comes at a significant cost, or model draw-down, i.e. forgetting of knowledge that was correct (Santurkar et al. 2021). Often the goal of model editing is to change the output associated with specific input from A to B, yet changing the output to something generic undefined is equally interesting case. Such changes can be important in privacy-sensitive applications, to e.g. forget learned faces other personal attributes.
Naively, one could retrain a model from scratch with an updated dataset, that no longer contains outdated facts and references to privacy-sensitive subjects, or includes more data on previously out-of-distribution data. To fully retrain on the new dataset, significant computational power and access to all previous training data necessary. Instoad, with effective continual lcarning, this naive approach can be improved by only changing what should be changed. An ideal solution would be able to continually fix mistakes, at much lower computational cost than retraining from scratch, without forgetting previously learned and unaffected information. Such solution would minimize computational cost, while maximizing performance. There is no inherent limitation on memory in this problem, although it can be limited if not all training data are freely accessible.
3.2 Personalization and specialization
Some of the most powerful machine learning models are trained very large datasets, usually scraped from the Internet. The result is model that is able to extract useful and diverse features from high-dimensional data. However, the vastness of the data they are trained on also has a downside. Internet data is generated by many different people, who all have their own preferences and interests. One model cannot fit these conflicting preferences, and the bost fit is close to the average internct user (Hu et al.] 2022b). However, machine learning models are often used by individuals or small groups, or for highly specific applications. This contradiction makes any possessive references such as 'my car' or 'my favorite band' by construction
4

---

Published in Transactions on Machine Learning Research Alarch/2024)
Example: 3.1
Lazaridou et al. 2021 used the customnews benchmark to evaluate how well a lan- guage model trained on news data from 1969 2017 performs on data from 2018 and 2019. They find that models perform worse on the newest data, mostly nouns (e.g. "Ardern" or "Khashoggi"), as well as words introduced because of soci- etal changes such "Covid-19" and "MeToo". They identify set of 287 new words that were not used in any document prior to 2018. Such new words are inevitable in futture texts too. To teach a model these changes they perform updates on the newly arriving data, which gradually improves the performance on the years 2018 and 2019 (a 10% decrease perplexity), yet at the cost of performance on earlier years 5% increase on all previous years). When weighing all years equally, the final model thus got worse than before updating.
proper
ambignous and impossible for the system to understand. Further, Internet scraped data do not always contain (enough) information to reach the best performance in specialized application domains like science and user sentiment analysis (Beltagy et al., 2019). In contrast to Section 3.1 here there are many new data that have strong relation to each other (e.g. scientific words), whereas model editing typically concerns fewer and less related, individual changes.
Domain adaptation and personalization are thus often nccessary. The topic has been investigated in the natural language processing (NLP) community for many different applications. Initially, fine-tuning on a supervised domain-specific dataset was the method of choice, but recently, with the success of very large language models (LLM), the focus has shifted towards changing only small subset of parameters with adapters (Houlsby et al. 2019). low-rank updates (Hu et al., 2022a) prompting (Jung et al. 2023). How- ever, these methods do not explicitly identily and preserve important knowledge in the original language model. This hampers the integration of general and domain-specific knowledge and produces weaker re- sults (Ke et al.j 2022). To identify the parameters that are important for the general knowledge in the LLM in order to protect them is a challenging problem. Recent works (Ke et al.) 2021 made some progress in balancing the trade-off between performance on in-domain and older data. In the computer vision field, similar work has also been done by adapting CLIP to different domains (Wortsman et al. 2022) and to include personal text and image pairs (Cohen al. 2022
No matter how large or sophisticated the pre-trained models become, there will always be data that they are not, or cannot be, trained on (e.g. tomorrow's 's data). Specialized and personalized data can be collected afterwards, either by collecting new data or by extracting the relevant bits from the larger, original dataset. The addition of such a specialized dataset is a distribution shift, and thus continual learning algorithms are requirod. The final goal is to train a specialized or personalized model, more computo-cffeiint than when trained from scratch, without losing relevant performance the pre-trained domain. This makes this problem different from transfer learning, where it is not requirement (Zhuang et al. 2020). When training is done on the original server, past data are usually available and can be used to prevent forgetting. Yet sometimes this is not the case, because training happens on more restricted (e.g. personal) device, then memory does become a constraint, which we elaborate on in the next subsection.
3.3 On-device learning
To offer experience aligned with user's preferences, adjusted to new personal environment, many deep learning applications require updates the deployed device. Cloud computing is often not available because of communication issues (c.g. in remote locations with restricted internet access, or when dealing with very large quantities of data), or to preserve the privacy of the user (e.g. for domestic robots, monitoring cameras). On such small devices, both memory and computational resources are typically constrained, and the primary goal to maximize model efficacy under these constraints. These tight constraints often make storing all user data and retraining from scratch infeasible, necessitating continual learning whenever the pre-trained capabilities should not be lost. during continued on-device training (see also Example 3.3
5

---

Published in Transactions on Machine Learning Research Mllrch/2024/
Example: 3.2
Dingliwal et al. 2023 personalize end-to-end speech recognition models with words that are personal to the user (e.g. family member names) or words that are very rare except in specialized environments (e.g. "ecchy- moses" in medical settings). With an extra attention module and pre- computed set of representations of the specialized vocabulary, they 'bias' the original model towards using the new rare and unknown words. The performance on the specialized words remarkably improved, yet with decrease in performance on non-specialist word recognition. In their exper- iments specialized tokens are less than 1% off all tokens, so even a relatively small decrease in performance on other tokens is non non-negligibl.
with ecchymosis result
...
with acto
These constraints, as well as increasingly complex computations for energy-accuracy trade-offs in real- time (Kudithipudi et al. 2023), limit the direct application of optimization typically used in cloud do- ployments. For example, existing methods only update the final classification layer of a pre-trained feature extractor (Hayes & Kanan 2022 Yet this relatively lightweight process becomes challenging when there is large domain gap between the initial training set and the on-device data. The latter is often hard to collect, since labeling large amounts of data by the user is impractical, requiring few-shot solutions. When devices shrink even further, the communication costs become significant, and reading and writing to memory can be up to ~99% of the total energy budget (Dally 2022). In addition to algorithmic optimizations for con- tinual learning, architectural optimizations offer interesting possibilities. These enhancements may include energy-efficien memory hierarchies, adaptable dataflow distribution, domain-specific compute optimizations like quantization and pruning, and hardware-software co-design techniques (Kudithipudi et al., 2022).
On-device learning from data that is collected locally almost certainly involves distribution shift from the original (pre-)ter )trainnng data. This means the sampling process is no longer i..d., thus requiring continual learning to maintain good performance on the initial training set. If these devices operate on longer time scales, the data they sample themselves will not be i.i.d. cither. To leverage the originally learned information as well as adapt to local distribution changes, such devices require continual learning operate effectively. Importantly, they should be able learn using only limited amount of labeled information, while operating under the memory and compute constraints of the device.
Example: 3.3
it was revealed how a robot vacuum cases sensitive ones, back to the company, be labeled and used in further training on central servers. In response, an R&D director of the company stated: "Road systems are quite standard, so for makers of relf-driving cors, you' 'll know how the lane looks f...j, but each home interior is vastly diffferent", acknowledging the need to adjust robots to the environment they working in. Our homes are highly diverse, but also one of the most intimate and private places that exist. Images can reveal every detail about them and should thus remain private. Adapting individual homes is necessary, but should not come at the cost initial smart abilities such as object recognition, collision prevention and planning, which are unlikely to be learned using only locally gathered data.
In 2022 article by MIT Review (Guo 2022
cleaner had sent images,
3.4 Faster retraining with warm starting
In many industrial settings, deep neural networks are periodically re-trained from scratch when new data are available, or when a distribution shift is detected. The newly gathered data is typically a lot smaller than the original dataset is, which makes starting from scratch a wasteful endeavor. As more and more data is collected, the computational requirements for retraining continue to grow over time. Instead, continual learning start from the inltial model and only update what is necessary to improve performance the

---

Published in Transactions on Machine Learning Research Mlarch/2024/
dataset. Most continual learning methods not designed for computational efficiency (Harun et al, 2023a), yet Harun et al. (003b)) show that reductions in training time by an order of magnitude are possible, while reaching similar performance. Successful continual learning would offer a way to drastically reduce the expenses and extraordinary carbon footprint associated with retraining from scratch (Amodei & Hernandez, 2018), without sacrificing accuracy.
The challenge is to achieve performance equal or better than a solution that is trained from scratch, but with fewer additional resources. One could say that it is the performance that is constrained, and computational cost that must be optimized. Simple approaches, like warm-starting, i.e. from previously trained network, can yicld poorer generalization than models trained from scratch on small datasets (Ash & Adams 2020), yet it is unclear whether this translates to larger datasets, and remains a debated question. Similar results were found in [Berariu et al, 2021 Dohare et al. 2023), which report a loss of plasticity, i.c. the ability to learn new knowledge after an initial training phase. In curriculum learning (Bengio et al., 2009), recent works have tried make learning more efficient by cleverly selecting which samples train when (Hacohen el. al. 2020). Similarly, active learning (Settles 2009) studies which unlabeled samples could best be labeled (given restricted budget) to most effectively learn. Today those fields have to balance learning new information with preventing forgetting, yet with successful continual lcarning they could focus more on learning new information as well and as quickly as possible.
Minimizing computational cost could also be rephrased as maximizing learning efficiency. Not having to re-learn from scratch whenever new data available, figuring out the best order to use data for learning, or the best samples to label can all contribute this goal. Crucially, maximizing knowledge accumulation from the available data is part of this challenge. Previous work (Hadsell et al. 2020: Hacohen et al. 2020: Pliushch et al. 2022) suggested that even when all data is used together, features are learned in sequential order. Exploiting this order to make learning efficient requires continual learning.
Example: 3.4
Continuous training one of the six important building blocks in MLOps (Machine Learning Operations, similar to DevOps), according to a Google white paper on the subject (Salama et al. 2021 This step is consid- ered necessary, response performance decays when incoming data characteristies change. They describe in great detail how to optimize this pipeline, from various ways to trigger retraining to automated approaches to deploy retrained models. However, retraining is implicitly considered to be from scratch, which makes most pipelines inherently inefficient. Simi- larly, other resources stating the importance of retraining ML models and efficient ML.Ops, at most very briefly consider other options than retraining from scratch [Kreuzberger et al. 2023 Komolafe 2023 Alla al. 2021
M1.0ps lifecycle
The efficiency that can be gained here represents an enormous opportunity for the continual learning field, which is clearly illustrated by Huyen
2022 from industry perspective.
3.5 Reinforcement learning
In reinforcement learning (RI.), agents learn by interacting with an environment. This creates a loop, where the agent takes an action within the environment, and receives from the environment an observation and a reward. The goal of the learning process is to learn a policy, i.e. strategy to choose the next action based the observations and rewards seen so far, which maximizes the rewards (Sutton & Barto 2018). Given that obscrvations and rewards are conditioned on the policy, this leads to a natural non-stationarity. where cach improvement step done on the policy can lead the agent to explore new parts of the environment. The implicit non stasitationarity of RL can be relaxed to a piece-wise stationary setting in off policy RL settings (Sutton
& Barto 2018), however this still implics continual learning problem. Offline RL (Levine et al. 22020) (c.g. imitation learning) completely decouples the policy used to collect data from the learning policy, leading static data distribution, though is not always applicable and lead to suboptimal solutions

---

Published in Transactions on Machine Learning Research Mlarch/20244
due to the inability of the agent to explore. Lastly, for real-world problems, the environment itself may be non-stationary, either intrinsically or through the actions of the agent.
The presence of non-stationarities in reinforcement learning makes efficient learning difficult. To accelerate learning, experience replay has been an essential part of reinforcement learning (Lin 1992 Mnih et al.
2015). While engaging new observations, previously encountered states and action pairs are replayed make training more ii.d. In contrast replay in supervised learning, in RL there is less focus on restricting the amount of stored examples, as the cost of obtaining them is considered very high. Instead the focus is how to select samples for replay (e.g. [Shaul et al. 2016) and how to create
experiences from stored
ones (Lin et al. (021). Additionally, loss of plasticity (e.g. Dohare et al. 2023 Lyle et al. 2022 inability of learning elficiently new lasks and formalizing the concept of continual learning (e.g. Kumar et al. 2023
Abel et al. 2023 also take a much more central role in the RL community.
Finally, besides the non-stationaritics encountered while learning a single task, agents are often required learn multiple lasks. This setting is an active area of research (Wotezyk et al. 2021 Kirkpatrick et al., 2017 Rolnick et al.] 2019), particularly since the external imposed non-stationarity allows the experimenter to control it and probe different aspects of the learning process. RL has its own specific problems with continual learning, c.g. trivially applying rehcarsal methods fails in the multi-task setting, and not all parts of the network should be regularized equally (Wolezyk et al.) (022). Issues considering the inability to learn continually versus the inability to explore an environment efficiently, as well as dealing with concepts like episodic and non-episodic RL, makes the study of continual learning in RL more challenging. Further research promises agents that train faster, learn multiple different tasks sequentially and effectively re-use knowledge from previous tasks to work faster and towards more complex goals.
Example: 3.5
Typical RL methods store millions more transitions in replay memory. Schaul jet al. (2016) showed that theoretically exponential training speed-ups are possible when cleverly selecting the transitions to replay. By approximating 'how much the model can learn' from a transition, they prioritize some samples over others and practically show linear speed-up compared uniform selection, the default at that point. Current state-of-the-art in the Atari-57 benchmark, MuZero (Schrittwieser al. 2020), relies on this prioritized selection and confirms its importance, yet. from the initial theoretical results, it is clear that improved continual learning could further improve convergence speeds and results (e.g. Pritzel al. 2017
4 Future directions for continual learning
In this section discuss interesting future directions for continual learning research, informed by what was discussed in the previous sections. We start by addressing the motivation for these rescarch directions, followed by brief overview of existing work, and finally justifying the importance of each concept.
4.1 Rethinking memory and compute assumptions
In all of the problems described in the previous section, optimizing or restricting compute complexity plays an important role, often more central one than memory capacity does. This is in stark contrast to the results in Section The vast majority of papers does not qualitatively approach compute complexity, while not storing, only very few, samples. Two popular reasons for arguing a low storage solution the cost of memory and privacy concerns, but these arguments are often not relevant in practice. Prabhu et al. (2023b) calculate that the price lo store ImageNet1K for one month is just 66e, while training a model on it requires 500S. This means storing the entire dataset for 63 years is as expensive as training ImageNet once. Further, privacy and copyright concerns are not solved by simply deleting data from the training set, as data can be recovered from derivative models (Haim et al.) 2022), and rulings to remove data might only be viable by re-training from scratch (Zhao 2022 (hence making continual learning superfluous), at least until reliable
8

---

Published in Transactions on Machine Learning Research Mlarch/2024/
KSTANH S [l+7) 1
FD 1: considering the current compute and memory assumptions
FD 2: better theoretic understanding of the limits and guarantees
FD 3: towards large scale continual learning and allowing small updates large models.
FD 4: integrating the labeling process and data acquiring steps into the CL-algorithm
Figure 2: An overview of the future directions (FD) discussed in Section
model editing exists (see Section 3.1. four of the five problems show, there many reasons to study algorithms that restrict computational cost just like restricted memory settings are studied today. We believe it is important to reconsider these common assumptions on memory and computational cost, and instead derive them from the real-world problems that continual algorithms aim to solve.
Section 3.3 showed that cases in low memory settings exist, but,
To achieve this goal, we should agroe on how to mcasure computational cost, which necessary to restrict it. Yet it is not straightforward to do so. Recent approaches the number of iterations (Prabhu et al. 2023a) and forward/bache bbckward passes (Kumari et al. 2022 Harun al. 2023c). which works well if the used model is exactly the same, but cannot capture architectural differences that influence computational cost. Similarly, when the number of parameters is used (Wang et al.) 2022a). more iterations or forward passes do not change the perceived cost. The number of floating point operations (FI.OPs) is often used to measure computational cost computer science, and is promising candidate, yet is sometimes hard to measure accurately (Wang et al. 2022b). In practice, limiting the size of the memory buffer can reduce the computational cost; it can for example reduce the convergence time, but it does not offer guarantees. Directly controlling the computational cost, rather than by proxy, can alter which algorithms perform best, e.g. in RL this is shown in Javed et al. 2202)) Additionally, time to convergence should also be considered, as faster convergence would also lower compute time Schwwrtz et al. (2020). To properly benchmark compute time and memory use in continual learning algorithins, we should build on this existing literature to attain strong standards for mcasuring both compute and memory cost and the improvements thereof.
As illustrated in Section there are works that have started to question our common assumptions continual learning (Harun et al. 2023b). SparCL optimizes compute time explicitly (Wang al. 2022b)) while (Prabhu et al. 2023b a) compare methods while constraining computational cost. Chavan et al. (023) establish DER Yan et al. 2021
promising method when compute complexity is constrained.
while other works suggest that experience replay is likely most efficient (Harun et al. 2023c Prabhu et al.
2023a). These early works have laid the groundwork, and we believe that it is in continual learning's best
interest to push further in this direction, and develop strategies for learning under a tight compute budget, with and especially without memory constraints.
4.2 Theory
In the past, continual learning rescarch has achieved interesting empirical results. In contrast to classic machine learning, not much is known about whether and under which conditions, we can expect results. Many theoretical results rely on the i.i. assumption, among which the convergence of stochastic gradient descent and the difference between expected and empirical risk in many PAC-bound analyses (although there are some exceptions, e.g. |Pentina & Lampert 2014). Crucially, the i..d. assumption is almost always broken in continual learning, illustrated by the problems in Section g To certain extent this also happens
9

---

Published in Transactions on Machine Learning Research Mlarch/2024]
in training with very large datasets, due to the computational cost of sampling data batches in an i.d.. fashion compared to ingesting them in fixed but random order. Not having theoretical guarantees means that continual learning research is often shooting in the dark, hoping to solve a problem that we do not know is solvable in the first place.
To understand when and under which assumptions we can find solutions, new concepts a number of directions need be developed in order to theoretically grasp continual learning in its full breadth. A key aspect is optimization. In which sense and under which conditions do continual learning algorithms converge to stable solutions? And what kind of generalization can expect? We want to emphasize that we should not be misguided by classical notions of those concepts. It might be, for instance, more insightful to think of continual learning as tracking a time-varying target when reasoning about convergence (e.g. Abel et al. 2023). Algorithms designed from this perspective can outperform solutions that are converged in the classical sense, a result of acknowledging the temporal coherence of our world (Sutton et al.) 2007 Silver et al. 2008). They show the potential for continual learning algorithms to improve over static systems, although that does not mean more classical, statie notions of convergence are not useful, as illustrated by Ziiin & Lampper (2019). Even if it is possible to find good solution, it is unclear whether this is achievable in reasonable time, and crucially, whether it can be more efficient than re-training from scratch. Knoblauch et al. (2020) show that even in ideal settings continual learning is No-complete, yet MMizadeh et al. (2021) empirically illustrate that often there are linear low-loss paths to the solution, reassuring that solutions that are easy to find are not unlikely to exist.
Not all continual learning is equally difficult. An important factor is the relatedness of old and new data. In domain adaptation, DDvid et al. (2010) have shown that without assumptions the data, some adaptation tasks are simply impossible. Empirically ŞZaiir et al. 2018 and to some extent theoretically (Prado & Riddle 2022), we know that in many cases transfer is successful because most tasks are related. Similar results for continual learning are scarce. Besides data similarity, the problem setting is an important second facot. For instance, class incremental lcarning is much harder than its task-incremental counterpart, as it additionally requires the predictions of task-identities (van de Ven el al., 2022: Kim et al. 2022). We believe that understanding the difficulty of a problem and having formal tools expressive enough describe understand relatedness between natural data will allow more principled approach, and better guarantees possible results.
Finally, theory in continual learning might simply be necessary to deploy continual learning models in a trustworthy manner. It requires models to be certified (Huang et al., 2020), i.e. they need to be thoroughly tested before deployment to work as intended. It is however unclear how this would fare in a continual learning setting, as by design, such models will be updated after deployment.
4.3 Lcrge-scale continual learning
Most of the problems in Section 3 start when there is a change in the environment of the model and it needs to be updated. These changes are often are small compared to the proceding training. The initial models, often referred to as foundation models, are typically powerful generalist models that can perform well on various downstream tasks, e.g. Oquab et al. (2023): Radford et al. (2021). However, performance gains are generally seen when adapting these models to specific tasks or environments, which compromises the initial knowledge in the pretrained model. In a contimuously evolving world, one would expect that this knowledge is subject to be continuous editing, updating, and expansion, without losses in performance. When investigating continual learning that starts with large-scale protrained models, the challenges might differ from those encountered in continual learning from random initializations and smaller models.
In contrast to smaller models, the required adjustments to accommodate new tasks usually limited com- pared to the initial training phase, which may result in forgetting being less pronounced than previously anticipated. It is an open questions which continual learning techniques are more effective in such a case. For example, Xiang et al. (2023) suggest that parameter regularization mechanisms (Kirkpatrick et al., 2017) are more effective than functional rogularization (o.g. distillation approaches (Li & Hoiem 2017]) in reducing forgetting in a large language model. Additionally, it might not be necessary to update all pa- rameters, which in itself is computationally expensive for large models. Approaches considering adapters
10

---

Published in Transactions on Machine Learning Research (March/202)
2019 Jia et al. 2022 Li & Liang 2021 J, low rank updates (Hu et al. 2022a or prompt- [Houlsby et al. ing (Jung et al. 2023). are argued to be more feasible in this setting. Freezing, or using non-uniform learning rates, might be necessary when the amount of data is limited to prevent optimization and overfitting issues, or to improve the implicit bias of a model when faced with a new data (Sutton 1992) How to adapt mod- els if the required changes are comparatively small to the original training remains an interesting research
direction, with promising initial results (Wang et al. 2022c Li et al., 2023 Panos et al., 2023
Lastly, in the large-scale learning setting there is paradigm shift from end-to-end learning towards more modular approaches, where different components are first trained and then stitched together. It is somewhat of an open question of what implication this has for continual learning (Ostapenko et al. 2022 Cossu et al. 2022). In the simplest scenario, one could decouple the learning of a representation, done with e.g. contrastive unsupervised learning, versus that of classifier with supervision (e.g. [llayrac et al.) 2022). ) Yet this idea can be extended towards using multiple (e.g. domain specific) experts (Ramesh & Chaudhari 2021) and using more than one modality (e.g. vision and speech) (Radford et al. (021). A better understanding of how continual learning algorithms can exploit. these setting is required to expand beyond the end-to-end paradigms currently used.
While there has been promising research in these directions, we believe that considerably more is needed. So far, we do not have a strong understanding of the possibilities and limits of small updates on large pre- trained models, and how the training dynamics are different than the smaller-scale models typically used in continual learning. Further research in the relation between new data and pre-training data might open up new opportunities to more effectively apply these smaller updates, and will ultimately make continual learning more effective in handling all sorts of changes in data distributions. Understanding the interplay between memory and learning, and how to exploit the modular structure of this large model could enable specific ways to address the continual learning problem.
4.4 Continual learning in real-world environment
Continual learning, in its predominant form, is centered around effective and resource-efficiet accumulation of knowledge. The problem description typically starts whenever there is some form of new data available, see Section 3 How the data is produced is a question that is much less considered continual learning. We want to emphasize that there is a considerable overlap between machine learning subfields (Mundt et al.) 2022). in particular in those that are concerned with both detecting change in data distributions and techniques that roduce the required effort in labeling data. It will be important to develop continual learning algorithms with these in mind. These fields depend on and need each other to solve real-world problems, making it crucial that their desiderata align.
Open world learning [Bendale & Boult learning focused on detecting novel classes of objects that were not seen during training, relaxing the typical closed-world assumption in machine learning. A first step to realize open-world learning detecting a change in incoming data, more formally known dustt--distiibution (OOD) or novelty detection. Detecting such of a model, i.. it should quantify what it does and
2015 is such a closely related field. Early work on open-world
changes requires certain level of mncetainty-awareen does not know. This uncertainty can be split into aleatoric uncertainty, which is an irreducible property of the data itself, and epistemic uncertainty, a result of the model not having learned enough (Hüllermeier & Waegeman 2021). When modeled right, the latter can provide a valuable signal to identify what should be changed in continually trained models (Ebrahimi et al. 2020). Alternatively, it provides a theoretically grounded way for active learning, which studics how solect the most efficient unlabeled data points for labeling (Settles] 2009 Nguyen et al. 2022).
Even when OOD data is properly detected, it might not be directly usable. It can be unlabeled, without sufficient meta-data, or in the worst case corrupted. Many CL algorithms require the new data to be labeled before training, which is always costly and often difficult in e.g. on-device applications. This process makes it likely that when solving problems as described in Section D a model has access to a set of unlabeled data, possibly extended by some labeled samples that are obtained using active learning tochniques. To successfully work in such an environment, a model should be able to update itself in self- or semi-supervised way, an idea recently explored in Fini et al. (2022).
11

---

Published in Transactions on Machine Learning Research MMarch/20244
Continual learning depends on the data available to update model. It is thus important to develop CL algorithms that are well calibrated, capable of OOD detection and learning in an open world. Further, in many settings (see Section 3.3), new data will not, or only partly, be labeled, which requires semi- or self- supervised contimual learning (Mundt et al. 2023). We recommend working towards future continual learning algorithms with these considerations mind, as methods that rely less on the fully labeled, closed-world assumption will likely be more practically usable in the future.
5 Conclusion
In this work, we first examined the current continual learning field, and showed that many papers study the memory-restricted setting with little concern for the computational cost. The problems we introduced all require some form of contimual learning, not because it is nice-to-hawe to-have, but because the solution inherently depends on continual learning. Finally, we established four rescarch directions in continual learning that we find promising, in the light of the scenarios we described. In summary, many of these applications are more compute-restzicten than memory-restricted, so we vouch for exploring this setting more. Further, we believe a better theoretical understanding, larger focus on pre-training and comparatively small future updates, and greater attention how data attained, will help us solving these problems, and make continual learning a practically useful tool to solve the described and other machine learning problems. A summary of the talks and discussions at the Deep Continual Learning seminar in Dagstuhl that inspired this paper can be found in Tuytelaars et al. (002)).
Broader impact
This paper does not present any new algorithm or dataset, hence the potential direct socictal and ethical implications are rather limited. However, continual learning and applications thereof, as we have examined, may have a long-term impact. Reducing computational cost can positively affect the environmental impact machine learning has. Easily editable networks, or ways to quickly update parts of networks as discussod in Section 3.1 and 3.4 may further democratize the training of machine learning model. Yet this also means that it be exploited by malicious actors to purposely inject false information in network. Predictions made by those networks could misinform people or lead to harmful decisions. Excessive personalization described in Section 3.2 may negatively impact community solidarity, yet benefit the individual.
References
David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition of continual reinforcement learning. arXiv preprint ar Xrxi:22077.1 11046, 2023.
Jean-Baptiste Alayrac, Jelf Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Xii preprint arXiv:2204.14196 2022.
Sridhar Alla, Suman Kalyan Adari, Sridhar Alla, and Suman Kalyan Adari What is MT.OPs? Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker, Google Cloud, and Microsoft Azure, pp. 79-124, 2021.
Dario Amodei and Danny Hernandez. Al and compute. httppp//ppgaaaa.ccm/eeeeaaaaaaaa 2018. Online; accessed 20-June-200 2023.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural Information Processing Systems, 33:3884-388 3894, 2020.
Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arviv preprint 1rXi::1903. 10676, 2019.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1893 1893-1902, 1902, 2015.
12

---

Published in Transactions on Machine Learning Research MMarch/2024
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmaal International Conference. on Machine Learning, pp. 41-48, 2009.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. or preprint arXiv:2106.00044, 2021.
Vivek Chavan, Paul Koch, Marian Schlüter, and Clemens Briese. Towards realistic evaluation of industrial continual learning scenarios with an emphasis on energy consumption and computational footprint. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11506 -1551,, 2023.
Niv Cohen, Rinon Gal, Eli Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language reprosentations. In European Conference on Computer Vision, pp. 558 577. Springer, 2022.
Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. arXiv preprint S5w:2205.09355 2022.
William Dally, On the model of computation: point. Communications of the ACM, 65 32, 2022.
Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. Tn Proceedings of the Thirleenth International Conference on Artificial Intelligence and Statistics, pp-
129 136. JMLR Workshop and Conference Proceedings, 2010.
Saket Dingliwal, Monica Sunkara, Srikanth Ronanki, Jeff Farris, Katrin Kirchhoff, and Sravan Bodapati. Personalization of CTC speech recognition models. In 2022 IEEE Spoken, Language Technology Workshop (SLT), pp. 302-30-0000 309. JEEE, 2023.
Shibhansh Dohare, Juan Sannandez-Garciaa Parash Rahman, Richard Sutton, and Rupam Mahmood. Loss of plasticity in deep continual learning. Research Square preprinl PPR: PPR727015, 2023. doi: 10.21203/ rs.3.rs- ts.3.rs-32564790 325647//v1.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guidee continual learning with bayesian neural networks. International Conference on Learning Representations, 2020.
European Union Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec /ec (general data protection regulation). Official Journal L110, 59:1-88, 2016.
Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda- Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models continual learners. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 9621-9630, 2022.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):
128-135, 1999.
Haotian Fu, Shangqun Yu, Michael Littman, and George Konidaris. Model-based lifelong reinforcement learning with bayesian exploration. Advances in Neural Information Processing Systems, 35:32369-322 -23882, 2022.
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation online continual learning: A new hope. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11888 11897, 2023.
13

---

Published in Transactions on Machine Learning Research March/2024
Eileen end up on roomba- facebook? accessed 11-October-2023.
Guo. A roomba recorded
woman the toilet. how did screenshots
https: http://wwcoogggeereeee.oo/////// 12666///////1
2022. Online;
Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3950-3960. PMLR, 2020.
Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12))1028 1048,,000000000000 2020.
Niv Haim, Gal Vardi, Gilad Ychudai, Ohad Shamir, and Michal Trani. Reconstructing training data from trained neural networks. Advances in Neural Information Processing Systems, 35:22911 22924, 2022.
Md Yousuf Harun, Jhair Gallardo, Tyler L. Hayes, and Christopher Kanan. How efficient are today's continual learning algorithms? In Proceedings of the IEEE/CVI Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2431 2436, June 2023a.
Md Yousuf Harun, Jhair Gallardo, Tyler L Hayes, Ronald Kemker, and Christopher Kanan. SIESTA: Efficient online continual learning with sleep. Transactions on Machine Learning Reseurch, 2023b.
Md Yousuf Harun, Thair Gallardo, and Christopher Kanan. GRASP: A rehearsal policy for efficient online continual learning. arXiv preprint arXie:2308.1300 2023c.
Tyler L Hayes and Christopher Kanan. Online continual learning for embedded devices. In Conference on Lifelong Learning Agents, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Paramcter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Represenlations, 2022a. URL https::/ wtt.://wwweee.ooo//e//eeeeeeeeee
Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Drinking from firehose: Continual learning with web-scale natural language. IEFF Transactions on Pattern Analysis and Machine Intelligence, 45(5): 5684 5696, 2022b.
Xiaowei Huang, Daniel Krocning, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi. A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability. Computer Science Review, 37:100270, 2020.
E. Hüllermeier and W. Wacgeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457 506, 2021. doi:
Chip Huyen. Real-time machine learning: challenges and solutions, Jan 2022. URL https:
hew.aaaeeeemcccci--ee-aaaaaaaaaa wwwrcchaallllaae aenlllnnee-n----ee anddsoluuionn html# Sowards-conntima---eeerre Online; accessed 14-November-2023.
Khurram Javed, Haseeb Shah, Richard S Sutton, and Martha White. Scalable real-time recurrent learning using columnar-constrrutive networks. Journal of Machine Learning Research, 24:1-34, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardic, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In Computer Vision-ECCT 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXXIII, pp. 709 727. Springer, 2022.
14

---

Published in Transactions on Machine Learning Research Mlarch/20244
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal-fiee continual learning. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 11847 11857, 2023.
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. Advances in Newral Information Processing Systems, 34, 2021.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. Adapting language model while preserving its general knowledge. In Proceedings of The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP-2022), 2022.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Kc, and Bing Liu. A thcorctical study on solving continual learning. In Advances in Neural Information Processing Systems, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwingg et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521133 (111333622-336,,, 2017.
Varsha Kishore, Chao Wan, Justin Lovelace, Yoav Artzi, and Kilian Q Weinberger. Incdsi: incrementally updatable document retrieval. In International Conference on Machine Learning, pp. 17122-17134. PMI.R,
2023.
Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is up-hard. In International Conference on Machine Learning, pp. 5327 5337. PMLR, 2020.
Akinwande Komolafe.
Retraining
model
during
deployment:
Continuous
training and continuous testing. retraining-mont sasg--moel-ddaee www.araawa.a.aaaaaaaaoooooooooaa hrgp:aainnn.oottiimooo-ttttttttt accessed mt.Jume-2023
2023.
URL https://neptune httpppuueeaa/bllg.... ai/blog/
Online;
Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl, Machine learning operations (MLOPS): Overview,
definition, and architecture. IEEE Access, 2023.
Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, al. Biological underpin- nings for lifelong learning machines. Nalure Machine Intelligence, 4(3):196 210, 2022.
Dhireesha Kudithipudi, Anurag Daram, Abdullah Zyarah, Fatima tuz Zohora, James B. Aimone, Angel Yanguas- Gil, Nicholas Soures, Emre Neftei, Matthew Mattina, Vincenzo Lomonaco, Clare D. Thiem, and Benjamin Epstein. Uncovering design principles for lifelong learning ai accelerators. Nature Flectronics (Final Revisions), 2023.
Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, and Benjamin Van Roy. Continual learning as computationally constrained reinforcement learning. arXiv preprint arXiv:2307.0.4 044344, 2023.
Lilly Kumari, Shengjie Wang, Tianyi Zhou, and Jeff A Bilmos. Retrospective adversarial replay for continual learning. Advances in Neural Informalion Processing Systems, 35:28530-28544 2022.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature commanications, 10(1):1096, 2019.
Angeliki Lazaridou, Adhi Kuncoro, Flena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' A'Attueee Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Newral Information Processing Systems, 34:29348-29388 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Xix preprint 2rXi::2005.01643, 2020.
15

---

Published in Transactions on Machine Learning Research Mlarch/20244
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. orXiv preprint MXXi:22011001190 2021.
Zhizhong and Derek Hoiem. Learning without forgetting. IEEE transactions on pollern analysis and machine intelligence, 40(12):2935 2947, 2017.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototype with prompt-tunnng for rehearsal-free continual learning. arXiv preprint (xXix2230300447 2023.
Junfan Lin, Zhongzhan Huang, Keze Wang, Xiaodan Liang, Weiwei Chen, and Liang Lin. Continuous transi- tion: Improving sample efficiency for continuous control problems via mixup. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 9490 9497. LEEE, 2021.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293 321, 1992.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In International Conference on Machine Learning, pp. 14560-14581. PMLR, 2022.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-inerementa learning: survey and performance evaluation on image classification. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2022.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascamu, and Hassan Ghasemzadeh. Lin- mode connectivity in multitask and contimual learning. In International Conference Learning Represenlations, 2021. URL htttps::/openreevie http:///ppaaeeeeee...eemooomaaaa fQYUeff
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing scale. In International Conference on Learning Representations, 2022. URL https::/openreevie.
het/fforumiid=DDczeeektttt
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting, CLEVA-compass A continual learn- ing evaluation assessment compass to promote rescarch transparency and comparability. International Conference on Learning Representations, 2022.
Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Newral Networks, 160:306 336, 2023.
V.L. Nguyen, M.H. Shaker, and E. Hüllermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1))80-122, 2022. doi:
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint NKKv:2334.07100 2023.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez, Md Rifat Arefin, Arthur Douillard, Trina Rish, and Laurent Charlin. Foundational models for continual learning: An empirical study of latent replay, 2022.
URL
Aristcidis Panos, Yuriko Kobe, Daniel Olmoda Reino, Rahaf Aljundi, and Richard E Turner. First scssion adaptation: A strong replay-free baseline for clas--nccrmmnntl learning. Xi preprint orXiv:2303.14 13199, 2023.
16

---

Published in Transactions on Machine Learning Research MMarch/20244
Anastasia Pentina and Christoph H. Lampert. A PAC-bayesian bound for lifelong learning. In JCML, 2014.
Iuliia Pliushch, Martin Mundt, Nicolas Lupp, and Visvanathan Ramesh. When Deep Classifiers Agree: Analyzing Correlations Between Learning Order and Image Statisties. European Conference on Computer Vision (ECCV), pp. 397-413, 2022.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3698-3 3707, 2023a.
Amcya Prabhu, Zhipeng Cai, Puncet Dokania, Philip Torr, Vladlen Koltun, and Ozan Scner. Online con- tinual learning without the storage constraint. arxiv preprinl urXiv:2305.0922 2023b.
Diana Benavides Prado and Patricia Riddle. A theory for knowledge transfer in continual learning. In Conference Lifelong Learning Agenls, 2022.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Has- sabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pp. 2827-2836. PMLR, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International Conference on Mochine Learning, pp. 8748- 8768. PMLR, 2021.
Rahul Ramesh and Pratik Chaudhari. Model zoo: growing' brain' that learns continually. arXio preprint arXiv:2106.0300 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual lcarning. Advances in Neural Information Processing Systems, 32, 2019.
Khalid Salama, Jarek Kazmierczak, and Donna Schut. Practitioners guide MLOPS: A framework for continuous delivery and automation of machine learning. Google Could White paper, 2021.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting ils prediction rules. Advances Neural Information Processing Systems, 34:23359 23373, 2021.
Tom Schaul, John Quan andloannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Gracpel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 6044666,, 2020.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green AI. Communications of the ACM, 63 (12):54 -3,, 2020.
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin-Madison, 2009.
David Silver, Richard S Sutton, and Martin Müller, Sample-based learning and search with permanent and transient memorics. In Proceedings of the 25th International Conference on Machine learning, pp. 968-975, 2008
Richard S Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI, volume 92, pp. 171-176. San Jose, CA, 1992.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://wmm...eeeaeaaaeeeeeaaaaaa
html
17

---

Published in Transactions on Machine Learning Research (March/202)
Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. In Proceedings of the 24th International Conference on Machine learning, pp. 871-878, 878, 2007.
Timne Tuytelaars, Bing Liu, Vincenzo Lomonaco, Gido van de Ven, and Andrea Cossu. Deep Continual Learning (Dagstuhl Seminar 23122). Dagsluhl Reports, 13(3):74 91, 2023. ISSN 2192-5283. doi: 10.4230/ DagRep.13.3.74 URL https:/ /aaa:://drrppp dasessdaassshhcocooo
http:///ccccmeett/o////////ggga/ 4230//agRep....... 13.3. 74
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185 1185-11977 2022.
Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than big one. In European Conference on Compuler Vision, pp. 254 271. Springer, 2022a.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00433 00487, 2023.
Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis loannidis, Yanzhi Wang, and Jennifer Dy. SparCL: Sparse continual learning on the edge. Advances in Neural Information Processing Systems, 35:20366 20380, 2022b.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perol, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 139 199, 2022c.
Maciej Wolezyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciński, and Piotr Milos. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Sys- tems, 34:28496-288 28510, 2021.
Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciúskii and Piotr Miloš. Disentangling transfer in continual reinforcement learning. Advonces in Neural Information Processing Systems, 35:6304-6317,
2022.
Mitchell Wortsman, Gabriel Ihharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zcro-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7959-7971, 7971, 2022.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. Language mod- els meet world models: Embodied experiences enhance language models. arXiv preprinl arXiv:2305.1060 2023.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremen- tal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014 3023, 2021.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEFE conference on Computer Vision and Pattern Recognétion, pp. 3712-3722, 2018.
Zeyu Zhao. The application of the right be forgotten in the machine learning context: From the perspective of european laws. Cath. UJL & Tech, 31:73, 2022.
Da-Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class- incremental learning: A survey. arXiv preprint arXiv:2302.0300 2023.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76 2020.
Alexander Zimin and Christoph H. Lampert. Tasks without borders: A new approach to online multi-task learning. In ICML Workshop on Adaptive & Multitosk Learning, 2019. URL https://openreviiet net/
forum?idh 1V5B241
18

---

Published in Transactions on Machine Learning Research Alarch/2024/
A Details of the analysis in Section 2
To verify the keywords 'incremental', 'continual", 'forgetting', lifelong' and 'catastrophic', used to filter the papers based on their titles, we tested them using a manually collected validation set of which we are certain that they are continual learning related. This set was manually collected while doing research on continual learning over the past few years. The keywords were present in 96% of the paper titles. From each conference, we randomly picked up 20 of all matched papers, disregarding false positives.
It is common for to evaluate the percentage of stored samples is not uniform across the experiments in paper. In Figure D we showed the minimum percentage used, in Figure 3 we show the maximum. The conclusion remains the same, and the amount of stored samples is constrained in all but two benchmarks.
methods and analyses
than benchmark. Often this means that
Table | we provide a table of all the papers we used the analysis of Section 2 showing their minimal and maximal sample store ratio (SSR) i.e. the percentage of samples stored, as well as possibly other memory consumption. The last column mentions how they approached the computational cost.
15
11
Optimized-
Comgatational cost
EOCV 122
0:0
Compared
0-0
JCME
diserssadd
Percentage of stored data
Figure Most papers strongly restrict memory and do not discuss computational cost. This figure is an alternate version of Figure D with the maximum percentage of stored samples rather than the minimum. Each dot represents one paper, illustrating what percenlage of data their methods store (horizontal axis) and how computational complexity is handled (vertical axis). The majority of examined papers are in the lower-left corner: those that strongly restrict memory use and do not quantitatively approach computational cost
19

---

Published in Transactions on Machine Learning Research (March/ /2024)
SSR refers to the sample store ratio, i.e. how much samples are stored in relation to the
Computational Cost
discussed not discused
not discused discussed
discussdd not discussed not discussed not discused
not discussed not discussed
discussed
not discusse
nor discussed not discused
not discussed diseussrd
not discussed
not discossed
not discused
discussed
not discussed
not discussed
not discussed discussse
not. discussed
not discussed discuseed
constrained
discussed
discussd
comp pared
constraized diserusse
comparod parrod
discussed
compared
discussed
discuseed
comm parred
Memory (other)
gradient bases
prototypes model copies
prototypes model coples
prompts model copies
generators model copies
model copies
modol coples
model. copies
model copies
Null spacess
prototypes prototypes
generstors
generators
prototypes
/
SSR (max)
0.2
0.n4
0.04
0.02
0.04
0.001
0.002
0.1
0.001
0.004
0.1
0.01
0.09
011
0.02
0.01
0.06
SSR (min)
0.02
0,01
0.001
0.001
0.001
0.002
0.02
0.001
0.1
0.001
0.001
0.006
0.003
0.002
0.01
0.01
0
Balancing Stability And Plasticity Through Advancei Null Spare In Continual Learning
Prutolype- Guudeed Guided Continual Adaptation For Class Increnental Unsupervised Do- main Adaptation
Few- Shot Claes-Incremente Leaening Vla Entropy- -Reguaarized Data-Free Replay
Dieell Relation- Guided Representation Learning For Data-Fred Froe Class Incremen-
Continual Somantic Segmentation Via Structure Preserving And Projected Feature
Balancing Between Forgetting And Acquisition Increnental Subpopalation
Mets- Learning With Less Forgetting On Large Scale Non- Stationary Tack Distri-
Continal Variational Autoemonde Learning Via Online Cooperative Memoriza-
Duslprompt: Complementary Prompting For Rehearzal Free Continuel Learuing tion
Semantic Segmentation with
n: a. Bilevvl Momory Frameworkk with Knowledge Projection for Task- Task-Iucrenne eok--ucreeeette Learning
Geometry and Uncertainty- Aware Aware 3D Point Cloud Class- Incrementl Semantic Segmentation.
Task Difficulty Aware Parameter Allocation & Regularization laliiiiiioon for Lifelong Learning Computationalyy Budgeted Continual Learning: What Does Matter? CoMFormer: Continual Learning in Somantic and Panoptic Sogmentattion
POR: Proxy- based Contrastive: Replay for Online Glass- Incremental Continual Learning
AMtrCLIP: A Non- wweremeereette Learner for Incremental Know Learning Leaning with Fantasy: Semantic Aware Virtual Contrastive Constraint for Few- Shot Mlas-Increinnn
the Stability- Plasticity Dilemus Class- Incremental Learning MetaMix: Towards Corruption Rabust Continall Learning with Temporally Self.
Dieft: Doep Linear Continual Fine-Tun Tuning For General Incr tncremetaal Learning Generative Negative Text Replay For Continual Vislou-Lauue La..uage Protrsining Online Coutinal Learning With Contractive V iision Transfur mer
Cooperation Of Small Continual Learners Stronger Than Big One
Contitual Semantic Segmentation with wilh Autonallc Li Memoory Sample Selection Adantive Pladiciity Improvement
TheeBeae Blased Coutext In Coutinual Semaatic Segneutation Course-Ib- Fine incremental Pbw Leurning
VQACL: A Novel Visual Question Answoring Continual Learning Setzing
Class: Intremental Exemplar Compression for Class Increnental Learning
Fow- Shot Inse-Incrementall Loarning For 3d Point Cloud Objects
Real- Reae-Time Time Evalustion in Online Continual Learning: A New Hope Heterngenenee Continual Learning
Coutinual Detoction Trensformer Incromentall Objoct Detection
Continual Learning
Large-Scale
Reli-Reeroaccivee Interference For Lifelong Learinn
PIVOT: Prompting for Video Continual Leacning
Table 1: All papers used the examination of Section 2 entire dataset.
Incrementer: Transformer for Slass-Ineremes
Knowlodge Distillation Focusing on Old Class
Novel Class Discovery Without Forgotting
Claes-lecrementa Novel Class Discovery
Long- Tailed Class Incrmmental Learning
Learniog
Adaptive Data Transfomaaton
Docoupling Learning and
Rbe: Rectifylng The
tal Learning
Algament
Loaring
butions
Title
Coscl:
Conference
BCCV
BCCV
BOCV
BCCV
BCCV
BCOV
BCCV
BCCV
BCCV
ECCV
BCCV
BCCV
BCCV
ECCV
BCCV
BCCV
BCCV
BCCV
BCCV
BCCV
CVPR
CVPR
CVPR
CVPR
CVPR.
CVPR
CVPR
CVPR
CVPR
CVPR
CVPR
CVPR
CVPR
34 CVPR
CVPR.
CVPR
CVPR
CVPR
CVPR
#
4
7
9
11
12
15
17
19
20
21
22
23
24
25
26
27
29
30
31
32
36
37
38
39
20

---

Published in Transactions on Machine Learning Research (March/ /2024)
not discussed not discussed
not. discussed
not discussdd
not discresed compared
discussed
discusse
not discused discussed
not discussed
not discussed
comn camporee not discussed discussed diacussed compured comm comparoo pampar
discusseed nor. discussd discussed
not discussed
not discussed
discussed
nor discussed
not discussed
not discussed
constrained
discussed
discussed
optimized comperod
discussed
compared
comn compared discneeed
optimised
com comparce
gradient bases
extrs modules prototypes
extra modules
model capies
model copies
model copies
model coples
prototypes
generators
generators
/
0.045
0.04
0.03
0.5
0.04
0.01
0.1
0.04
0.2
0.1
0.5
0.04
0.2
0.16
0.1
0.04
0.04
u.04
0.16
0.1
0.05
0
0.004
0.001
0.5
0.008
0.004
0.004
0.01
0.004
0.04
0.1
0.5
0.01
0.004
0.1
0.01
0.02
0.003
0.02
0.002
0
0
0
Table 1 contimued from previous
Uncertainty- Aware Aware Hieearchical Reftuement for Incremental Implictlyy R--Rfeedd Classiftcation
Loarning Condensed Frame for Memory- Elffeient Video Insrrmmenta Learn-
ing Prompts Leariing with Prestrained Transformen An Ozcam's 's Razor for Da- main incremental Learning
Navigating Memory Construction by Global Pseudo- Taek Simulation for Continual
simple strong baseline online contintal learning Repeated Augoented
Liislong Noural Predictive Coding: Leatming Commlattvelly Online without For-
ALIFE; Adaptive Logit Regulariinn aigiarrierrr and Foature Replay for Incremental Semantic Segmeutation
ACIL: Analytic Mlss--nnrremnnall Learning with Absolute Memorizationn and Pri- Retrospective
Surr-SSymbllc Continnal Loarning: Knowledge, Rensoning Shortoats And Con- cept Rehearsel
Do You Remember? Overcoming Calastropic: Porgetting For Pale Audio Deter-
Continual Vision- Language Representatin Learingg Wilh, Oil- oil-Diagunal Informa- tion
Prototype- Saanpy Sample Relation Distillation Towards Repley- Free Continual Learring ter-
NOTE: Robust Continual Test- time Adaptation Against Temporal Correlation
Decomposed Kkowleedge Distillation for Class- Increnenettlll Semantic Segmentation
Beyond Not- Forgetttg Coutinual Learning with Backward Knowledge Transfe
Paisoning Generativn Replay In Continall Learning Promote Forgetting Dualbsic: Hsie- Bottleneek And Alignment For Continual Learning
Thskk Free Continnal Learning vin Online Discrepancy Distance Learning Disenranginng Trausfor Continual Reinforeement Learning Leas- fen-furggetingg Multi- ling ual Fine- Luning
Model based Liislong Reinforenmen Learning with Bayesian Exploration
Ddgr: Continual Learning With Deep Diffusion- Bosed Generative Replay
Birt: Bie Inspiced Replay In Visian Transformen For Continual Learning Optimizing Mode Connectiviny For Class Incremental Learning Coutinual Learners Are Incremental Model Generalizers
Does Continull Learning Equally Forget All Farameters? Lifslong Language Protraining minrainng With Distribotion-Se Spocialized Experts Continusl Task Allocation In Meta-Polies Network Via Sparse Prompting
Adversarial Replay for Continual Learning
Memory Elficient Concitual Leacuing with Transforners
Paraneter Seremnetr--Leel Level Soft- Soft-Maekina Macking For Continual Learaing
A Theoretical Study on Solving Continual Learning gotting
Learnabiltyy And Algoritha For Continual Learning
Inedai: Incrementally Updatalaee Document Retrieval
Exploring Data Goometry for Continual Learning
Adaptive Compositionss Continual Meta- Learning
Flw Pew- Shot Continual Active Learning by a Rohot
SparCL: Sparse Contimal Learning on the Edge
vacy Protoction
Learning
Rehearsal
tion
CVPR NeurIPS
Nour
NeurlPS
NeurIPS
NearPPS
NeurIPS
NeurIPS
NeurlPS
NourIPS
NearlPS
NPu
NS
NeurIPS
NownIPS
NeurlPS
NearIPS
NeurTPS
NeurlPS
NeurIPS
NeurIPS
IGML
ICML
1CML
ICML
IOML
ICML
1CML.
ICML
JCML
ICML
ICML
ICML
ICML
ICML
ICMI.
IOML
1CML
40
41
46
47
48
52
53
54
55
67
58
59
60
62
63
64
65
66
68
69
70
73
74
75
76
77
21


Reviews:
[{"review_id": "8CYqPAWryp", "reviewer": "Reviewer_tRJw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThis is a much needed paper for the field of continual learning. A large part of the CL community is plagued by a hand-crafted problem setting---memory constrained supervised learning---that only covers a small aspect of the need for continual learning, and this paper makes a clear and concise case for why CL has a much wider appeal.\n\nI also wholly agree with the authors that the motivations used by the current supervised CL community---privacy and memory constrained---are not well thought out, and the meat of the need for CL lies elsewhere---adapting to distribution shifts, adding and removing knowledge, and specializing the models to their environment at the time of deployment, etc. \n\nI would strongly recommend the paper to be accepted.\n\n## Small gripes\n\n1. I think the paper can emphasize some points better. More specifically, prior work by Silver, Koop, Müller, and Sutton has empirically shown that continually learning systems can out-perform fixed systems, laying a strong case for CL. A discussion on them would be useful. The two relevant papers are: \n\ni). Sutton, Richard S., Anna Koop, and David Silver. \"On the role of tracking in stationary environments.\" Proceedings of the 24th international conference on Machine learning. 2007.\nii) Silver, David, Richard S. Sutton, and Martin Müller. \"Sample-based learning and search with permanent and transient memories.\" Proceedings of the 25th international conference on Machine learning. 2008.\n\n\n2. There has been some work that empirically verifies the impact of computational constraints on online learning/lifelong learning systems. The following paper shows that under computational constraints of FLOPs-per-step, algorithms that are believed to be inferior can out-perform the popular algorithms (See Section 2 for problem setting and computational constraints): \n\nJaved, K., Shah, H., Sutton, R. S., & White, M. (2023). Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks. Journal of Machine Learning Research, 24, 1-34.\n\n3. \"Freezing, or using non-uniform learning rates, might also be necessary when data is limited to prevent optimization and overfitting issues\"\n\nUsing non-uniform learning rates can also help with better credit assignment for continual learning. Mainly, it can allow continual learning systems to learn to preserve some knowledge, and overwrite other. The following paper touches on these ideas:\n\nSutton, R. S. (1992, July). Adapting bias by gradient descent: An incremental version of delta-bar-delta. In AAAI (Vol. 92, pp. 171-176).\n\nThere are other works with similar ideas as well, such as Stochastic Meta-Descent (Schraudolph 1999).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper argues that the field of continual learning is limiting itself to a few arbitrary use-cases, when in-fact continual learning has much wider applicability. The authors argue about multiple use-cases of CL that are ignored by most of the current research. These use-cases include (1) specializing the AI systems to their environment (as opposed to fixing them at the time of deployment), editing the knowledge of the systems when they make mistakes, (3) saving computation by not retraining models from scratch, (4) putting more emphasis on computational constraints, as opposed to memory constraints,  and tacking continual learning in a more principled way from the lens of theory.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would request the authors to read or skim the papers I listed, and support their arguments with the evidence in the papers (if possible). I'm not making any hard constraints, as the authors have already done an excellent job of supporting their arguments using existing literature, and I trust that they will do a good job including the papers I mentioned.", "broader_impact_concerns": "No comments", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YkeGjp3yCX", "reviewer": "Reviewer_A78Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Summarizing very recent papers and pointing out computational cost is much less explored in the literature.\n- Writing is mostly clear and easy to read\n\nWeaknesses\n\n- The number of surveyed papers is limited. The surveyed papers come from only three conferences (ECCV 22, NeurIPS 22 and CVPR 23), which is far from enough for a survey paper. It would be important to cover other important venues such as ICML and ICLR.\n- The categorization is unclear. As a survey paper, it is crucial to provide a clear categorization of existing methods, distinguishing the key differences and applicability of various algorithms, which is unfortunately missing for the current paper. For example, The title of Sec.3.1 is “Adapting machine learning models locally” but the content is more about correcting prior mistakes and there seems to be nothing specific about “locally”. Moreover, the abstract calls it “model-editing”, which overlaps with Sec.3.2 personalization as one may want to adjust the model for the purpose of personalization/adaptation.\n- Some motivations are not well connected to continual learning. For example, Sec.3.2 is more related to transfer learning (one-time; adapting a model learned from vast data to a specific domain) than continual learning (lifelong). “Further, Internet scraped data often do not contain (enough) information to reach the best performance in specialized application domains like science and user sentiment analysis.” I beg to disagree. The problem here is more about identifying the relevant information than not having enough information. There are several similar claims that are not substantiated by papers or experiments.\n- Only pointing out potential problems without providing enough solutions", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a brief survey of recent papers on continual learning (CL for short, a.k.a. lifelong learning), provides five motivating scenarios (error correction, personalization, on-device learning, model re-training and reinforcement learning) and finally identifies four possible future research directions (computational cost, theory, large-scale learning and open-world learning).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) Include more papers from recent years and from more venues\n\n(2) Provide a clear categorization\n\nMinor comments\n\n- 10.000 GPU -> 10,000 GPU\n- Figure 1: why use dots when a number suffices. If visualization is preferred, one can use a heat map.\n- make learning efficiently -> make learning efficient\n- Citation: use \\citet when the citation is part of the sentence and \\citep otherwise. For example, in Sec.3.5, “in off policy RL settings Sutton & Barto (2018)” should be “in off policy RL settings (Sutton & Barto 2018)”", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sIgVAFGz1v", "reviewer": "Reviewer_vbve", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThis is a plainly written survey paper without too much detail, so this article is very accessible to any person with some knowledge of machine learning.\n\nWeakness:\n\n1. This paper shows some surveys in quantitative method, i.e. Figure 1. However, the background context of the past research is not well discussed. For instance, author wrote \"strongly restrict memory and do not discuss computation cost\". There are valid reasons of such setting. For example, continual learning is often deployment-purposed, so they restrain the training with future memory usage. Subsequently, such training will take less amount of data, so the computational cost could be less important because of the reduced data to process compared to the main training data. These pros and cons are not discussed, and only authors view is prevalent. \n\nUnbiased survey is crucial in writing a survey paper.\n\n2. Only very limited survey is done with three conference events: ECCV 22, NIPS 22, CVPR 23. Now, we are at the end of 2023, so there will be plenty more events to cover. Moreover, why not include ICCV, ICML, etc? I would say that the survey should be exhaustive to be a published paper in TMLR.\n\n3. More indepth insight is needed. The key development of continual learning is not being covered in any technical depth. There is no trace of the loss function development, or problem setting evolution in the field of continual learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a survey paper to discuss the current state and future direction of continual learning. This paper presents the necessity of the continual learning; the past discussion points, such as memory usage and computation costs; and the future open problems.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the weakness", "broader_impact_concerns": "Not much of an ethical concern paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision to accept the paper is based on its substantial contribution to the discourse on continual learning in machine learning. The authors effectively identify and challenge the current limitations within the field, advocating for a better perspective on the applications and implications of CL. Their engagement with recent literature and incorporation of feedback from reviewers have strengthened the paper’s arguments.

While the paper may not provide an exhaustive survey of the field it is well-positioned to stimulate further discussion and research in the field, making it a valuable addition to the journal.

I strongly recommend authors to further incorporate suggestions from reviews in their revision and extend the scope of the paper with the work published during the review process.
Decision: Accept (poster)