Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling
Authors: Minhyuk Seo
Hyunseo Koh
Jonghyun Choi
Venue: ICLR 2025 Spotlight (2025)
Keywords: Continual Learning
Lifelong Learning
Efficient Training
Layer Freezing
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: The majority of online continual learning (CL) advocates single-epoch training and imposes restrictions on the size of replay memory. However, single-epoch training would incur a different amount of computations per CL algorithm, and the additional storage cost to store logit or model in addition to replay memory is largely ignored in calculating the storage budget. Arguing different computational and storage budgets hinder fair comparison among CL algorithms in practice, we propose to use floating point operations (FLOPs) and total memory size in Byte as a metric for computational and memory budgets, respectively, to compare and develop CL algorithms in the same ‘total resource budget.’ To improve a CL method in a limited total budget, we propose adaptive layer freezing that does not update the layers for less informative batches to reduce computational costs with a negligible loss of accuracy. In addition, we propose a memory retrieval method that allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations. Empirical validations on the CIFAR-10/100, CLEAR-10/100, and ImageNet-1K datasets demonstrate that the proposed approach outperforms the state-of-the-art methods within the same total budget. Furthermore, we validate its effectiveness in the Multi-modal Concept incremental Learning setup with multimodal large language models, such as LLaVA-1.5-7B. Code is available at https://github.com/snumprlab/budgeted-cl.

Published as a conference paper at ICLR 2025
BUDGETED ONLINE CONTINUAL LEARNING BY ADAPTIVE LAYER FREEZING AND FREQUENCY-BASED SAMPLING
Minhyuk Sea
Hyunsco Koh*
Jonghyun Choi!
Seoul National University
mi minbyakseo@yonsei..ce ac. khs81570 10881170gmai.... com, jonghyunchoiestu.ce ac kr
ABSTRACT
The majority of online continual learning (CL) advocates single-epoch training and imposes restrictions on the size of replay memory. However, single-epoch training would incur different amount of computations per CL. algorithm, and the additional storage cost to store logit or model in addition to replay memory is largely ignored in calculating the storage budget. Arguing different computational and storage budgets hinder fair comparison among CL algorithms in practice, we propose to use floating point operations (FLOPs) and total memory size in Byte as metric for computational and memory budgets, respectively, compare and de- velop CL algorithms in the same 'total resource budget.' To improve a CL method in limited total budget, we propose adaptive layer freezing that does not update the layers for less informative batches to reduce computational costs with neg- ligible loss of accuracy. In addition, we propose memory retrieval method that allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations. Empirical validations on the 10FAR-, CLEAR- 1EAR--O/IK 10/100, 100, and ImageNet- KK datasets demonstrate that the proposed approach outperforms the state-of the-art methods within the same total budget. Furthermore, we vali- date its effectiveness in the Multi-modal Concept incremental Learning setup with multimodal large language models, such as LLaVA-1,5-7B, Code is available attps:/ / /ggttuu.b...... com/ canumplaabbbdugteee..
INTRODUCTION
In a realistic scenario for continual learning (CL), data arrive in a streaming manner, prompting interest in online CL, which assumes one or a few samples arrive at a time. To effectively learn from new data while mitigating catastrophic forgetting (McCloskey & Cohen 1989) of previously learned
knowledge, various CL methods have been proposed, including replay-based approaches (Bang
et al. 2021 Seo et al., 2024), network expansion methods (Wu et al. 2022 Zhou et al. 2023). and
distillation- based methods (Koh et al. 2023 Wang et al. 2024).
For the practicality of online CL, most online CL methods impose resource restrictions, such as the single training epoch and limited replay memory, which restrict the number of streamed samples stored (Koh al, 2022; WWang et al. 2022a). While the 'one-epoch training' may give a rough sense of the computational constraint, the actual budget varies across methods (Prabhu et al.] [023] Ghunaim et al., 2023) since each method requires a different amount of computations in single epoch. Several rehearsal- CL methods require additional storage to store the previous models and logits (Buzzega et al. 2020 Zhou et al., 2023), which was usually not included in the memory budget, which mainly considers the size of episodic memory to store samples in previous tasks. To this end, we compare CL methods with the same computational and memory budget considering all storage and computational costs. We argue that the total budget of memory and computation will ensure the practicality of the proposed online CL algorithms.
For fair comparison in computational budget, we use training FLOPs per sample instead of the number of epochs, as some methods require significantly more computations per epoch than others.
Equual contribution. is affiliated with ECF, IPAT ASRI and corresponding author.

---

Published as a conference paper at ICLR 2025
EWC |PNAS 2017)
MR (NauriPS 2019)
DER (NwuiPP 2020)
X-DER (TPAM 2022)
MEMO (IG)R 2023)
CAMA 2024)
(NeuriSS REMIND (ECDV 2023)
OCS
LIDER (NurrPSS
GOL-DG (CVPR 2024)
aL-SAR (Durs)
Asuc
SWC WIR ROMIN DER X-Dor Lock W.10
GANA BL-20
MR RRAINN DUR ORS X-3ox Loch V8.00
63
AUAST 00
REMIN DER ous Xaur Lock
GAWA aL-02
MR REMIN DER ORS Xa1x Lack 1E:00 CMA aLAA
Constraint with epoch training
Constraint with 13.5 GFLOPs
Figure 1: Comparison of CL methods w/o total constraint (left) and total constraint (right) on CIFAR- 10 Gaussian setup. In the left plot, we compare CL methods with the same number of iterations and the same episodic memory size, i.e.. conventional setup, In the right plot, we compare CL methods with the same training FLOPs and fixed storage budget that includes both episodic memory and model storage, Le., our total- constrained CL setup. Compared the conventional setup, aL-SAR shows improved performance under the conal-constrained setup, since it can utilize the saved computational cost for further training. AAvC and AlaSt refer to the area under the curve of accuracy and last accuracy (i.e., accuracy at the end of CL), respectively.
FLOPs provide an exact measure of computational budget regardless of implementation details (Ko- rthikanti al.. 2023), following Zhao et al. (2023); ; Ghunaim et al. (223). For the same memory budget, we use an aggregated budget for various forms of extra storage, including replay memory, logits, and model parameters, converting them into bytes to obtain the actual memory cost, follow- ing [Zhou et al. (2023). Upon comparing the results under fair cost, we found that the performances were often different from what was reported in the original papers proposing each method.
Considering the total memory and computational budget. we propose an online CL method using computation-awan layer freezing strategy. Specifically, we propose to selectively learn (or freeze) layers per mini-bated based on the previously learned information to reduce computation In partic- ular, as more layers are frozen, training computation can be saved at the cost of losing information for the mini-batch since frozen layers cannot acquire any information. Considering this trade-off, we propose "adaptive layer freezing'. which chooses the best layers to freeze by maximizing the Fisher Information (FI) gained by the model for each batch, given fixed computation budget. Unlike previous freezing methods (Lee et al.] 2019 Hayes et al. 2020) Yuan et al. 20022,, which predefine the rule of when and which layers to freeze, not generalizing various datasets and models, adaptively select layers to freeze based on the varying information in each batch.
For the loss of accuracy by the reduced computational cost, propose novel sample retrieval method to improve accuracy with negligible training cost. While several online CL methods, such as MIR (Aljundi et al., (019a) and ASER (Shim et al. 202I), aim retrieve more informative training batches than random retrieval from replay memory. they require forward/backwward passes of large candidate set to select informative batches, increasing computation by 4 times and 3times compared to ER (Rolnick et al.) 2019), respectively. Applying these methods negates the computa- tional savings of the proposed adaptive layer freezing approach. To this end, we propose to retrieve samples that the model has not learned much about information stored in episodic memory for train- ing without incurring additional computational costs. To quantify the degree of learning, we employ the frequency of recent usage of each sample in training and the similarity of the gradients between classes. They are acquired during the training process, i.e., no need for additional inference.
For our empirical validations, compare the state-of-the online CL algorithms under the same FLOPs of computations and the same bytes of storage in Fig. [] We observe that several high- performance CL. methods do not maintain competitiveness under fixed FLOPPs and memory budget, unexpectedly trailing behind a simple Experience Replay(Rolmn et al., (201)). On the contrary, the proposed method outperforms them by a noticeable margin under the same total budget.
We summarize our contributions as follows:
Proposing measure computational and memory budgets of CL algorithms by using training FLOPs and total memory size in Bytes, to fairly compare different algorithms.
Proposing computationally efficient adaptive layer freezing that maximizes Fl per computation, as well as a memory retrieval strategy that prioritizes samples that the model has learned least.
Empirical analysis on the computational and memory costs of various CL algorithms, showing that many of CL methods are less beneficial under the same budget and showing that the proposed method outperforms them by noticeable margin across multiple benchmarks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
2.1 ONLINE CONTINUAL LEARNING WITH MEMORY BUDGET
Replay-base based online CL methods use episodic memory and consider the memory budget. Since we also consider using episodic memory, we review them in detail as follows. Replay-based meth- ods (Aljundi et al., 2019b; [ang et al. 2021: Koh et al.) 2022) store part of the past data stream in episodic memory to replay them in future learning. Although there are simple sampling strategies such random sampling (Guo et al., 2020) and reservoir sampling (Vitter 1985). they are often insufficient to adapt to changing data distributions. Rather than simple methods, researchers have developed advanced sampling strategies considering factors such as uncertainty (Bang al.. 2021 importance (Koh et al.) 2022), and gradient (Tiwari et al.) 2022). However, these advanced methods often entail high computational overhead, making them impractical for deployment in real-world applications. RM (Bang et al. 2021) requires significant amount of computational cost to calcu- late the uncertainty for diversified sampling. Similarly, CLIB (Koh et al.) 2022) involves additional forward and backward passes to calculate the decrease in loss for each batch iteration.
In addition to the memory management schemes, researchers investigate the memory usage schemes, ie., sample retrieval strategies from the rehearsal buffers. In addition to random retrieval (Chaudhry al., 2019), the determination of retrieval based on the degree of interference (Aljundi et al., (001a))
and the adversarial Shapley value (Shim et al. (2021) has been explored. However, such methods
require an inference of candidate samples, which leads to a nontrivial amount of computation in computing the loss (Aljundi et al. 2019a) or the Shapely value (Shim et al. 2021
2.2 LAYER FREEZING
Freezing layers have been investigated to reduce computational costs during training joint train- ing (i.e., ordinary training scenario other than CI.) (Brock et al. [017]] Xiao et al. 2019 Goutam cct al. 2020). A common freezing approach (Wang et al., 2023] Li et al. (022) includes determin- ing whether to freeze a layer, based on the reference model and representation similarity, such as
CKA (Cortes et al. 2012) and SP loss (Tung & Mori 2019). Additionally, EGERIA (Wang et al.,
2023
unfreezes layers based on changes in the learning rate.
However, in CL. both online and offline, it challenging to determine when to freeze a layer because metrics such as Euclidean distance and CKA cannot be used compare the degree of convergence compared to the reference model (Mirzadeh et al.] 2020). Additionally, continual learning involves non-i.idd setup, where the data distribution continues change (Criado al., (022). Therefore, in addition to changes in learning rate, it is important to consider the current data distribution when determining whether freeze or unfreeze a layer in continual learning. (Hayes et al. 2020) have explored freezing methods for continual learning. However, they use predefined freezing config- urations such as the freezing backbone block 0 after task 1, while our freezing method adaptively freezes the layers using information per batch.
APPROACH
For efficient learning in computation and storage budget, we consider two strategies; (1) reducing the computational cost of each iteration and (2) reducing the number of iterations. To implement both strategies, we propose a method employing two techniques; (1) adaptive layer-freezing and (2) similarity-aware retrieval of samples from episodic memory.
Specifically, for every training batch, the adaptive layer freezing method adaptively freezes layers so that the amount of information that can be gained from the mini-batch is maximized relative to the required computation. The memory retrieval method retrieves training batches that the model has not learned sufficiently using the number of times each sample has been used for training, i.e., use-frequency, and class-wise gradient similarity. This allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations, consequently reducing the overall number of training iterations. We call our method adaptive Layer freezing and Similarity-Aware Re- trieval (aL-SAR), illustrating the gradient update procedure of the proposed aL-SAR in Fig. and providing pseudocode in Sec. A.3
3

---

updated by retrieval results. After the (4) forward pass of model with the batch, (5) we compute the freezing criterion (BFC), for each layer Published as a conference paper at ICLR 2025
Similarity-Aware Retrieval
Adaptive Layer Freezing
L Update
① Update
Class Similarity
Sab
She
a0
Freeze
Backward
S
Replay Memory
batch
Fisher Information
Retrieval Probability
T
Forward
Max
Update
(BFC)
Co
C
Cg
Cn.3 C..2 G43 Gn
Discounted use frequency
72
Gradient update procedure the proposed aL-SAR. The colors the 'Simillaityy Aware Re- trieval' box denote different classes. (1) 'Retrieval Probability' is calculated using class similarity S and dis- counted use frequency 6; where c; tracks the number of times the 7th sample has heen used for training. (2) A batch sampled from memory by the 'Retrieval Probability' and (3)
Figure 2:
of the model, using Fisher Information (FI) and at (Sec. 3.1 (6) Layers to Tmmx arg max, (BFC), Bey
(in this example, Timax = 2) are frozen in the (7) backward pass. (8) Siy and FI are updated using the gradient de obtained from the backward pass.
3.1
ADAPTIVE LAYER FREEZING FOR ONLINE CL.
To reduce the computational cost of learning a neural network with minimal accuracy loss, there have been several studies on the freezing of neural network layers in non-CL literature (Liu al. 2021 He et al. 2021; Wang et al., 2023). These methods often rely on the learning convergence of each layer determine which layers freeze since converged layers no longer require further training. However, in online CL, achieving convergence is often challenging due to the limited training budget and the ever-evolving training data distribution. This requíres a new approach to determine when and which layers to freeze for the incoming data in the online CI. scenarios.
Selectively Freezing Layers by Maximum Fisher Information (FI). For a computationally ef- ficient freezing criterion in online CL, we propose freeze layers that learn little information per computational cost by measuring 'information' (I) gained by each layer during training. Here, we define the information (/) using Fisher Information (FI), since it is widely used metric measure the amount of information that each parameter in a neural network obtains from data (Durant et al., 2021 Desjardins al. 2015 Ollivier 2015). So. we use FI measure the layer-wise information that each layer gains from the input data to determine which layers to freeze. Note that freezing some layers facilitates training model for more mini-batches within fixed computational budget as it reduces the computations per mini-batch.
However, as trade-off, as the number of frozen layers increases, the number of updated parameters decreases, reducing the amount of information obtained per mini-batch. To maximize the informa- tion (/) in the model while minimizing the computational cost (C), we propose to maximize the expected amount of information gained per computation (T/C).
Formally, we try find Ammax arg max,, (T/C),, for E [1, L. where we define (I/C), h as the amount of information gained per computational cost when updating the model with layers I to frozen. refers to the total number of layers. To compute (I/C),, n. we factorize it as:
(T/C), = (T/mb), (mb/C)
(1)
where 'mb' refers to the mini-batch. (I/mb),; and (mb/C',, represent the amount of information gained per mini-batch and the number of mini-batch iterations per computation, respectively, when layers 1 to are frozen.
Amount of Information Gained per Mini-batch (I/mb),. To compute (L/mb)... we use F(O,), Fisher Information Matrix F()) of layer i, where 0 and 0, denote the parameters of the model po(-) and the parameters of the layer of p-(.). respectively. But computing all components costly Hessian is required, which involves second-order derivatives and can be computationally inefficient. To avoid the cost, we use first-order approximation of F(O,) by using the diagonal components of the F(O.) following (Kirkpatrick al. 2017 Soen & Sun 2021), i.e., using the
F(0.) is
4

---

Published as a conference paper at ICLR 2025
trace operator tr(-) as:
L E ur((()))) i=++1
where F(0,) (=)) ED ((
Oe (00))
ae
(1/mb)n
(2)
30;
where is the data stream, z E DD is the training data batch in the data stream, and & log the loss function. Note that since layers I to n are frozen, the model gains information of layers + 1 to L, the unfrozen layers only.
Number of Mini-batch per Computational Cost inb/C)n. To compute (mb/C)n, we initially determine its inverse, denoted (C/mb)n. which represents the computational cost per mini- batch when layers 1 to frozen. We calculate (C/mb),, by splitting it into forward and backward propagation as: (C/mb)nn = Σ (FF); + Ei (BF); where (FF), and (BF); refer to the forward FLOPSs and the backward FLOPPs of layer respectively. By taking an inverse of it, we obtain
(mb/C)n
E (FT), E (BF);
Note that since layers I to are frozen, no backward computation is performed for those layers. When the number of frozen layers (i.e., n) increases, the number of layers performing backward operations is reduced, i.e., Σ imm+t (BF); decreases, thus leading to an increase in the possible number of mini-batch within the given computational budget C.
Combining Equation 2 and Equation 3 we can finally calculate (I/C),, which represents the infor- mation the model can gain within given computational budget when layers are frozen, by product of ( //mb,, and (mb/ (:):
El tr((()))) (I/C),, (I/mb), (mb/C),, = E (FF); + Ei=+1 (BF);
(4)
By freezing layer 1 to layer Remax arg max, (I/C)n, we can maximize the expected amount of information gained per computation during training.
Batch-wise Freezing for Online CL. (I/C), is supposed to be calculated on the entire data stream since F(0) is defined an expectation over the whole dataset in Equation 2 Thus, (T/C),, does not account for the variable amount of information of each batch. In online CL where the incoming batch distribution continuously shifts, the variation is not negligible. Specifically, for batches containing sufficiently trained classes, it is advantageous to freeze more layers, while for batches with insufficiently trained classes, is beneficial freeze fewer layers. To address this, we propose the batch freezing criterion (BFC) which quantifies the net benefit in the information gained by the model when we freeze layers given an input batch Et-
To define the BFC, we compare (1) the amount of information we would lose from the current batch by freezing and (2) the expected amount of information we would gain the future using the saved computation from freezing. Then, we can estimate the net benefit from freezing in terms of the information gained by the model by subtracting (I) from (2).
To estimate (I), we use the trace of layer-wise FI, i.e., tr(F(0,)). defined in Equation 2 Note that F(Ox) is the FI over the whole dataset; thus we convert it into batch-specific FI F (0.) =
Epo(=)) 30,
at by multiplying the ratio of the FI of the batch relative to the FI of the 30;
entire dataset. Using Fz, (..), the amount of information we would lose from the current batch 2, by freezing layers 1 to is obtained as > tr( (Fz((0)) Please refer to Sec. A. for more details on the estimation of Fz, (0.) from F(O.).
To estimate (2), we calculate the amount of computations saved by freezing layers 1 to n, and the amount of information we can obtain from the saved computations. Here, the saved com- putation refers to the sum of the backward FLOPs of the frozen layers, denoted as (,F,(BF) Using (I/C), defined Equation
we estimate the expected information (/) obtainable from the saved computations (C) by multiplying the saved computation and (I/C),. With optimal freezing that maximizes (I/C), the anticipated information obtained from the saved calculations max (I/C)mm Σ (BF),
5

---

Published as a conference paper at ICLR 2025
Finally, by subtracting (1) from (2), we obtain BFC(2))n as:
(z) = max (I/O)m m Σ (BF),- - (F((F,,
(5)
m
The positive BFC();), implies that freezing layer to n is beneficial in terms of information, and a negative value indicates otherwise. By freezing layer 1 to (=)) = arg max, BFC(=)),. we can select the most beneficial freezing strategy for batch 2,. We argue that it allows us to dynamically select the freezing strategy by the learning capacity of the layers and the batch's informativeness. We empirically demonstrate the distribution of the BFC(2)), in Sec. A.10
3.2 SIMILARITY-WWARE RETRIEVAL BASED ON "REEUERRE
In rehearsal-based CL methods, sample retrieval strategies such as MIR (Aljundi et al., 2019a) and ASER (Shim et al. 202T) have a high computational cost, requiring multiple additional model inferences. Thus, in the same computational budget. their performances are surprisingly inferior to simple random retrieval (i.e., ER) as shown in Fig. Here, we propose computationally efficient sample retrieval strategy that does not require additional model inference.
In online CL., new data continuously stream in, and old data remain in memory, causing an imbalance in 'the number of times each sample used for training'. which we call "use-frequency. We argue that samples with high use-frequency yield marginal knowledge gains in training. and samples with low use-frequency are likely to contain knowledge that the model has not yet learned. So, we propose to sample data with low use-frequency for training with high probability.
Discounted Use Frequency (c.). But using the useffeeyyycyy for sampling does not consider the knowledge forgetting in the CL setup. If a sample was frequently used in the past but seldom used in recent iterations, its knowledge may have been forgotten, despite its high use-frequency. Inspired by the exponential decaying model of forgetting (Shin & Lee) 2020 Mahto et al. 2020; Chien al., 2021), , we propose a decay factor (0 < r < 1) in the use frequency at each iteration, calling bbscountedduss-ffreqeenn
for 7th sample (c,). For example, if 7th sample is used times for training at a specific time point, after iterations, we define its discounted use frequency as c, = n =n.7..
Effective Use Frequency (c.). However, the model can learn knowledge about a sample by train- ing other similar samples (e.g., samples from the same classes). Thus, e of the other samples could also affects of the sample. These similar samples effectively increase the use-frequency of the particular sample. At the same time, the model may lose knowledge about the sample when training on different samples (e.g., from other classes), effectively decreasing the neeffreuency.
To account for this, we define 'effective-use ueserriveu by adding the other samples' use-ffrequee use-fireuency multiplied by the similarity of samples to c, For the sample similarity score, inspired by (Du et al.) 2018). which uses gradient similarity to assess the helpfulness or harmfulness of auxiliary task to the original task, we hypothesize that samples with similar gradients bear similar information. So, for the proxy of sample similarity, we use cosine similarity between the gradients.
However, tracking the gradient similarities between all sample pairs requires excessive memory (~ 10 pairs for ImageNet) and computation. Thus, we approximate it to class-wise similarities, which is the expected gradient similarity between samples from two classes. Formally, we define the class-wis wise similarity Sy. for classes 3/1 and y/ as:
Dr, Sy. =x (Cos(Val(21)) V
(6)
where Dy, is the training data for class y/ and V=((=;) is gradient of 7th sample. Using the approxi- mated class-wiss similarities, we define the effective--us--requuees è for the 7thh sample as:
é = c + Σ Sy.u. Cy.
NEY
where )/ is the set of all seen classes, Sy.v, is class similarity between class y and yi. and Cy Σy;=y 9j is the sum of the discounted use- fse-frequesssss for all samples of class y.
Unfortunately, calculating the expected value in S Sp,,3 (Equation 6) from scratch for each iteration requires gradient calculation for all samples in the classes y: and y/, which is computationally

---

Published as a conference paper at ICLR 2025
expensive. As computationally efficient alternative, we further propose using the EMA to update the previous estimate of Sy.v, rather than calculating the expectation from scratch. Note that we only utilize gradients from unfrozen layers, obtained during the training process, i.e., incurring no additional cost. Please refer to Sec. A. for detailed information on the calculation of Sy...
Finally, we obtain the retrieval probabilities for 1th sample with the efflective-us--feeqeeeey as:
4/T
P: = D 4//
(8)
where is temperature hyperparameter. Samples with low & have high chances of being retrieved, so insufficiently trained samples are preferred to sufficiently trained ones, thereby accelerating train- ing. We present several toy experiments and an ablation study the components of SAR to validate the empirical benefit of our retrieval algorithm in Sec. AA.66) and Sec. A.8 respectively.
4 EXPERIMENTS
4.1 SETUP
For empirical validation, adopt the total budget for memory and computation. For the memory budget, we use Bytes following (Zhou et al. 2023). which considers memory costs not only for the samples in episodic memory but also for additional model parameters used in regularization or distillation. For the computational budget, we use training FL.OPs. For the dataset, we use CIFAR- 10/100, CLEAR-I0/1000, and ImageNet-IK. We evaluate the methods a conventional disjoint task setup and newly proposed Gaussian task setup (Shanahan et al. 2021 Wang et al. 2022b Koh al. 2023). For all experiments, we averaged 3 different random seeds, except ImageNet- IK due to computational cost (Bang et al. [2021; Koh et al. (0233). We conducted Welch's 's t-test with significance level of 0.05. We highlighted the highest performance in bold. In cases where statistical significance was not observed, we underlined all other results within the significance level.
Metrics. We report the last accuracy Aust and the area under the curve of accuracy AaUC (Koh ct al. 2022). The Aga measures the accuracy at the end of CL. The AxUC measures the accuracy per time step using the accumulated test set of all previously seen tasks and then computes the area under the accuracy curve. For each evaluation, we evaluate using the entire test set for the classes seen so far. We argue that AAUC a suitable metric measure prompt learning of new knowledge.
Baselines. We compare SAR to state-of- of-the-art art online CL methods such as ER (Rolnick al., 2019) DER++ (Buzzega et al.) 2020), MIR (Aljundi et al.) 2019a), MEMO (Zhou et al. (0233). RE- MIND (Hayes et al. 2020), EWC (Kirkpatrick et al. 2017), OCS YYoon et al. 2022). LiDER (Bon- icelli et al., 2022). X-DER (Boschini et al. 2023). CCL-DC (Wang al.] 2024). and CAMA (Kim
ct al. 2024). We describe the implementation details and hyperparameters in Sec. A.4
4.2 QUANTITATIVE ANALYSIS
We evaluated CL methods, including aL-SAR, with strictly restricted computation and memory budgets specified in Sec. 4.1 Note that while we set the training iterations of aL-SAR to be the same as other baselines, AAR adaptively freezes layers, resulting in fewer FLOPs consumed. For experiments various computational and memory budgets, we use the relatively small CIFAR datasets to cover a wide range given budgets. To validate our methods on large datasets and datasets with temporal domain shift, we also show experiments ImageNet and CLEARR datasets. Note that these experiments are conducted on various CL. setups, including Gaussian task setup, Disjoint task setup, and domain-incremental setup. Furthermore, we applied our adaptive layer freezing method to LLaVA-v 1.5-7B, demonstrating a significant reduction in the computational cost large language model while preserving performance.
of training the
Various Computational Budget under the Same Memory Budget. We compare CL methods under fixed memory budgets and various computational budgets in Fig. 3 We observe that aL-SAR significantly outperforms others in all datasets and both setups, especially under a low computational retrieval effectively promotes rapid learning by retrieving
budget. It shows that our similarity informative training batches even with a limited computational budget, while random retrieval and MIR (Aljundi et al. 2019a) require high computation to achieve comparable performance.
7

---

Published as a conference paper at ICLR 2025
EWC -
ER -
MIR
REMIN
DER+
MENO
OC5
LDER
-
XDER
60.00C
GAMA N AR-AAR (Oum)
70
Aust 2
15
Asuc
80
Auous
80
GFLlPssammle
GFLOPslsample
GFLoPs/sample
GFLOPs/sample
(a) CIFAR-10 Gaussian task setup
(b) CIFAR-10 Disjoint task setup
EWC ER
MIR
REMIN
DER
MENO
OCS
LDER
X
COL DC
GAMS
BAR (Ous)
g
AJuc 4
2
Auas
E
Auout
GFLOPs/sample
GFLOPstsample
GFLOPsisample
GFLOPs/sample
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 3: Accuracy on Gaussian and Disjoint CL setup CIFAR-10 and CIFAR-100 for a wide range of FLOPs per sample. aL-SAR outperforms all CL methods compared. We use a memory budget of 30.4MB.
Furthermore, observe notable increase in the FLOPs saved by aL-SAR through freezing, par- ticularly pronounced at higher computational budgets. As the model undergoes more iterations, the amount of information that the model gains from the training data decreases. Thus, our adap- tive layer freezing adaptively adjusts the freezing criterion to freeze more layers, leading to lower FLOPs, thus the line stops the earlier GFLOPs value than the baselines. We provide comprehen- sive analysis when varying computations under various memory constraints in Sec.A.5 Please refer Sec. A. for more details on the computational budget.
OCL
SAR
Aauc
Auast
Auest
50
Memory Size (MB)
Memory Size (MB)
Memory Size (MB)
Memory Size (MB)
(a) CIFAR-10 10 Gaussian task setup
(b) CIFAR-10 10 Disjoint task sctup
CCL-DC
Alost
Alost
20-
Memory Size (MB)
Memory Size (MB)
Memory Size (MB)
Memory Size (MB)
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 100 Disjoint task setup
Figure 4: Accuracy on Gaussian and Disjoint CL setup in CIFAR-I0 and CIFAR-100 for various memory budget. aL- outperforms all CL methods compared. We use computational budget of 171.94 TFLOPs.
Various Memory Budget under the Same Computational Budget. We now fix the computa- tional budget and various memory budgets for CL methods Fig. With its minimal additional memory usage and effective utilization of episodic memory through similarity- retrieval, aL- SAR again outperforms other methods by significant margin in datasets, indicating its suitability for both large and small memory budgets.
Large Datasets and Temporal Domain Shifts. To investigate the scalability of CL methods large datasets and temporal domain shifts, we report the performances on ImageNet- Gaus- sian setup and CLEAR-10/10 1000000000 with fixed computational and memory budget in Tab. [ In the
8

---

Published as a conference paper at ICLR 2025
ImageNet-IK
CLEAR-10
CLEAR-100
Methods
AAUC t
Alast 1
AAuC t
Abst 1
AAUC 1
Aast
EWC (Kirkpatrick et al. 2017
20.05
32.55
62.40=+0.17
1..25++1229
29.62+0044
40.36 1.34
ER (Rölnick et al. 2019
20.01
32.60
64.00:t0.06
12.28+1.115
01.56.10.42
44.90:1:.74
ERMMIK [Tuund et al.
2019g
7.45
17.04
62.29:10.17
70.52::1.13
23.2411014
1.
REMIND [Haes
2020
15.58
22.42
62.64-10.04
70.68:10.90
26.76110.30
30.99110.55
+ER++ [B Buuzzeg el
2020
23.51
29.82
61.30=0.77
71.52:1.02
25.83-0.43
39.89=1.899
OCS Yoon ct al. 2022
6.90
19.57
58.69-0.25
68.6941.21
15.2411.27
22.52:+2.02
LiDER (Bonicelli cl
2022
27.65
34.66
58.36=0075
69.16+1.21
28.75=0028
39.92=
X-DER Boschim al
2023
27.46
34.47
58.19=0.23
69.70+0.97
23.01=0.15
33.70==0.00
MЕMO
Zhou
20245
11.15
19.98
60.29+0.31
68.69+1.21
20.33+0.28
25.53++0.22
CAMA
Kim
2024)
7.94
17.44
60.97=0.52
71.661118
26.27-0.37
39.42:10.67
CCL-DC
Wang 2024
19.70
32.12
61.08
71.25+0.82
23.81=0.32
35.07=00667
aL-SAR (Ours)
33.24
35.02
6..61
7..3610.80
38.65
47.56.10.37
Table 1: Quantitative comparison between different CL methods CIL setup. aL-SAR outperforms baselines with Jower computational budget and same storage budget. Specifically, while all other base- lines consume 112,000, 716, and 5,250 TFLOPs, aL-SSA consumes 93,000, 598, and 4,059 TFLOPs the ImageNet-IK Gaussian setup, CLEAR-10, and CLEAR-100, respectively. The memory budget is fixed at 5,736MB, 564MB, and 1,148MB in ImageNet-IK, CLEAR--O, and CLEAR-I00, respectively.
ImageNet-1K K Gaussian setup, -AA outperforms other methods with slightly fewer FLOOPs In this setup, training distribution shifts continuously, so the model has to constantly adapt to the new distribution, resulting in less freezing and more benefit from the fast adaptation enabled by the trieval method.
We then investigate CL methods under temporal domain shift with fixed computational and memory budget using the CLEAR-I0/ datasets. Unlike the class-incremental, where new classes are added, this As shown in the table, aL-SAR also outperforms the state-of-the-art in domain-incremental setups. It is partly because so that the model learns more on relatively less-learned domains, allowing fast adaptation to new domains. Note that aL-SAR also significantly saves FLOPs thanks adaptive layer freezing. Please refer to Sec. A.23 and Sec. A for more details about the memory budget and the computational budget, respectively. Moreover, we provide extensive experimental results in the memory infinite setup and comparisons with efficient CL baselines in Sec A.22 and Sec A.12 respectively.
setup introduces new domains, while the classes remain the same.
retrieval method balances the fre-ueeyyy of samples in different domains
Comparison of aL with Layer Freezing Methods. We compare proposed adaptive freezing method (aL.) with REMIND (Hayes et al. 2020) and PTLF (Yang et al. 2023). layer freezing meth- ods for offline CL, well as EGERIA (Wang et al. 2023), layer freezing method for non-CL. The results are summarized in Tab. 2 As we can see in the table, PTLF and EGERIA show negli- gible differences in TFLOPs with No Freezing, indicating minimal layer freezing. While REMIND significantly reduces training FLOPs, it leads to performance degradation. In contrast, our proposed aL significantly reduces training FLOPs while maintaining performance. Notably, REMIND, PTLF, and EGERIA require task identity information during training, while aL operates without it. This further highlights the practicality and adaptability of al. in online CL setups. We provide detailed analysis of the results and further comparisons in the disjoint setup in Sec A.15 for the sake of space.
CIFAR-10
CIFAR-100
Methods
Asuc
Aus
TFLOPs J
AwuC 1
Ausi
TFLOPs
freezing
64.60
0.83
72.43
0.38
171.94
42.49 0.75
50.49 + 0.29
515.82
REMIND
59.25 0.37
65.01
1.31
151.31
(-12.0%)
35.50 0.56
40.00
+ 0.86
453.92(-12.0%)
PTLF
63.82 + 0.24
71.30
0.91
164.14 (-4.5%)
42.09 0.58
49.92 + 0.65
495.67 (-4.1%)
EGERIA
63.95 0.72
71.72 0.81
169.25 (-1.6%)
42.18 0.63
50.05
1 0.72
506.54 (-1.8%
aL (Ours)
64.38 0.32
72.57
0.79
146.80 (-14.6%)
42.38 0.76
50.62 + 0.87
427.90 (-17.0%)
Table 2: Comparison between our proposed adaptive layer freezing and other freezing methods. We compare them in the CIFAR-10 Gaussian setup and the CIFAR-100 Gaussian setup.
Comparison of SAR with Memory Retrieval Methods. We compare our proposed retrieval method, ie.. SAR with ASER (Shim al. 2021) and MIR (Aljundi et al. 2019a in Sec. A.19

---

Published as a conference paper at ICLR 2025
Application of aL in Training Multi-Modal Large Language Models (MLLMs) We demon- strate the effectiveness of aL. in training MLLMs by applying it to training of the LLaVA-1.5-7B model (Liu et al.] 2023). Following Ye al. (2023). we update only the pretrained projection MIP layers and I LoRA adapters (Hu et al. 2021). ). keeping the LLM frozen for training efficiency.
We summarize the result in Tab. 3 We provide implementation details and datasets (i.e., Bongard- HOI (Jiang ct al. 2022) and Longard-Opento World (Wu et al. 2024)) of MLLM training in Sec. Sc1.
As shown in the table, aL saves approximately 12% in training FLOPs while maintaining perfor- mance, by adaptively freezing LoRA layers during the training process. We believe that aL can be effectively integrated in a plug-and-play manner with CL methods in large models, such as LLMs and MLLMs, to reduce training costs while achieving strong performance.
Bongard HOL-PNN
Bongard Open WorlWWord-PPo P/N
Methods
AAcC
Aua
TFLOPs
AAuC
Aast f
TFLOPs 1
No freezing
08.01
65.39
1578.02
56.71 15.34
16.15
2959.61
aL (Ours)
1.77
04.79
1431.27 (-10.3%)
57.17:12.61
2722.23 (-8.0%)
Table 3: The effect adaptive freezing on MLLM training, We use LLaVA-1.5-78 model with LoRA.
4.3
ABLATION STUDY
We ablate the model to investigate the benefit of each proposed component in CIFAR-10/100 and summarize the results in Tab. E In Tab. + similarity-aware retrieval (SAR) increases the perfor- mance while using the same number of iterations. This shows that SAR increases the amount of knowledge learned per iteration, as we claim in Sec. 3 While computational cost also increases, its increase is modest compared other retrieval methods such as MIR (Aljundi al. 2019a) or ASER (Shim et al. (021) that require 3 ~ 4x more computations. Furthermore, we observe that the aL significantly reduces FLOPs with negligible loss in accuracy. In summary, our method out- performs the baseline while using fewer FLOPs than the baseline, each by a noticeable margin. We provide further ablation studies of the proposed components in the Disjoint setup in Sec. A.7
Furthermore, we present an ablation study of the proposed retrieval method, i.e. SAR, in Sec A.8
CIFAR-IU 10
CIFAR-100 100
Methods
AAUI
Aus
TFLOPs
1
AaiC
Aug 1
TFLOPs
Vanilla
60.76:10.1
70.08++0.97
163.74
31.97:1..9
37.80=1.30
245.91
+ al.
60.38:10.54
69.04410.83
142.23
31.77-00.0
38.03=10.35
217.40
+ SAR
64.60-+0.83
72.43+00.88
171.94
37.60+00.40
42.69-10.18
257.97
aL & SAR (Ours)
64.38:10.22
72.5710.79
146.80
37.2010.73
42.55
221.49
Table 4: Benefits of the proposed components of aL-SAR, adaptive layer freezing (aL) and similarity aware
retrieval (SAR), in Gaussian task sctup. 'Vanilla' is simple replay-based method that trains on randomly retrieved batches from a balanced reservoir memory. The memory budget is 7.6MB for CIFAR- 10 and 13.44MB for CIFAR-100. We train for 1 iter per sample for CIFAR-I0 and 1.5 iter per sample for CIFAR-100.
Moreover, we provide detailed studies in the appendix for the space sake. Specifically, we investigate the performance with different freezing strategies See. AA. the detailed effect of freezing on accuracy and FLOPss in Sec. A.9 the effect of temperature 7 in Sec. A.20 and the application of our freezing method on Vision Transformer (ViT) (Dosovitskiy et al. 2020)) is discussed in Sec. A.
CONCLUSION
We address the challenge of achieving high performance on both old and new data with minimal computational cost and limited storage budget in online CL. While CL with fixed episodic memory size has been extensively studied, we have investigated the total storage budget required for online CL as well the computational budget for developing practically useful online CL methods.
To this end, we proposed aL-SAR, a computationally efficient CL method comprising two com- ponents: similarity-awe aware retrieval and adaptive layer freezing. Our empirical validations show that several nigh-performing CL methods are not competitive under a fixed computational budget, falling behind simple baseline of training on randomly retrieved batches from memory.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We propose better learning scheme for online continual learning for realistic learning scenarios. While the authors do not explicitly aim for this, the increasing adoption of deep learning models in real-world contexts with streaming data could potentially raise concerns such as inadvertently introducing biases or discrimination. We note that we are committed to implementing all feasible precautions to avert such consequences, as they unequivocally contrary to our intentions.
REPRODUCIBILITY STATEMENT
To further facilitate the reproduction, we provide open-source implementations our proposed method (Sec 3. along with data splits and baseline models used in our experiments (Sec available at https: : /igttuu...... com/ htt.aa/......aamaaaa/a//aaeeeeee /tt.m//llabbbudgeeeeeee...
ACKNOWLEDGMENT
This work partly supported by Al Center, Samsung Electronics, and the IITP grants (No.RS- 2022-11220077, RS-2022-11220113 N02RS-2022-12229999 No.RS-2021-1211333 (SNU AI), No.RS-20011-1211206 (AI Innov. Hub)) funded by the Korea government (MSIT).
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In NeurIPS, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS, 2019b.
Soumya Banerjee, Vinay K Verma, Avideep Mukherjee, Deepak Gupta, Vinay Namboodiri, and Piyush Rai. Verse: graddintt aware streaming lifelong learning with anytime inference. arXiv preprint arXiv:2309.082277 2023.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In CVPR, pp. 8218-8227, 2021.
Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. In NeurIPS, 2022.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions Pattern Analy- sis and Machine Intelligence, 45(5):5497-5512, 2023.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. In arXiv: 2017.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. 2020.
Lucas Caccia, Jing Xu, Myle Ott, Marcaurelio Ranzato, and Ludovic Denoyer. On anytime learning at macroscale. In CoLLAs, 2022.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Tomr and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.

---

Published as a conference paper at ICLR 2025
Hsiang- Yun Sherry Chien, Javier S Turek, Nicole Beckage, Vy A Vo, Christopher Honey, and Ted L Willke. Slower is better: revisiting the forgetting mechanism in lstm for slower information decay. arXiv preprint arXiv:2105.05944 2021.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kemelss based on centered alignment. The Journal of Machine Learning Research, 13(1):795 8288, 2012.
Marcos F Criado, Fernando E Casado, Roberto Iglesias, Carlos V Regueiro, and Senén Barro. Non- iid data and continual learning processes in federated learning: long road ahead. In Information Fusion, 2022.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, pp. 702-703, 2020.
Stefan Depeweg, Constantin A Rothkopf, and Frank Jäkel. Solving bongard problems with visual language and pragmatic reasoning. arXiv preprint orXiv.180.004455 2018.
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, et al. Natural neural networks. In
NeurIPS, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Yunshu Du, Wojeiech Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pas- canu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. In arXiv:1812.02224. 2018.
James H Durant, Lucas Wilkins, Keith Butler, and Joshaniel FK Cooper. Determining the maximum information gain and optimizing experimental design in neutron reflectometry using the fisher information. Journal of Applied Crystallography, 2021.
Yasir Ghunaim, Adel Bibi, Kumail Alhamoud, Motasem Alfarra, Hasan Abed Al Kader Hammoud, Ameya Prabhu, Philip HS Torr, and Bernard Ghanem. Real-time evaluation in online continual learning: A new hope. In CVPR, pp. 11888-1811 11897, 2023.
Kelam Goutam, S Balasubramanian, Darshan Gera, and R Raghunatha Sarma. Layerout: Freezing layers in deep neural networks. SN Computer Science, 1(5):295, 2020.
Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning. 2020.
Tyler L Hayes, Kushal Kafle, Robik Shrestha. Manoj Acharya, and Christopher Kanan, Remind your neural network to prevent catastrophic forgetting. In ECCV, 2020.
Chaoyang Hc, Shen Li, Mahdi Soltanolkotabi, and Salman Avestimehr, Pipetransformer. automated clastic pipelining for distributed training of large-scale models. In ICML, pp. 4150-4159, PMLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0940 2021.
Huaizu Jiang. Xiaojian Ma. Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard- hoi: Benchmarking few-shot visual reasoning for human-object interactions. In CVPR, 2022.
Byeonghwi Kim, Minhyuk Sco, and Jonghyun Choi. Online continual learning for interactive in- struction following agents. In ICLR, 2024.
12

---

Published as a conference paper at ICLR 2025
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Grabska- Baroskka-ar,iskku, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. PNAS, 2017.
Hyunseo Koh, Dahyun Kim, Jung Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In ICLR, 2022.
Hyunseo Koh, Minhyuk Seo, Jihwan Bang. Hwanjun Song, Deokki Hong, Seulki Park, Jung- Woo Ha, and Jonghyun Choi. Online boundary-free free continual learning by scheduled data prior. In ICLR, 2023.
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mo- hammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. In PMLR, 2023.
Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. In 191iiv:/11..33100 1.00900, 2019.
Sheng Li, Geng Yuan, Yue Dai, Youtao Zhang, Yanzhi Wang, and Xulong Tang. Smartfiz: An efficient training framework using attention-baan layer freezing. In ICLR, 2022.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023.
Yuhan Liu, Saurabh Agarwal. and Shivaram Venkataraman. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning, arXiv preprint arXiv:2102.01386, 2021.
Shivangi Mahto, Vy Ai Vo, Javier S Turek, and Alexander Huth. Multi-timessen representation
learning in Istm language models. In ICLR, 2020.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In ICLR, 2020.
Yann Ollivier. Riemannian metrics for neural networks i: feedforward networks. Information and
Inference: A Journal of the IMA, 1022:108-15,,, 2015.
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real-time continual learning. In IROS, pp. 10203-10209. JEEE, 2020.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, pp. 524-540, 2020.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR, 2023.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. 2019.
Michael Rotman and Lior Wolf. Shuffling recurrent neural networks. In AAAI, 2021.
Minhyuk Seo, Hyunseo Koh, Wonje Jeung, Minjae Lee, San Kim, Hankook Lee, Sungjun Cho, Sungik Choi, Hyunwoo Kim, and Jonghyun Choi. Learning equi-angular representations for online continual learning. In CVPR, 2024.
Murray Shanahan, Christos Kaplanis, and Jovana Mitrović. Encoders and ensembles for task-free continual learning. arXiv preprint arXiv:2 13327, 2021.
Alex Sherstinsky. Fundamentals recurrent neural network (mn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132308 2020.
13

---

Published as a conference paper at ICLR 2025
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. On- line stass-incremental continual learning with adversarial shapley value. In AAAI, 2021.
Hyo-Sang Shin and Hae-In Lee. A new exponential forgetting algorithm for recursive least-squaree parameter estimation. arXiv preprint arXiv:2004. 200,20000.03910, 2020.
Alexander Soen and Ke Sun. On the variance of the fisher information for deep learning. 2021.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In CVPR, 2022.
Frederick Tung and Greg Mori. Similarity knowledge distillation. In ICCV, pp. 1365- 1374, 2019.
Jeffrey S Vitter. Random sampling with a reservoir. In ACM TOMS. ACM New York, NY, USA, 1985.
Liyuan Wang, Xingxing Zhang, Kuo Yang. Longhui Yu, Chongxuan Li, HONG Lanqing, Shifeng Zhang, Zhenguo Li. Yi Zhong, and Jun Zhu. Memory replay with data compression for continual learning. In ICLR, 2022a.
Maorong Wang, Nicolas Michel, Ling Xiao, and Toshihiko Yamasaki. Improving plasticity in online continual learning via collaborative learning. In CVPR, pp. 23460-23469, 2024.
Yiding Wang, Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowdhury. Egeria: Efficient dnn training with knowledge-guider layer freezing. In EuroSys, 2023.
Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022b.
Rujie Wu, Xiaojian Ma. Qing Li, Wei Wang, Zhenliang Zhang, Song- Chun Zhu, and Yizhou Wang. Bongard-openworld: Few-shot reasoning for free-form visual concepts in the real world. In ICLR, 2024.
Tz- Ying WU, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In CVPR, pp. 9601-9610, 2022.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
Xueli Xiao, Thosini Bamunu Mudiyanselage, Chunyan Ji, Jie Hu, and Yi Pan. Fast deep learn- Cereeeeooo-CPCCoSSmmee
ing training through intelligently freezing layers. In Cybermatics, 2019.
Li Yang, Sen Lin, Fan Zhang, Junshan Zhang, and Deliang Fan. Efficient self-supervised continual learning with progressive task-correlated layer freezing, arXiv preprint arX1v:2303.0747, 2023.
Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-basse continual learning. In ICLR, 2022.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024.
Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong. Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of generic framework for sparse training. In NeurlPS, 2022.
14

---

Published as a conference paper at ICLR 2025
Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Does continual learning equally forget all parameters? In ICML, 2023.
Da-We Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficiet class-incremental learning. In ICLR, 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DETAILS ON THE ESTIMATION OF BATCH-WISG WISE FISHER INFORMATION Fs, (o.)
To estimate the batch-wise Fl Fz, (0t) from Fl F(O:), we compute the Fl ratio, i.e., the ratio of the FI of the batch 2, to the FI of the entire dataset, using the property that the FI is quadratically proportional to the magnitude of the gradient (equation 2) To be specific, we compute the ratio between V [=((=)) of the current batch and the expectation of | v=](2 on the whole dataset.
Note that, we only use the gradient of the last layer feature zL to estimate the magnitude of the gradient following (Koh et al. 2023), since the gradients of the preceding layers are proportional to the gradient of the final layer due to the chain rule. Using the estimate, we compute the batch-wise Fl Fz, (0:) from batch when freezing layers 1 to n as:
Vv {(=)12 F (0.) = E, VV { F(O.),
where Eg is the expectation over all input batches.
For the calculation of Fz. (0.), we have to compute the expected values in the average gradient magnitude (E= [V,,,, .,. {=)) and FI (F(O,)'s (0)) (equation 2}. Since calculating the expected values (using all samples in replay memory) in every learning iteration is computationally expensive, we estimate them by the exponential moving average (EMA) of the estimated expectations computed by the mini-batch of the past iterations. However, the EMA estimate of tr( (00))) requires a gradient calculation for all layers, so it cannot be used with freezing, which stops the gradient computations. Since the estimation of tr (r,) and the freezing cannot be performed at the same time, at each m iteration, train (i.e., unfreeze) all layers to update the estimate of tr( (F())) for all i. For the other m iterations, we do not update tr( ((.)) and freeze the model based on the values of (BFC), using the previously estimated value of tr( (0.)).
A.2 DETAILS ABOUT ESTIMATION OF FISHER INFORMATION TRACE
To check how accurate our Fisher Information trace estimate is, we ran an experiment comparing the Fisher Information trace estimated with a batch size of 16 once every four steps and ii) a batch size of 64 for every step (i.e. 16 times larger sample size) on CIFAR-10 Gaussian task setup. We use ResNet-32 as the backbone and show the trace of the Fisher Information of the last layers for each block, i.e. layers 8, 16. 24, and 32. From the result in Fig. 5] we observe that the estimation with i) batch size of 16 once every four steps does not deviate much from the estimation with ii) a batch size of 64 for every step, showing that our estimation is reasonably accurate.
A.3 DETAILED ALGORITIM OF AL-SAR
Algorithm [ a comprehensive pseudocode for the aL-SAR method. aL-SAR has two com- ponents: similarity-awar retrieval and adaptive layer freezing. In the algorithm box, lines 6-13, and 25-26 describe the similarity-awar retrieval method, and lines 1 15-24 describe the adaptive layer freezing method.
A.4 IMPLEMENTATION DETAILS
We use ResNet-32 He et al. 2016) for 10, CIFAR- 100, CLEAR-10 and CLEAR-100, and use ResNet- 18 as the network architecture for ImageNet- -K. We set the training hyperparameters as follows (Prabhu et al. 2020 Bang et al. 2021 Koh et al. 2022). For CIFAR- CIFAR- 100, CLEAR-10, and ImageNet, we use batchsize of 16, 16, 16, and 256, respectively, and Adam op- evaluation and 8000 samples for ImageNet- IK. used memory size of 7.6MB, 13.44MB, 25. 25.B for CIFAR-10 and
timizer with LR of 0.0003 for all datasets and setup. To calculate AAUC, period of 100 samples for CIFAR- 10/1000 and CLEAR- For memory constraints, CIFAR- 100, 617MB for CLEAR- 10, 5.8GB for ImageNet.
For data augmentation, we apply RandAugment (Cubuk et al. 2020) to all CL methods. For hyper- parameters, we set all the EMA ratios required for aL-SAR to 0.01 for all datasets. For the values of k and T used in memory retrieval, we use k 4 and T 0.125 for all experiments.
16

---

Published as a conference paper at ICLR 2025
layer
Ryer baye 16 (16x sample size)
leyer layer 24 (16x sumple sice)
layer
layer 8 (16x sample siee)
layer 32 (16x sample sice)
0.18
0.14
0.10
tn(F)
0.08
0.04
0.02
0.00
lterations
Figure 5: The estimated of Fisher Information for layers 8, 16, 24, and of ResNet-32 CIFAR- Gaussian Task setup, comparing the estimation used in aL-SAR and the estimation with a 16 times bigger sample size.
We found the hyperparameters through search on CIFAR-10 and applied them to other datasets and setups without further tuning. This decision was made due to the lack of access to the dataset before actual training real-world continual learning applications, as data arrive as an online stream Thus, cannot perform a datasct-specific hyperparameter search. Similarly, distribution of the data in the future, so a aetup-specifie (e.g., Disjoint/Gaussan task) hyperparameter search is also not possible. Therefore, realistic scenario is performing hyperparameter search using a known dataset and setup in a test environment, and applying to real-world with unknown datasets and distributions. To validate method's s ability to adapt unknown data in this scenario, we search for hyperparameters in CIFAR-10 Gaussian task setup and apply them to all other datasets and setups.
do not know the
For aL-SAR, we use memory-only training, where the training batch is retrieved from the episodic memory at every iteration. And we use the Greedy Balanced Sampling strategy (Prabhu 2020) for memory sampling. We use m = 4 for all datasets and setups, where m refers to the batch cycles where layer freezing not applied.
For LiDER (Bonicelli et al. 2022). it follows plug-in approach, integrated with existing methods. Based on the experimental results in the paper, the combination with X-DER showed the most promising performance. Consequently, the results obtained by combining X-DER were reported as the results for LiDER.
A.5 EXPERIMENT RESULTS USING ADDITIONAL MEMORY CONSTRAINTS
The results obtained using 13.46MB memory budgets in CIFAR-10 and CIFAR-100 are shown in Fig. 6 In addition to results in the 7.6MB memory budget in Fig. 3 our method outperforms other methods in all tested memory budgets, further showing that our method robust across various memory constraints.
A.6 COMPARISON BETWEEN ADAPTIVE LAYER FREEZING AND NAIVE LAYER FREEZING
We compare the proposed adaptive layer freezing method with various naive freezing methods, in both Gaussian and disjoint setup in CIFAR-10 10. The results are summarized in Tab Each freezing strategy chooses the number of frozen layers n E [0, L] where L is the total number of layers, so that when n 2 1, layer to layer are frozen. The compared freezing strategies
random freezing (n
randomly selected from [0, m every iteration for fixed n,, az E [0, L)), constant freezing (n
fixed initially) and linear freezing (n increases linearly from 0 to Am for a fixed am
[0, T.).
17

---

Published as a conference paper at ICLR 2025
Algorithm adaptive Layer freezing and Similarit--Aware Retrieval (aL-SAR)
1: Input model Sa. Layer parameters 0,, Training data stream D, Batch size B, Learning rate μ., EMA ratio a, Frequency scale k, Retrieval temperature T, Number layers L, Total Forward
FLOPs (FF). Backward FLOPs per layer (BF),
2: Initialize Episodic memory M <- O.. Sample frequency C < 0, Class frequency Cy < 0, Class Similarity S9192 < 0, Layer Fisher trace (trF), t- 0, Expected gradient norm 9xx; 4
3: 8g RandomSubset(8, 0.0005)
& Random subset of 8 containing 0.05% of the parameters, for updating class similarity S samples from data stream
4: for (3rt, we) D do
5: Update M + SreedyBalnncingSampler (M U Ve))
Memory update with Greedy Balancing Sampler
6: 2 + EyEy Syn Cy A (II. E M
Calculate effecciiv--us--freueee by Eq.
7: I = RandomChoice((M), B, softmax
Sample batch indices from memory
8:
KiM
B
Calculate decay rate
9:
Update a (1 r)c (1
Decay sample frequencies
10:
Update Cy + (1 (1-r) YyEy
Decay the class frequencies
Update G G +1 WieI
Increase sample frequency selected samples
12:
Update y) + C +1 ViEI
Increase class frequency for selected samples b Obtain training batch st Calculate loss
2t ((rs: Vi) Vi E
14:
[(=)) ((z,)) CrossEntropy(Joll)..))
9z, (2+) Var, [(z))
P Obtain gradient for last feature a4
16:
if 1% = 0 then
Update (trF)t
VI I....L
P Update Fisher every 4 batches
n² =
b No freezing When Fisher update
19:
else
(I/C), (FF)+ (BF),
Eienty (trF):
Vn 1, I.....I
Compute (I/C) by Eq.
19,; (=1)
21:
BFC(2)) = Σi= (BF): max (I/C)m
Σl (trF); Vn E
Compute (BFC) by Eq. > Determine optimal freezing
n2" argmax, argmax,BFC(2)).
23:
end if
Update Dori t (1-a) Dot, + a 19x; (2:)lp Update expected gradient norm for feature Use only unfrozen paramcters for updating similarity
25:
Os.n- 0g (0(n(+1.....))
Update Sycu: & (1 ())Synu + a CosineSimilarity (V/) L(zz), v(i 0s.n. [(z+)
26:
0s..=
VaI M.), (2r,, (1) E Z jj Update (m +l......) &- +l,.)) 4. V (n 41..... ((2))
Update class similarity using sample-wise gradients
27:
Update model except frozen layers
28: end for
Output fo
All layer freezing strategies contribute reducing computational costs. However, adaptive layer freezing has the least performance decrease. Note that the goal of layer freezing is not to freeze as much possible but rather save computational costs while preserving performance.
A.7 ABLATION STUDY
In addition the ablation study in the CIFAR-10/100 Gaussian task setup in Sec. 4.3 ablate the model to investigate the benefit of each of the proposed components in CIFAR- 10/100 Disjoint task setup and summarize the results in Tab.
Summing up the effect of the two components, our method outperforms the bascline while using fewer FL.PP than the baseline, each by noticeable margin.
18

---

Published as a conference paper at ICLR 2025
EWC -
ER
MIR
REMIND
DER
MEMO
OCS -
LIEER -
X-DER
aL-SAR (Ours)
AAuC
Aust
AAUC
Aast
20
80
GFLOPs/sampte
GFLOPs/samp
GFLOPs/samplc
(a) CIFAR- 10 Gaussian task setup
(b) CIFAR-10 10 Disjoint task setup
EWC
MIR
REMIND
DER+
MEMO
OCS
LIDER
X-DER
al-SAR (Ours)
40
AAuC
50
Ausz
AAuC
40
Aust
20
20
GFLOPs/sample
GFLoPs/sample
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-100 100 Gaussian task setup
(b) CIFAR- 1-0AR-1001A Disjoint task setup
Figure 6: Accuracy on the Gaussian and the Disjoint CL setup in CIFAR- 10 and CIFAR- 100 for various FLOPs per sample. aL-SAR outperforms all the compared CL methods. The memory budget is fixed to 7.6MB for both CIFAR-10 and CIFAR-100.
Gaussian
Disjoint
Methods
AAtI f
Ana
TFLOPs
AxTI f
Aus
TFLOPs
No Freezing
64.60=0.88
72.43+0.38
171.94
79.10+40.44
0.57
171.94
Random Freezing (Vamx
10.47.11.15 63.14
150.56
77.6910.51
69.30.1..77
150.62
Random Freezing (fhmx
32)
61.79-0.54
69.88 09.84
122.99
77.31
68.8910064
120.92
Constant Freezing (n 8)
60.91-0.80
67.70++0.83
147.12
74.99+00.4
65.89+0.50
147.12
Constant Freezing (u 16)
0.65
57.31110.90
109.48
67.64.10.61
55.59
109.48
Linear Freezing (Maax 16)
63.11-00990
7..00.11.04
150.64
77.53-10.47
68. 68.30-1.52
150.64
Linear Freezing (Amax = 32)
62.06-0.90
66.95+22.31
120.83
75.69+00.7
64.49+1.22
120.83
Adaptive Freezing (Ours)
64.38 0.32
72.57
146.80
79.75
70.70110.88
143.51
Table 5: Comparison between adaptive layer freezing and naive freezing in CIFAR-10. The memory budget is 7.6MB.
CIFAR-10
CIFAR-100
Methods
Aauc 1
Aus 1
TFLOPs
AALC 1
Aua
TFLOPs
Vanilla
77. 17.1080.58
70.26 0.91
163.73
43.59-40.86
38.64+0.35
245.85
Freezing
76.98-+0.15
70.58=0063
141.50
43.20++0.95
38.44+0.27
225.92
SAR
79.10:10.44
71.77=0.57
171.94
45.61::1.06
39.6810.66
257.91
SAR Freezing (aL-SAR)
79.75=0.38
70.70=0.88
143.51
45.00=+1.28
39.39:1062
228.14
Table 6: Benefits of the proposed components of our method in CIFAR-I0 and CIFAR-100 for disjoint task setup. SAR refers to our proposed Similarity-Aware Retrieval method. The memory budget is 7.6MB for CIFAR-I0 and 13. 13.44MB for CIFAR-100. CIFAR-10 10 We train for iter per sample for CIFAR-IO and 1.5 iter per sample CIFAR-100.
A.8 ABLATION STUDY OF SIMILARITY-WWARE-- RETRIEVAL (SAR)
We report the ablation results on the sub-components of Similarity-Aware Retrieval (SAR) in the Tab. | and Tab. X Applying only the use-frequency leads to lower performance than random re- trieval, since the samples from old tasks which accumulated high use frequencies in the past will be rarely used, causing severe forgetting past tasks. Applying discounted fie-frequency performs marginally better than random retrieval. Considering class similarity with effective use-frequne frequreyyncyy (i.e., Similarity- Aware Retrieval) further improves performance and outperforms random retrieval. For the ablation study, memory budget is set to 7.6MB for CIFAR- 10 and 13. 13.44MB for CIFAR- 100. CIFAR-10 10 We train for 1 iter per sample for CIFAR-10 and 1.5 iter per sample for CIFAR-100.
19

---

Published as a conference paper at ICLR 2025
CIFAR 10
CIFAR 100
Methods
AAUC
Ause
AAuC
Auss
Vanilla
60.16:10.11
70.08
31.97-10.89
37.
(+) use-urequue use-frequency
56.15+0..3
69.40+0.38
30.74++1.09
36.51+1.32
(+) discounted-uee use freeureuyyccy
62.72:40.18
71.10:0.53
33.76:10.67
39.29:+0.98 42.69:10.18
(+) effective-uue use frequenny (SAR)
64.60:10.83
72.43:10.38
37.60:10.40
Table 7: Ablation study Similarity-Awan Retrieval on Continuous setup. 'Vanilla" a simple replay-
based method that trains on randomly retrieved batches from a balanced reservoir memory.
CIFAR-10 10
CIFAR-10U
Methods
AAUC
Ause
AAUC
Ause
Vanilla
77.10=0..58
70.29:10.91
43.59:10.86
38.64::0.35
(+) use-freeupe nre-freuenccy
75.40:+0.36
69.77#1.11
42.11=0.81
37.83+20445
(+)
77.59+00.50
71.31+144
44.45-+0.85
39.38+0.72
(+) sffective-uee-rrequeee (SAR)
79.10+0.44
71.77+0.57
45.61+1.06
39.68+0.66
Table Ablation study of Similarity-Aware Retrieval Disjoint setup. 'Vanilla' a simple replay-based method that trains on randomly retrieved batches from a balanced reservoir memory.
A.9
EFFECT OF LAYER FREEZING IN AL-SAR
1.0
20
15
Accuracy
Cumulative TFL
10
0.4
freazing w/ freezing
0%
20000
30030
40000
50000
%
30000
40000
samples
# sammples
Figure 7: The effect using adaptive freezing aL-SAR. when training iteration per sample in CIFAR- 10 Gaussian task setup. The black dotted line is the linear trends from the first 10,000 samples
In Fig. we present the amount of FLOPs adaptive freezing method saves, under the same number of iterations. Our results demonstrate that adaptive freezing can substantial amount of training
FLOPPs up to 22% while maintaining accuracy. Also, considering the FL.OPs trends during the first 10,000 samples, represented the black dotted line, observe that the ratio of saved FLOPs increases with training progress. This shows that our freezing scheme can capture the training progress, enabling more layers be frozen once sufficient information has been learned. These observations align with the trends identified in Sec. 4.3 that the ratio of saved FLOPs increases as the total computational budget increases.
A. DETAILED DISTRIBUTION OF BFC ACROSS LAYERS
We plot the BFC values for selected layers and how they evolve during training in Fig. 8 Note that the figure is smoothed to show overall trends of BFC, and the actual values fluctuate depending on the input batch. We observe that (1) BFC values tend to stay negative on average. It indicates that freezing is not beneficial for most batches, because CL imposes small number of training itcrations and continuous streaming of new samples. However, for input batches with little information, BFC temporarily becomes positive to allow freezing. (2) At the task boundaries, we observe sharp drop of BFC values, which means that the layers are less likely to be frozen so that they can learn from the new, previously unseen data. This shows that BFC correctly handles the shift in informational contents, even though our method does not use any task boundary information. (3) BFC of earlier layers increases training progresses, and no longer shows significant drop of BFC values at the task boundaries. We believe this is because the earlier layers learn low-level features that are shared across different tasks.
20

---

Published as a conference paper at ICLR 2025
-2
BFC
Layer 10
Layer 20
-10
Layer 30
10000
20000
30000
40000
50000
# of iteration
Figure 8: BFC of layer 10, 20, and 30 in ResNet-32 on CIFAR- --AR-0 disjoint setup. The red dotted lines indicate the task boundaries.
A.11 EXPERIMENTAL RESULTS ON TASKS FROM VARIOUS DATASETS
We compare alRR with baselines on 5-datasets Waag et al. 2022b), which consist five distinct datasets, demonstrate its generalizability in Fig. / In the figure, we can see that the x-coordinates of the aL-SAR points are shifted to the left compared the x-coordinates of the points from other baselines, indicating that less computation was used due to freezing. Note that as the training is done with a higher computational budget, amount of information that can learned from the training data relative to the computation decreases. Therefore, the freezing rate increases. In the rightmost point, it uses approximately 20% less computational cost while achieving better performance than other baselines.
the task of continual learning. We report the results
ER
MIR
DER+
MEMO
OCS
al SAR (Ours)
60
GFLOPsisample
GFLOPs/sampl
Figure Comparison of baselines 5-datasets.
A.12 COMPARISON WITH SOTA EFFICIENT CL METHODS
We compare aL-SAR with with existing SOTA efficient CL methods and summarize the results in Fig. 10 TriRE incorporates distillation through Exponential Moving Average (EMA) model, necessitating extra forward computation and memory allocation for storage. Consequently, within total-constrained setup, TriRE retains fewer samples in episodic memory. As result, TriRE exhibits lower overall performance compared to aL-SAR and SparCL. Especially, TriRE has a significantly lower AAUC than Auss- This is because AAUC not only includes inference performance of task but also incorporates anytime inference performance. Specifically, in TriRE, the inference performance at points where all three stages of TriRE, namely Retain, Revise, and Rewind, are not completed tends to be low. Note that both TriRE and SparCL rely task boundary information, while aL-SAR does not depend on task boundary information, leading to practical utilization in real-world scenarios.
the end
A.13 DETAILS ON TRAINING MULTI-MODAL LARGE LANGUAGE MODELS
Datasets. Beyond the concept-ineremental learning (MCIL.) setup. Specifically, we evaluate the Bongard-HOI (Jiang al.,
learning (CIL) setup, we extend aL to the multi-modal
21

---

Published as a conference paper at ICLR 2025
(Durs)
SparCL_0.50
SparCL_0.75
SparCL_0.30
THIRE
60
Asuc
AuesT
Axuc
70
Arust
60
60
40
GFLOPs/sample
GFLOPs/sample
GFLOPslsample
GFLOPs/sample
(a) CTFAR-10 Gaussian task setup
(h) CIFAR-10 Disjoint task setup
AL-SAR (Ours)
SparCI_a.50
SparC1_00775
SparCI_0.50
TriRE
40
Aauc
AuoTT
30
Aauc
Auset
20-
30
20
GFLpPs/sample
GFLOPa/sample
GFLOPs/sample
GFLOPs/sample (b) CIFAR-100 Disjoint task setup
(a) CIFAR-100 Gaussian task setup
Comparison with efficient methods under both memory and computation constrained setup. SparCL, the underscore represents sparsity. SparCL. exhibits its best performance when the sparsity is set to 0.75.
Figure 10:
2022) and Bongard- Seggard-OpenWorlee (Wu et al. 2024) benchmarks, which feature multiple 'concepts'
(e.g.. ride a bike, The top of a snow-covered mountain) in each benchmark. We split the concepts into 5 disjoint tasks for the MCIL setup. These benchmarks leverage two key attributes of traditional Bongard problems (Depeweg et al. 2018): (1) the capacity for few-shot concept learning and (2) reasoning that is dependent a limited number of examples, while the latter suggests that the classification differ based the context provided (i.e., the positive and negative support sets). In the Bongard problem, with a positive support set and a negative support set, we tackle two specific tasks: (1) Which concept is exlusively represented by the positive support ser? and (2) For a given query image, does it belong to the positive or negative support set? We denote these tasks as CA (Concept Answering) and P/N, respectively.
context. The former entails the ability to derive visual concepts from
query image can
Metrics. For the Positive/Negative (P/N) task, where answers are either Positive or Negative, we use accuracy the evaluation metric. Similarly, for the Bongard-HOI CA task, which has simple text answers (e.g., kick ball), we also use accuracy the metric. In contrast, for the Bongard- OpenWorld-CA task, where answers are full sentences, we compare the generated sentences from the MILM model with ground truth sentences. Specifically, following Liu et al. (2023): Yu et al. (2024), we use GPT-3.5 to assess response quality, evaluating helpfulness, relevance, accuracy, and detail. It provides overall score from to 10, with higher scores indicating better perfor- mance. We provide details on the prompts used for GPT-4 evaluation Sec. [A.14] Similar to class-incremental tasks, also measure the area under the curve (AUC) performance and the final performance within this multi-modal framework.
Implementation Details. For the MCIL setup. assume infinite episodic memory size and use the Adam optimizer with learning rate x 10 and a Constant LR scheduler. The number of images in the support set is for each positive and negative support set in Bongard-HOl and Bongard-Open World. We use the L..VA--15-77 model and train it on NVIDIA A 1000 80GB GPUs.
Hingard-HO1-Ce
Seerand-OpenWoriddee
Methods
AAUC
Auat
TFLOPs
AAUC f
Aus 1
TFLOPs
+
freezing
24.7
72.41+22.02
788.35
73.72++0.49
73.26-40.56
1478.78
al. (Ours)
75.27+2.15
72.31+3.26
715.43 (-9.25%)
73.06+1.15
73.92++2.51
1401.28 (-5.24%)
Table 9: The effect of adaptive freezing on MLLM training. We use ALaVA-1.55-7 model with LoRA.
22

---

Published as a conference paper at ICLR 2025
In Tab. in Sec. 4.2 and Bongard- Longard-OpenWWeee WoeenWord-P/N tasks.
empirically demonstrate the effectiveness of aL in the Bongard- -OO-PP
Effect of aL in CA tasks. In addition to the P/N tasks presented in Table 3 we also apply aL the CA tasks and summarize the results in Table 9 Similar the P/N tasks, aL effectively reduces training costs in CA tasks. However, compared to the relatively simple P/N tasks, which involve binary classification, the CA tasks require generating text or sentence-level answers, making them more complex. As a result, aL leads to relatively smaller reduction in FLOPs in CA tasks, compared to P/N tasks. It shows that al. adaptively freezes fewer layers for more complex tasks that require more training.
Notably, our proposed adaptive layer freezing method operates independently of downstream tasks, model architecture, and task identity information, thus we expect it to be applicable across wide range of tasks and model architectures.
A.14 DETAILS OF PROMPTS USED FOR GPT-3.5 EVALUATION
We use GPT-3.5 to assess the quality of predicted sentences against ground truth sentences using the following prompts:
Ground truth]: 'Ground-Truie Truth sentence'
[Prediction]: 'Prediction sentence'
Compare the ground truth and prediction from Al models, to give correctness score for the prediction. The user asks the question about observing image multiple images. The images replace <image> tokens in [Input]. Please rate the relevance and accuracy of [Assistant]] compared to [Ground-Trutee The assistant receives an overall score on a scale of 1 (totally wrong) to 10 (totally right), where a higher score indicates better overall performance. Please first provide comprehensive explanation of your evaluation explanation. In the subsequent line, please provide single line containing the score for Assistant, in the format of Score: <an integer value scale to 10>
A.15 COMPARISON BETWEEN AL AND OTHER FREEZING METHODS
In addition to the quantitative comparison of aL with freezing methods in the continuous setup, we also compare them in the disjoint setup. The results are summarized in Tab. [O] Similar to the continuous setup, aL effectively reduces the training FLOPs in disjoint setup while maintaining per- formance. contrast, REMIND (Hayes et al., 2020). PTLF YYng et al. 2023), and EGERIA (Wang et al. 2023) result in negligible reductions in training costs, indicating less freezing. The reasons are as follows: (1) EGERIA freezes layers where the difference between the outputs of the current model and the reference model has been marginal over recent iterations. The reference model is periodically updated using the training model to evaluate layer plasticity. This approach can be ef- fective in joint training scenarios, the training loss typically converges zero over time, causing the layers to stabilize. However, in online CL, training a current model with new data can cause the model diverge from the reference model, resulting in less effective layer freezing. (2) PTLF freezes layers with top-k task correlation ratio. However, since it freezes intermediate layers rather than layers to n, it cannot fully reduce the backward FL.OPs of the frozen layers the gradients for layer input should still be calculated for backpropagation.
CIFAR-10
CIFAR-100
Methods
AauC
Aus
TFLOPs
AACC
Aus
TFLOPs
No freezing
79.10
0.44
71.77
0.57
171.94
53.38 1.13
49.43 + 0.72
515.82
REMIND
73.20
0.67
62.95 1:955000 0.87
151.31 (-12.0%)
48.14 1.30
41.28 + 0.14
453.92
(-12.0%)
PTLF
79.29
0.36
70.50 0.43
164.38 (-4.4%)
53.22 1.07
49.25 + 0.65
497.77 (-3.5%)
EGERIA
78.70 + 0.54
70.72 0.68
169.52 (-1.4%)
52.95 1055100 0.77
48.88
+ 0.46
502.41 (-2.6%)
al. (Ours)
79.75 + 0.38
70.70 + 0.88
143.51 (-16.5%)
1.21 53.33
48.91 + 0.87
425.65 (-17.5%)
Table 10: Comparison between proposed adaptive layer freezing and other freezing methods. We compare them in both the CIFAR-I0 disjoint setup and the CIFAR-100 disjoint setup. train for iter per sample for CIFAR-10 3 iter per sample for CIFAR-100.
23

---

Published as a conference paper at ICLR 2025
A.16 APPLYING ADAPTIVE LAYER FREEZING TO ATTENTION BASED MODEL
We investigate the effect of the proposed adaptive layer freezing not only in ResNet but also in attention-based models such as the Vision Tra)sformer(ViT)) (Dosovitskiy al., 2020). We com- pare the freezing effects for both ViT-base and ViT-large models, considering those pretrained on ImageNet-IK: IK and those trained from scratch. The results are summarized in T1 and Tab. [2 respectively.
When using a pretrained model, the adaptive layer freezing reduces the computational cost by nearly 15% with minimal impact on AAuC and Alast, compared to the Vanilla training without freezing. Since pretrained models have already been sufficiently trained on large dataset, the amount of information that the model will learn from the training data may be relatively small compared to training from scratch. Thus, leads to the freezing of many layers by adaptive layer freezing. This not only reduces computational costs but also ensures high performance, since the model is updated only in truly informative batches, thus preserving the advantages of pretrained initialization.
In the case of training from scratch, the decrease in TFLOPs is significantly small compared to using a pretrained model, which implies that the layers did not freeze much. This is due to the large model capacity of ViT and the small number of training iterations in online CL. which leads severe underfitting of the model when training from scratch. Specifically, Tab. [T that when training from scratch, the accuracy only reaches around 30% even at the end of the training (Atast). Thus, since the model is not sufficiently trained yet, the adaptive layer freezing scheme tends to freeze fewer layers so that the model can learn more information. This shows that the proposed adaptive freezing method can indeed provide a reasonable freezing strategy.
CIFRRI
CIEAR HI)
Aus:
Auc
Aua
Aue
Auue
Aue.
Aux
Aui
TILLEN
Adlphh Powolay
56.20 1.88
607
3.46642
30,41 4.22
41 2.66
46.58 1,71
5234385
21.33 4.54
24.8 (1.1
5828.75
Table 11: Effect of layer freeznng in ViT-base. We used CIFAR-10 and CIFAR- 100 as the dataset, Gaussian Setup for setup. The memory budget is 334.76MB for both CIFAR-10 10 and CIFAR- 100.
Dajaar
Gazmia
Metbods
Pustsised
Frocs
Pretuainal
Seatch
Auce
Al
TFL.0%s
Aue
Awe
TRLOPs
Avre
Alw
TRL.Ps
Auy
An..
TRLOP
Varilla
14316,37
41.37=0.75
1952
14.366.7
60.11:44:
143100755
24 34
34
Alligtiee Feceinng
70776:4.00
58.44:486
11,962.00
41.3:1.89
2L19:190
13,90264
6416::200
11,366.0
28.93-208
14817.93
Table 12: Effect of layer freezing ViT-Large. We used CIFAR-10 and CIFAR- 100 as the dataset, Gaussian Sctup for setup. The memory budget is 1.20GB for both CIFAR- 10 and CIFAR-100
A.17 APPLYING AL-SAR ACROSS DIVERSE NETWORK ARCHITECTURES
aL-SAR can be applied to any feedforward neural network, as long as layers can be defined, in- cluding CNNs and Vision Transformers (Dosovitskiy et al., (020)) as shown in Sec. A.16 Note that since our layer freezing methods require evaluating the information gained and FLOPs used by individual layers, network should be dissected into layers.
CIRAR-10
CIPAR- 100
Dijeknt
Gansiai
Aue
Aux
TFLOP>
Auge
TELOPs
Aape
TELOPs
Aux
Aue
TELOP:
Houelino
506
11
516
35 10----1
KI
75081
(Can))
Table 13: Comparison between Baseline and al. on Disjoint and Gaussian in CIFAR- 10 and CIFAR-100 with naive 8-CNN layers. The baseline refers removing the two components of aL-SAR: similarity- retrieval and adaptive layer freezing.
To analyze the effect of the number of layers in aL-SAR, we first conduct experiments with ResNet- 20 and ResNet-56 on CIFAR-10 and CIFAR- 100, in addition to ResNet-32 and ResNet-18 as used in the main results. As shown in Fig. 11 and Fig. [2 aL-SAR consistently outperforms baseline, which refers to removing two components of aL-SAR: adaptive layer freezing and similarity- aware retrieval, irrespective of the number of layers. We also compare SAR with other CL. baselines
24

---

Published as a conference paper at ICLR 2025
CHARR-O
CIFAR-10
Model
Dijint
Gausson
Digont Aue
Chansian
Auce 1
Aue
Aue
Ale
TRLOPs
TELOPS
Juce
Aue
Bouiito
64
10.56
1296-1
01-01-046
123611
4098-118
11 11-66---K
1896
25
1854-7 1.51
al-SAR (Tun)
7941 8.34
7X62
1.31
102536
62.7
(873
74622 2,46
1002222
46.22 1.1
30.70
BLIX
1631.7
NL.66
1.04
41.35
1679.66
Table Comparison between Baseline alR on Disjoint Gaussian in CIFAR- 10 and CIFAR-100 with naive 16- (NN layers. The baseline refers to removing the two components of aL-SAR: similarity aware retrieval and adaptive layer freezing.
aware
Digait Aus
Cuussin Aua
Digiint
Gansian
Aus
TELOEs
Aue
TELOR
Aue
Aua
TELORs
Aux
Aue
TELO0s
al-SAR (Gam)
T3442 6.41
0..35
1394.3
$3.21 331
6699 3377
2815.75
29,80 8.96
2088
1.80
9918.97
24,68
210
20,9
2,65
9948566
Table 15: Comparison between Baseline and aL--SAR on Disjoint and Gaussian in CIFAR-IO and CIFAR-100 with naive 32-CNN layers. The baseline refers to removing the two components of aL-SAR: similarity-aware retrieval and adaptive layer freezing.
and summarize the results in Fig 13 and Fig. [4 respectively. aL-SAR consistently outperforms other methods in both shallower and deeper networks, showing that aL-SAR is robust in various model sizes.
Moreover, we perform additional experiments on the naive CNN without skip connection, i.e., con- sisting only of convolution, batch normalization, activation, and fully connected layers. We report the result in Tab. 13 Table Tab. [4] and Tab. [5] for 8-layer, 16-layer, and 32-layer CNNs, respec- tively. the table, aL-SAR improves performance and reduces computational cost compared to baseline also when using simple CNN, regardless of the number of layers. Note that a deeper CNN shows lower performance due to the absence of skip connections, as reported in [1] (ResNet).
However, our proposed adaptive layer freezing cannot apply to recurrent neural networks (Sherstin- sky. 2020) since the gradient of layer affects not only the preceding layers but also the subsequent layers (Rotman & Wolf (202I) while in the feedforward network, the gradient of layer influences only the preceding layers.
(Ours)
83
AAuI
Auost
AAUC
Arast
54
GFLOPsisample
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-10 Gaussian task setup
(h) CIFAR-10 Disjoint task setup
(Ours)
51-
52
RAIC
96
Aust
48
Aust
75
33
42
49
50
39
27
39
3
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 11: Comparison of the baseline and aL-SAR on Gaussian and Disjoint CL setup in CIFAR-10 and 100 with ResNet-20. The baseline refers to removing the two components of aL-SAR, i.e, adaptive layer freezing and similarity-aware retrieval, from AL-SAR itself.
25

---

Published as a conference paper at ICLR 2025
(Ours)
83
61
AAui
Apast
AAuC
Auast
78
76
57
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs'sample
(a) CIFAR-10 Gaussian task setup
(h) CIFAR-10 Disjoint task setup
(Ours)
AAVC
Ausst
Asuic
Aust
46
53-
GFLOPs/sample
GFLOPstsample
GFLOPs/sample
GFLOPs/samplc
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 12: Comparison of the baseline and aL-SAR Gaussian and Disjoint CL setup in CIFAR-10 and CIFAR- 100 with ResNet-56. The baseline refers to removing the two components of aL-SAR, i.e., adaptive layer freezing and similarity-aware retrieval, from aL-SAR itself.
aL-SAR (Ours)
80
80
Aauc
Abast
AxuC
Auset
70
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-10 10 Gaussian task setup
(b) CIFAR-10 Disjoint task setup
X-DER
(Ours)
50p
50
Azuc
30
Aust
Auaic
Aust
30
20
GFLOPs/sample
GFLoPs/sample
GFLOPs/sample
GFLOPs/sample
(a) 1IFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 13: Comparison of CL methods on Gaussian and Disjoint CL setup in CIFAR-10 and CIFAR- 100 with 2esNet-20.
A.18 DETAILED INFORMATION ON THE CALCCUATIONN OF Sy..us
We calculate the cosine similarity of the gradients between pairs of samples in the training batch and update the EMA estimate of class-wise gradients, which are then used for calculating Sy,y To further reduce the computational cost of calculating similarity, we use only 0.05% of the model parameters for the calculation of similarity, since the gradient distribution of the subset of randomly selected weights similar to the gradient of the entire weight set (Li et al. (022).
Specifically, randomly select 0.05% of the model parameters across all layers before training. During training, we use the gradients of the mre-elected, which are unfrozen by adaptive layer freezing, update the class-wise gradients. Specifically, we employ EMA estimation of the class-
26

---

Published as a conference paper at ICLR 2025
(Ours)
AAuc
Auast
Aauc
Arust
70
70
60
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs'sample
(a) CIFAR-10 Gaussian task setup
(h) CIFAR-I0 Disjoint task setup
(Ours)
50
40
50
Aguic
Aust
AAucC
40
Aust
20
20
GFLOPs/sample
GFLOPs/sample
GFLOPs/sampl
GFLOPs/samplc
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 100 Disjoint task setup
Figure 14: Comparison of CL methods on Gaussian and Disjoint CL setup in CIFAR-10 and CIFAR-10M with ResNet-56.
wise gradients. updating them every batch iteration using the gradients of selected parameters in an EMA manner. While not all eet-mssimated class-wise gradients are updated every iteration (since gradients of frozen layers are not computed), all elements the estimated class-wise gradients are updated every m iterations, as we unfreeze all layers every m iterations, mentioned in Sec. Sec.[A.T To form the similarity matrix, we first compute the element-wise product of class-wise gradients and then average them over all dlasswwise pairs.
A.19 COMPARISON WITH RETRIEVAL-BASED CL METHODS
To train models with informative training batches, recent studies propose retrieving batches from episodic memory, such as MIR (Aljundi et al.) (019a) and ASER (Shim al.) [202T). However, calculating the loss in MIR and the adversarial shapely value in ASER requires additional forward and backward processes, leading to high computational requirements. similarity-awart retrieval methods only utilize the gradient vector, which can be obtained naturally
contrast, our proposed
during the training process, incurring no additional cost. Therefore, as depicted in Fig. [5] aL-SAR outperforms MIR and ASER under fixed computational budgets.
A.20 EFFECT OF TEMPERATURE T
Temperature is selected by a hyperparameter search on CIFAR-10. The lower the temperature, the more the retrieval focuses on samples with low effective neefrreuuncy, enabling a faster gain in knowledge. However, since need diverse samples to maintain accurate estimate of similarity, a too low temperature would also hinder the performance. Thus, we select an adequate temperature via hyperparameter search. We report the result of the hyperparameter search in Fig. [6] Note that while a too high or low temperature results in a diminished performance, there is a wide range of temperature values that show stable performances.
A.21 DETAILS ABOUT MEASURING FLOPS
With the exception of the simplest baseline, ER, the other baseline methods involve additional com- putational costs. REMIND, MIR, OCS, X-DER, and MEMO require additional forward/backwwrd processes of the model, resulting in notable increase in the relative FLOPs compared to ER. On the contrary, for DER, EWC, and aL-SAR, additional computational cost is neglible as they do not involve model forward and backward processes ie., the relative FLOPPs is approximately 1. We
27

---

Published as a conference paper at ICLR 2025
(Ours)
70
86
AAuc
60
Auast
70
Aauc
Avast
70
00
80
GFLoPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-10 Gaussian task setup
(h) CIFAR-10 Disjoint task setup
(Ours)
40
50
50
AAuc
Aust
AAuc
40
Aust
20
20
20
30
GFLOPsismmple
GFLOPs/sample
GFLOPs/sample
GFLOPs/sampic
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 15: Comparison of various retrieval-basee CL methods on Gaussian and Disjoint CL setup in CIFAR-10
and CIFAR-100.
Effect of Temperature
65
74
73
81
Aauic
Alast
60
71
58
70
Temperature T
Temperature T
Figure 16: Experiment for the effect of temperature T in the CIFAR-10 Gaussian scheduled setup. We ResNet-32 as backbone
compare the types of additional computational costs and relative flops compared ER (Rolnick et al. 2019) for each method in Tab. [6
Although aL-SAR requires various types of computation, these amounts to significantly fewer oper- ations compared to the costs incurred during model forward and backward passes. In addition, since aL-SAR freezes layers depending on the input batch, we provided a range for backward FLOPs. Note that Remind incurs additional costs during base initialization to convert all memory data into features and perform product quantization. However, since this process occurs only once during the model training process and not in every iteration, it is not included in the table.
A.22 EXPERIMENTAL RESULTS IN MEMORY INFINITE SETUP
aL-SAR demonstrates even stronger performance with an unlimited memory budget setup in various benchmarks such CIFAR-I0, CIFAR-100, and ImageNet-IK, as shown in Fig. 17 We believe because the effect the retrieval strategy is more significant with unlimited memory than with limited memory. Provided that the retrieval strategy successfully distinguishes useful samples, retrieving from the full past data (unlimited memory) is likely to yield more useful samples than retrieving only from small portion of past data (limited memory). Thus, strong performance in the
28

---

Published as a conference paper at ICLR 2025
Type Compusaiin
FLoPstanple (TFLO0))
R.cesisppll (TRLON))
Forsonl
Mourendd by pelope
0-57
Bockrood FEXIIN
Mrsserd by pilppa
1.
(A pesmesn)
Fercani FLAIPs
Moasrrat by pillps
0.7
Buckoaad FLUIS
Mazerzdd pillas
1.14
Cétralaice Tirtitatiinn
(4)
1.0% 10-11
Peeeant UELAIR
Mearered by pitlons pthlep GN pasmmiii)
0.57
X.DER
Model (Astur)
xurt
3.99
233
Angrsorations
Hetnerting bapsl with Deterrzt Lousing
xx (Forwart 11 DPO)
2.78
Foroard FLOPs
Maosuzi ptlgg Movernd by (eflis 12 14 pauncim))
0.57
Made (pshtte(Atkss) Fieerut
Teckloond FININ
3.42
11
1.21
Forvord FLOPS
Massraalyy by pellps
0:57
BR-MIK
Model pulatc (Adles)
12 GN pusunzio))
5a2 xuu
7:08
4.12
Forword ES TIN Fanditions
1.78
Backourd URLIP Lancidane
3.56
Buckoaad ELOPS
Maosrzd by pllgo 13 x pasmers)) (P toka)
1.14
MENO
Madel peberccatorr)
2.55
1.40
Foreaz lie Expanded Neseak
0.15
DG7
Forword LIP
Mossrad by pllpss pfflpg 12xx ponotesin))
0:57
Madel Updhts (hdee))
LIDER
Auxmoraloon Contranive Leasint
Forwarinng Teyat with Different
6x (Damwant TLD00
2:28
3.9
233
Cakelation Lipedite Conttrr
x22 (abapdl)-1)
Forward FLOPS
Massual by ptlgg 12 66 paunck))
1.14
CUL-DC
Madol Upaltecchazz)
5.52 =10
7:4
4.06
Angrontancee
Rawasiing layah witi Diffrratt Leoning
9% (Porvad ELOP)) (Budovad FLOPS
6.27
Ferount PT STP
Moseand by pefope
0:47
Buckooard ELOPS
Manerzily by pilloss
1.14
CAMA
L8 permeon))
1.0 10-11
Lags Uppearng
5% (c) zbear))
1.5 11
Backwardd (2etor hes intidintioo:
by peflggs
Maaerrdd 12 (N panstcion)
1.14
Middl Uppaioo (Acaes befors bas sutaiiance)
5,32 x211
Beckoaad RLOPs ialke hac inttiiatioon
677 (Badoad ELD in mnsiz))
0.88
1.32
62
Modd Uydkiee (Adani alia bes sateiiatios
0.75 OMrecl uplits fo whole modill
5.52 -jin
Backward FLUPS
Mewared pillas (dc2osds frceiiu
82x 10
Model (pstace
2. (8 pazatcknn)
5.32 x1ir1
ALSAR
Calodations Classeise sisilaity
(b pazamaten)) <1000 5)
5.6
0,50
0
Caleiaaion of Fidter Informaiion
2 (#) pasazaann 1.3 layen)
x u0
Calcaldion of BPC
(e lapen x S katzz)) x
3.5 10
Calcalaion of Rutrival Pubbiiiiity Progency Uppdase
damas?? deno))
Table 16: Details the additional computational budget. To measure forward/hackwwrd FL.OPs of the model, we use ptflops which is a widely used Python library to calculate FLOPPs FLOOPs from other operations were manually calculated.
unlimited-memory setup indicates that our similarity-awar retrieval effectively distinguishes useful samples.
Interestingly, aL-SAR freezes fewer layers in the unlimited memory setup than the limited memory setup. In the unlimited memory setup, the model can learn more knowledge from the samples stored in episodic memory than in the limited memory setup, so our adaptive freezing scheme chooses to freeze fewer layers so that the model can acquire more knowledge. It shows that our adaptive freezing method works also in the unlimited memory setup without the need to modify.
A.23 DETAILS ABOUT THE MEMORY BBUDGET IN TOTAL-CONSTRAINED CL
The memory budget allocated to episodic memory, model parameters, and additional memory costs specific to each CL algorithm, such as classwise similarities and logits. In this section, B denotes the additional memory budget, S(BB) denotes the size of the additional memory budget (in MB), & denotes episodic memory, and [8| represents the number of stored instances in E
In our total-constrained setup, the memory budget is restricted to the cost of storing 7.6MB, 13.44MB and 25. MB in CIFAR-10/100. Since storing the ResNet-32 model parameters requires memory cost equivalent to saving 603 instances of CIFAR- 100 images (463,504 floats x bytes/fioat (3 x 32 x 32) bytes/image R 603 instances), for methods that store the model for distillation or
29

---

Published as a conference paper at ICLR 2025
REMIND
0C5
LIDER
X-DER
SUR (Ours)
85
Aauc
60
Alast
AxuIC
A
38
GFLOPs/sample
GFLOPs/sample
GFLOPs'sampl
GFLOPs/sample
(a) CIFAR-10 Gaussian task setup
(h) CIFAR-I0 Disjoint task setup
REMIND
LIDER
X-DER
SL. (Ours)
80
Aauc
-0
AxuC
40
Arset
30
20
GFLoPsismmpl
GFLOPs/sample
GFLOPs/sample
(a) CIFAR-100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
ER
MIR
REMIND
DER+
MEMO
OCS
LIDER
X-DER
aL-SAR (Ours)
50
50
30
AAuC
20
Asut
AAuC
Aust
30
10
20
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
GFLOPs/sample
(a) ImageNet-IK IK Gaussian task setup
(b) ImageNet-IK Disjoint task setup
Figure 17: Atost and AADe Gaussian and Disjoint CL setup in CIFAR-I0 and CIFAR-IO1 for wide range FLOPs per sample with infinite memory budget. employ ResNet-32 CIFAR- 10/100 and ResNet- 18 for ImageNet- IK as a backbone.
regularization, we subtract the memory cost of the model parameters from the episodic memory size (Zhou et al.] 2023). In ImageNet and CLEAR-10, we use the ResNet-18 model and apply the same policy of subtracting model parameters and logits from the memory budget mentioned
above.
ER does not require additional memory beyond episodic memory. Similarly, MIR (Aljundi al, 2019a) and ASER (Shim et al. 202T) do not require additional memory despite being computation- ally heavy.
On the contrary, EWC KKikkpatrick et al. 2017) requires storing the previous model parameters and the wase Fisher Information(FI) for all parameters. Therefore, subtract the memory cost of storing two models from the episodic memory size. Similarly, BiC (Wu et al. 2019) also stores the previous model for distillation, | was reduced much as the size of the model. For example, with a total memory budget of 7.6MB and ResNet-32 model type in CIFAR-100, ER can store up to 2000 instances in E, while EWC limited to storing only 794 (= 2000 2x603) instances. Similarly, BiC can store only 1397 (= 2000 - 603) instances in È
Some methods incur additional memory costs other than episodic memory or model parameters. We handle such costs in a similar way by reducing the episodic memory size by the number of samples equivalent the additional memory cost. For example, DER (Buzzega et al., 2020) uses the previous logits of the samples for distillation, we subtract the cost of storing the logits from the episodic memory size. More specifically, DER needs additional storage which size is [ x 4 bytes/float, where di denotes logit dimension, which is 100 in CIFAR-100 and 10 in CIFAR-10 10.
30

---

Published as a conference paper at ICLR 2025
aL-SAR stores similarities between classes, the training frequency of each sample, and the trace of FIM for each layer. AR needs 400 bytes 4 bytes/float x10² for saving class-wise similarities, bytes/int x|E| for saving frequency of each sample, and 4 bytes/float Xn, for saving trace of FIM for each layer, where ny is total number of layers. However, such additional memory cost is negligible compared to episodic memory model parameters (only 0.1 of memory budget). We summarize implementation details of the total memory budget for each dataset in 17 and Tab
Tab Tab 19 Tab 20
Methods
B Type
S((B)) )
[E]
S((E))
Model Type
Parameters
Model Size
ER
2,000
5.85MB
Resnet32
0.46M
1.76BB
REMIND
Feature replay Logits
5.85MB
Resnet32
0.46M
1.6MBB
DER
0.08MB
1,974
5.77MB
Resnet32
0.46M
1.76BB
ER-MIR
2,000
5.85MB
Resnet32
0.46M
1.6MBB
EWC
FI Previous Model
3.52MB
2.33MB
Resnet32
0.46M
1.76BB
OCS
2,000
5.85MB
Resnet32
0.46M
1.76MB
X-DER
Logits Logits
0.08MB
1,974
5.77MB
Resnet32
0.46M
1.76BB
LiDER
0.08MBB
1,974
5.77MB
Resnet32
0,46M
1.76MB
MEMO
Expanded Network
3.52MB
794
2.33MB
Resnet32
0.46M
1.76MB
CAMA
Logits
0.08MBB
1,974
5.77MB
Resnet32
0,46M
1.76MB
Teacher model for
CCL-DC
1.76MB
1397
4.08MB
Resnet32
0.46M
1.76MB
collaborative learning
Class-wise similarity & frequency of each sample
aL-SAR
8.52KB
1,997
5.84MB
Resnet32
0.46M
1.76MB
Table 17: Implementation details of total memory budget=7.6MBB in CIFAR-10
Methods
B Type
S((B)
[E]
S((E))
Model Type
Parameters
Model Size
ER
2,000
5.85MB
Resnet32
0.46M
1.76BB
REMIND
Feature replay
5.85MB
Resnet32
0.46M
1.76BB
DER
Logits
0.71MB
1,770
5.14MB
Resnet32
0.46M
1.76BB
ER-MIR
2,000
5.85MB
Resnet32
0.46M
1.7MBB
EWC
FI & Previous Model
3.52MB
794
2.33MB
Resnet32
0.46M
1.7MBB
OCS
2,000
5.85MB
Resnet32
0.46M
1.76MB
X-DER
Logits
0.71MB
1,770
5.14MB
Resnet32
0.46M
1.76MB
LiDER
Logits
0.71MB
1,770
5.14MB
Resnet32
0.46M
1.76MB
MEMO
Expanded Network
1.33MB
1,542
4.51MB
Resnet32
0.46M
1.76MB
CAMA
Logits
0.71MB
1,770
5.14MB
Resnet32
0,46M
1.76MB
Teacher model for
CCL-DC
1.76MB
1397
4.08MB
Resnet32
0.46M
1.76MB
collaborative learning
Class-wise similarity & frequency of each sample
aL-SAR
0.05MB
1,987
5.83MB
Resnet32
0.46M
1.76MB
Table 18: Implementation details of total memory budget=7.6MB in CIFAR-100
Methods
B Type
S((55)
S((E))
Model Type
# Parameters
Model Size
ER
4,000
574.0MB
Resnet32
0.46M
1.76MB
REMIND
Feature replay Logits
574.0MB
Resnet32
0.46M
1.76MB
DER
0.IMB
3,999
573.9MB
Resnet32
0.46M
1.7MBB
ER-MIR
4,000
574.0MB
Resnet32
0.46M
1.76MB
EWC
Previous Model
4.0MB
3,972
570.0MB
Resnct32
0.46M
1.76MB
OCS
4,000
574.0MB
Resnet32
0.46M
1.76MB
X-DER
Logits
0.IMB
3,999
573.9MB
Resnet32
0.46M
1.76MB
LiDER
Logits
0.IMB
3,999
573.9MB
Resnet32
0.46M
1.76MB
MEMO
Expanded Network
1.4MB
3,990
572.6MB
Resnet32
0.46M
1.76MB
CAMA
Logits
U.IMB
3,999
573.9MB
Resnet32
0.46M
1.76MB
Teacher model
CCL-DC
1.76MB
3,987
572.2MB
Resnet32
0.46M
1.76MB
collaborative learning
Class-wise similarity frequency each sample
aL-SAR
0.IMB
3,999
573.9MB
Resnet32
0.46M
1.76MB
Table 19: Implementation details of total memory budget=574.4MB in CLEAR-10
31

---

Published as a conference paper at ICLR 2025
Methods
B Type
S( Bl)
[C]
S((C1)
Model Type
# Purameters
Model Size
ER
8,000
1,148.0MB
Resnet32
0.46M
1.76BB
REMIND
Teature replay Logits
1,148.0MB
Resnet32
0,46M
1.76BB
DER
3.2MB
7,978
1,144.8MB
Resnet32
0.46M
1.76BB
ER-MIR
8,000
1.148.0MB
Resnet32
0.46M
1.76MB
EWC
F & Previous Model
4.0MB
7,972
1,144.0MB
Resnet32
0,46M
1.76MB
OCS
8,000
1,148.0MB
Resnet32
0.46M
1.76MB
X-DER
Logits Logits
3.2MB
7,978
1,144.8MB
Resnet32
0.46M
1.76BB
LiDER
3.2MB
7.978
1,144.8MB
Resnct32
0.46M
1.76MB
MEMO
Expanded Network
1.4MB
7,990
Resnet32
0,46M
1.76MB
CAMA
Logits
3.2MB
7,978
1,144.8MB
Resnet32
0.46M
1.76MB
Teacher model for
CCL-DC
1.76MB
7,987
1146.2MB
Resnet32
0,46M
1.66BB
collaborative learning
Class-wise similarity frequency each sample
aL-SAR
0.IMB
7,999
1,147.9MB
Resnet32
0.46M
1.76MB
Table 20: Implementation details of total memory budget= 149.811499 in CLEAR- 100
Methods
8 Type
S((B))
S((C))
Model Type
Parameters
Model Size
ER
40,000
5,740.0MB
Resnet18
11,17M
42.6MB
REMIND
Feature replay Logits
5,740.0MB
Resnet18
11.17M
42,6MB
DER
148.6MB
38,964
5.591.4MB
Resnet18
11.17M
42.6MB
ER-MIR
40,000
5.740.0MB
Resnet18
11.17M
42.6MB
EWC
Fl & Previous Model
85.2MB
39,406
5,654.8MB
Resnet18
11.17M
42,6MB
OCS
40,000
5.740.0MB
Resnet1
11.17M
42,6MB
X-DER
Logits
148.6MB
38,964
5,591 .MBB
Resnet1 8
11.17M
42.6MB
LiDER
Logits
148 6..6MB
38,964
5,591 .MB
Resner1
11.17M
42.6MB
MEMO
Expanded Network
32,0MB
39,777
5.708.0MB
Resnet18
11.17M
4266BB
Teacher model for collahorative leaming
CCL-DC
42.6MB
39,703
5,697.4MB
Resnet18
11.17M
42.6MB
Class-wise similarity & frequency of each sample
aL-SAR
3,9MB
39,973
5.736.1MB
Resnet18 8
11.17M
42.6MB
Table 21: Implementation details of total memory budgct=5.782.6MB in ImageNet
A.24 DETAILS ABOUT AAUC
Recent studies (Pellegrini et al. 2020 Caccia et al. 2022 Banerjee et al. 2023 Ghunaim et al., 2023) suggest that having good inference performance any intermediate time points during train- ing is important for CL. To evaluate intermediate performance during training, (Koh et al. 2022) proposed AAUC, which measures the area under the curve of average accuracy. In contrast Alast or Aung which measures performance only at the end of the task (i.e., after sufficient training), AxuC consistently measures performance over the course of training. If two methods reach the same aceu- racy at the end of a task, but one method converges faster than the other, their Alast and Aaeg would be equal, but the faster model would show higher AAUC. Thus, how fast the model adapts to the new task is reflected in AAUC-
A.25 COMPARISON OF FORGETTING
We compare the forgetting of aL-SAR with other baselines on CIFAR-IO, CIFAR-I00, and ImageNet- IK, and summarize the results in Fig 18, and Tab. 22 respectively, Specifically, in disjoint setup. we re- port Flast (Chaudhry et al. 2018). following Bang et al.) (2021): Koh et al. (022). In the Gaussian setup, however, the continuous shift in data distri- bution lacks explicit task boundaries, making tradi- tional metrics like Flast (Chaudhry et al. 2018) un- suitable. Therefore, Ratio (KLR) (Koh et al. 2023), which do not require task boundaries.
Methods
KLR 4
TFLOPs +
61.58
ER-MIR
61.05
DER+
60.21
LiDER
59.21
114,014
X-DER
59.72
MEMO
59.25
CAMA
60.02
CCL-DC
61.26
report the Knowledge Loss
aL-SAR (Ours)
56.28
947,128
Table Comparison of KLR and KGR on ImageNet- IK Gaussian scheduled setup.
As shown in Fig 18 forgetting decreases as training FLOPs increase. We believe this is because the in-
32

---

Published as a conference paper at ICLR 2025
creased computational budget allows the model to sufficiently train on previously encountered data before being exposed to new data.
ER DER+
LIDER
X-DER
CCLDC
CAMA
LAR (Ours)
80
70
20
60
40-
30
GFL Ps/samae
GFLOPs/sample
(a) CIFAR-10 Gaussian task setup
(b) CIFAR-10 10 Disjoint task setup
ER
MIR
DER+
MEMO
LIDER
X-DER
GCLDC
CAMA
aL SAR (Ouns)
100
80
30
10
GFL OPsisample
GFLOPs/sample
(a) CIFAR-106 100 Gaussian task setup
(b) CIFAR-100 Disjoint task setup
Figure 18: Comparison of forgetting on Gaussian and Disjoint CL setup in CIFAR-10 and CIFAR-100.
A.26 TOY EXPERIMENTS ON SIMILARITY-AWARE RETRIEVAL
We performe several toy experiments to validate the motivations for our proposed retrieval method. To isolate the effect of each sample, we train ResNet-20 from scratch with batch size of 1 using subset of CIFAR-10 10. First, to validate our claim "extensively used samples provide little knowledge the model", we plot (use frequency) (training loss decrease caused by training with the sample) in Fig. 19 We observe that training with a frequently used sample results in a small decrease in training loss.
Next, we validate the exponential decaying model of forgetting. To measure this, we first train the model with one sample and measure the use frequency along with the corresponding training loss. Next, when training the model with different samples, we measure the loss of the previously trained sample. At this point, due to the training of other samples, forgetting occurred for the sample we used for initial training, causing the training loss to increase again. We then use the measured use- frequency and corresponding training loss values, which are measured in the first stage, to reverse- transform the training loss into an effective use-frequency. As shown in Fig. 20 as the number of training iterations for other samples increases, we observe that the training loss for previously trained samples increases, leading scale and the plot shows a linear trend, showing that the loss increase from forgetting corresponds to exponential decay of use frequency.
decrease in the effective use- use-frequency. Note that the y-axis is in log
Finally, we validate that cosine similarity between gradients can be used to measure the effective change in frequency from using other samples. Using the same setup the previous exper- iment, in Fig. [1] we show a scatter plot between gradient similarity and the effective change in use-frequency obtained by the relation between the sample's loss and use frequency. When training
33

---

Published as a conference paper at ICLR 2025
with samples that have high gradient similarity with the reference samples, the effective fre- quency of the reference samples increases. On the contrary, when training with samples that have low gradient similarity, the effective use frequency of reference samples decreases. We observe linear relationship between gradient similarity and the change effective ase-frequency. In other words, the change in effective use frequency be predicted using gradient similarity.
2.5
2.0
loss decrease
1.5
1.0
0.0
0
10
20
use frequency
Figure 19: Relation between sample's 's frequency and training loss decrease from training with the sample, measured CIFAR-10 with ResNet-20. When a sample used more frequently in training, has less effect on reducing training loss.
40
corresponding use frequency
30
20
5
10
20
25
iterations
Figure 20: Plot showing the number of iterations from the point where the sample has not been used for training. and the decrease effective frequency corresponding to the increase loss. It measured CIFAR-10 with ResNet-20.
34

---

Published as a conference paper at ICLR 2025
effective use frequency
0.5
0.0
-0.5
-1.0
0.5
cosine similarity
The seatter plot shows the cosine similarity between gradient of a reference sample and gradients of other samples, along with the change in the effective use frequency of the reference sample when training with other samples. It is performed in CIFAR-10 10 with ResNet-20.
Figure 21:
A.27 LIMITATIONS AND FUTURE WORK
While method only requires negligible additional memory other than episodic memory, it does not actively optimize the memory efficiency CL algorithms. is interesting to explore a method to use the limited storage budget more efficiently, e.g., storing quantized versions of models and exemplars.
A.28 IMPACT STATEMENT
This work aims to update a model in computationally efficient and online manner without access to training samples used before. Thus, although there is no intent from the authors. this method may exacerbate unsolved issues in deep learning such as model bias and less aligned to ethical standards. We will take all available measures to prevent such outcomes, though that is not our intention at all.
35


Reviews:
[{"review_id": "t8YDTvdXAX", "reviewer": "Reviewer_1YwR", "summary": "This paper proposes the method *aL-SAR*, which standardizes computational and memory budgets of continual learning (CL) algorithms based on training floating point operations (FLOPs) and total memory size in Byte to address the challenge of fair comparison in online CL algorithms. Besides, this paper introduces adaptive layer freezing to reduce computation costs by selectively freezing layers and frequency-based sampling to prioritize under-learned samples. Experiments on CIFAR-10/100 and ImageNet-1K datasets show that aL-SAR consistently outperforms existing CL methods like ER (Experience Replay), DER++, MIR (Maximally Interfered Retrieval), MEMO, REMIND, EWC, OCS, LiDER, X-DER, CCL-DC, and CAMA by up to 5-10% on average across AAUC and last accuracy, especially under stringent budget constraints.", "questions": "C1:\n  - FLOPs served as a direct measure of computational demand across different models, making it possible to compare CL algorithms fairly. However, it doesn’t account for real-world runtime or memory access patterns, which vary based on hardware and software implementations. For instance, two architectures with similar FLOPs might have different inference times on the same hardware due to parallelization or memory bandwidth usage differences.\n  \nC2:\n  - The adaptive layer freezing proposed in this paper can reduce FLOPs but adds complexity. Calculating the full Fisher Information Matrix (FIM) is computationally intensive because it involves second-order derivatives, which can be prohibitive, especially for large neural networks. The paper must explain why calculating FIM does not introduce additional computation costs.\n  - Fisher Information (FI) is calculated based on the current mini-batch. However, the calculation results based on the current mini-batch may not generalize across diverse batches in continual learning. For instance, a layer might appear less informative for one mini-batch but could be essential for others due to shifting data distributions, especially under continual learning settings. Hence, the paper needs to justify this.\n\nC3:\n  - The strategy might prioritize rare samples in the dataset but may need to be more informative. This overemphasis can lead to inefficient learning, as these rare samples may not enhance generalization and introduce noise. The paper needs to justify that such a retrieval strategy would maintain the acquired knowledge during training.\n\n**Writing Issues**\n\n  1. Line 071: 'ImageNet)' -> 'ImageNet'.\n  2. Line 087: 'ER' has no citation and explanation for the abbreviation.\n  3. Fig. 1: No explanation of the y-axis.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ Standardizing computational and memory budgets of continual learning (CL) algorithms is important for evaluating algorithm efficiency and learning system design.\n\n++ The paper provides comprehensive and detailed experimental results.", "weaknesses": "-- Using FLOPs as a metric focuses on raw operations, ignoring algorithm-specific optimizations for training software and hardware systems. See C1.\n\n-- The additional computations required to decide which layers to freeze may introduce overhead that is not directly part of the core training but still contributes to the total computational cost. See C2.\n\n-- The retrieval strategy based on the use-frequency cannot ensure that the model does learn sufficiently by preferring less frequently used samples. See C3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CfOdvpxhtE", "reviewer": "Reviewer_j3Dm", "summary": "This paper proposes a new algorithm for the online CL problem. The authors first explain that a fair comparison between different methods should consider the total number of FLOPs instead of the total number of iterations, as some methods can perform expensive operations each round. Also, it is essential to include all types of memory usage rather than only considering the number of examples in the episodic memory as the actual memory cost. Next they propose an efficient algorithm to address the problems in online continual learning. They use batch fisher information to find the optimal layers that can be frozen while learning from the data. Then, they introduce a new metric, 'appearance frequency,' to measure the contributions of each training example.", "questions": "* What is the memory selection mechanism? In other words, how memory samples are selected, and how does the memory get updated?\n\n* Besides the similarity-aware retrieval, is there any other difference between the in-memory and new samples in the training?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The arguments about the FLOPS and memory costs are critical. To have a fair and meaningful comparison between different methods, we *need* to know the actual cost.\n\n* The idea of having batch-wise and dynamic layer freezing makes the training more suitable for online settings.\n\n* The authors did an excellent job of explaining the motivation and algorithm. All the different components are adequately explained and justified.", "weaknesses": "* In my opinion, the introduction lacks proper background knowledge, and it jumps to the solutions for the existing problems in online CL. I suggest adding at least one paragraph explaining the background and current state of online continual learning and the current high-level challenges that prior work has attempted to solve, so the readers are on the same page and maybe appreciate your work even more. \n\n* Have you considered reporting forgetting as an additional metric? This is commonly used in CL papers and could provide further insight into the efficacy of your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxmYAMhoxm", "reviewer": "Reviewer_VVoh", "summary": "This paper addresses the challenge of fair comparison in online continual learning (CL) by proposing a total resource budget framework that considers both computational costs (FLOPs) and memory usage (bytes). The authors introduce aL-SAR, which combines two key innovations:\n\n1. Adaptive layer freezing: A method that selectively freezes network layers based on Fisher Information to reduce computational costs while maintaining performance\n2. Similarity-aware retrieval: A computationally efficient strategy for retrieving training samples based on usage frequency and gradient similarity\n\nThe method is extensively validated on CIFAR-10/100, CLEAR-10/100, and ImageNet-1K datasets, showing superior performance compared to state-of-the-art methods under the same resource constraints.", "questions": "1. Could the adaptive layer freezing strategy be extended to handle cross-layer dependencies, potentially leading to more optimal freezing decisions?\n2. Although authors have illustrated the process of calculating *Effective Use Frequency* in Sec.3.2 and Sec.A.17, it is still vague for me that how to calculate cosine similarity of the gradients between classes? In A.17, do it that using *0.05% of the model parameters* means that randomly sample params among unfrozen layers?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel framework for comparing CL methods using total resource budgets including innovative adaptive layer freezing approach based on Fisher Information and computationally efficient memory retrieval strategy.\n\n2. Rigorous empirical validation across multiple datasets and setups containing thorough ablation studies and comparisons with state-of-the-art methods and successful application to large models (LLaVA-1.5-7B).\n\n3. Addresses practical constraints in online CL and shows significant performance improvements while reducing computational costs.", "weaknesses": "1. Using the layer freezing technque for recuding the model params would always block the shallow layers (i.e., layer 1-n) and their representation would be outdated. This would be problematic for frequent distribution shifts.\n   \n2. The paper's adaptive layer freezing strategy treats layers somewhat independently when making freezing decision. However, neural networks typically have complex inter-layer relationships that this approach might not fully capture.\n   \n3. The method doesn't address catastrophic forgetting directly, instead focusing on resource efficiency. It's unclear how the method balances the trade-off between resource efficiency and forgetting prevention.\n   \n4. The computational overhead of maintaining class-wise gradient similarities might become prohibitive for problems with a large number of classes or frequent distribution shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccKQzHJUND", "reviewer": "Reviewer_3EaD", "summary": "This paper addresses the challenge of resource constraints in online continual learning (CL) by introducing an efficient method named aL-SAR, which combines adaptive layer freezing and similarity-aware sample retrieval. The adaptive layer freezing component selectively freezes layers to minimize computation based on Fisher Information, while the similarity-aware retrieval component focuses on sampling underused and informative samples from episodic memory to enhance training efficiency. Experiments on CIFAR-10/100, CLEAR-10/100, and ImageNet-1K datasets demonstrate that aL-SAR outperforms several state-of-the-art methods under the same computational and memory budgets.", "questions": "Please also refer the comments in the strengths and weaknesses sections:\n\n1. Can the authors compare the al-SAR with other methods under the epoch-based setup to show the effectiveness of the method with larger memory constraint? \n\n2. Can you consider applying aL-SAR on more diverse datasets (e.g., text, multimodal data) to demonstrate the approach’s generalizability?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Pros:\n\n1. The paper introduces a unique approach to online CL by integrating adaptive layer freezing and frequency-based sampling, effectively reducing both computational and memory costs without significant performance trade-offs.\n\n2. The empirical results across multiple datasets comparing to multiple existing SOTA benchmarks show the competitiveness of al-SAR, especially under constrained resource settings.", "weaknesses": "1. Though designed to be efficient by only computing class based gradient, the similarity-aware retrieval approach still requires gradient similarity calculations, which could introduce overhead in large-scale applications. \n\n2. The experimental setups impose strict memory and computational constraints, limiting the contribution primarily to incremental improvements in the efficiency of existing online continual learning approaches.\n\n3. The authors should compare the method with model based like iCaRL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)