Compact Memory for Continual Logistic Regression
Authors: Yohan Jung
Hyungi Lee
Wenlong Chen
Thomas Möllenhoff
Yingzhen Li
Juho Lee
Mohammad Emtiyaz Khan
Venue: NeurIPS 2025 poster ()
Keywords: Compact memory
Continual learning
Memory efficiency
K-prior
Primary area: general_machine_learning
TL;DR: We present a new continual leanring method that builds and reuses compact memory for logistic regression

Abstract: Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.

Compact Memory for Continual Logistic Regression
Yohan Jung' Hyungi Lee2*
Wenlong Chen
Thomas Möllenhoff'
Yingzhen Li
Juho Lee
Mohammad Emtiyaz Khan
RIKEN Center for Al Project "Koomin University
3 $Imperial College London
*KAIST
Work performed in part during internship at the RIKEN Center for Al Project.
Abstract
Despite recent progress, continual learning still does not match the performance batch training. To avoid catastrophic forgetting, we need to build compact memory essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and S Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose probabilistic PCA method to estimate them. Our approach drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy 74%, closing the gap the batch accuracy of 77.6% on this task. Our work opens new direction for building compact memory that can also be useful in the future for continual deep learning.
Introduction
Continual learning aims for a continual lifelong acquisition of knowledge which is challenging because requires a delicate balance between the old and new knowledge [Mermillod et al.. 2013]. New knowledge can interfere with old knowledge [Sutton, 1986] and lead to catastrophic forget- ting (Kirkpatrick al., 2017]. Thus, it is necessary to compactly memorize all the past knowledge required in the future to quickly learn new things. This is difficult problem and no satisfactory solution has been found yet, even for simple cases such as logistic regression and shallow neural networks. The problem is important because solving it can also drastically reduce the cost and environmental impact of batch training which requires access to all of the data the time.
The most straightforward method to build memory is to simply store old models and data examples, but such approaches do not perform as well as batch training. For instance, weight-regularizztte uses old parameters to regularize future training [Kirkpatrick et al., 2017, Li and Hoiem, 2017, Lee et al., 2017, Zenke et al., 2017, Ebrahimi al., 2020, Ritter et al., 2018, Nguyen et al., 2018). While this is memory efficient, it often performs worse than replay method that simply store old raw data for reuse in the future [Rolnick et al., 2019, Lopez-Paz and Ranzato, 2017, Chaudhry al., 2019, Titsias et al., 2020, Buzzega et al., 2020, Pan et al., 2020, Scannell et al., 2024]. Many replay- based methods have been devised to effectively select past data points based on gradient [Aljundi et al., 2019a, Yoon et al., 2022], interference [Aljundi et al., 2019b), Bilevel optimization [Borsos al., 2020]. The memory cost of replay can reduced by combining with weight regularization. but far the performance still lags behind the batch training [Daxberger et al., 2023]. The goal of this work is to propose new methods to build compact memory for the
cotmmatiion-hhooeett metric [Sun et al., 2022], and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Standard continull-eeaning methods, such as those using weight regularization, use past parameters O, to update 0.+1 for the new task Di+1 (shown on the left). We propose a new method (shown on the right) that also builds compact memory and reuses to continually learn. The memory Continual Learning
Continual Learning with Compact Memory
θ 0[+1
1
0
Ur Wt
Uz+1 We+1
Dr
Do+1
Dt
De+1
consists of set U, of memory vectors and a weight vector w.- These are used to update 0i+1 when new D arrives. Afterward, the memory is updated to get the new (U++1, W1+1).
simplest non-trivial case of logistic regression. With this, we hope to gain new insights to be able to do the same in the future for continual deep learning.
We build upon the work of Khan and Swaroop [2021] who show the existence of compact memory for logistic regression but do not give practical method to estimate them. They propose a prior called the
prior (K-prior) which uses a 'memory' set to approximate the full-batch gradient. More precisely. given parameter 0₂ after training until task t, subset Mr of the past data Dia (Di, D2,..., D.} is used to construct the K-prior K,(0; such that the following holds:
Vi,(0,D,) (o.,..
where (0; denotes the loss over examples in Dj. Khan and Swaroop [2021, App. A] show that, for logistic regression, it is also possible to construct optimal memory by using the singular vectors of the feature matrix. The optimality refers to the fact that the memory can, theory, perfectly recover full-batch gradient, that is, the approximation in Eq. becomes memory size depends on the rank of the feature matrix, not on the size of whole Di.s. yielding a compact optimal memory. The main issue with the method is that it requires access to the whole D1:t to construct the memory which is not possible when learning continually, Our goal here is to propose practical alternative to estimate such memory for continual logistic regression.
equality. The optimal
In this paper, we reformulate the search for the optimal memory as Hessian-matchinn and propose a practical method to estimate them (Fig. 1). We are motivated by the fact that, in linear regression, estimation of optimal memory can be reformulated as Hessian matching. We generalize this to logistic regression by posing as maximum likelihood estimation problem which uses the classical Probabilistic Principal Component Analysis [PPCA; Tipping and Bishop. 1999a]. The method does not yield perfect gradient reconstruction anymore, but can still drastically improve performance compared to replay of raw old data. For instance, on the accuracy compared to the 30% accuracy obtained by replay with memory-size equivalent to 0.3% of the data size. small memory size of 2% of the data size is sufficient to obtain an accuracy of this task. Our work opens a new direction
dataset, we obtain 60%
74%, closing the gap to the batch accuracy of 77.6% for building compact memory that can also be useful in the future for continual deep learning. Our code is available at https:/ //t::///ttttttt www.:eammccce//e//oo caamm-appox---a aappox-baaee////eeepppppaaaaaeee htapeeesccpppct__mecceeeee cortst_mmmmo_o_c_ee
2 Compact Memory for Continual Learning
We consider continual learning for supervised problems where the data of the j'th task consists of N; input-output pairs of form (xy: yy). The goal is to learn the parameter vector 0 whose predictions closely match the true labels y/- The empirical loss over task j is defined
((0,D))
Σ L Lv,, (()xx))))) EED
where C(y, 9) is chosen to the cross-entropy loss over predictions ý obtained by passing the logits fo(x) through the softmax function. As discussed carlier. learning continually requires delicate

---

balance between old and new knowledge. Tasks may arrive one at a time and there may be little flexibility revisit them again. This is unlike the standard 'batch' training where the goal is to obtain the parameter o obtained by jointly training over the sum Σ l over all T tasks. Such batch training typically requires access to all the data all the time. Continual learning eliminates this requirements and aim to recover 0 by incrementally training of tasks one at a time.
Continual learning however is extremely challenging, even for simple cases such as logistic regression and shallow neural networks. If new task interferes with the old knowledge, there is a chance the model may catastrophically forget the previously acquired knowledge. Because the tasks are not revisited, the model may never be able to relearn it. It is therefore important to compactly memorize all essential knowledge that may be needed to learn new things in the future. This is challenging because the future is uncertain and new tasks may be very different from old ones. No satisfactory solution has been found yet to build compact memory that facilitates accurate continual learning.
Currently, there are two most popular methods to build memory and both do not perform as well as batch training. The first method is to store the old model parameters O, after training until task and reuse it to train over the next task. For example, we can use quadratic weight-regularizer
1++1 - arg min f (0; D(+1) + 6 - 0.11?.
where &, > is the regularization parameter. This memory efficient because the memory cost does not grow with tasks and requires storing only one model parameters. It is also possible to use multiple checkpoints as long as the memory does not grow too fast.
The second method is to store old raw data, which is often not the most memory efficient approach. This is because the memory size could grow larger as new tasks are seen. However, in practice, the method often performs much better than weight regularization. The method is also easy to implement within standard stochastic training pipeline. For instance, given a set &, of replay examples, we can simply us the following update,
O141 - arg min (0: + C ((f(xx)))
jeE.
Another alternative use a function-spac space regularizer, defined as C(U(So, (x (x,)), (xx)))))) where instead of predicting the true labels 3/5 we predict prediction @fe, (x,)) of O.. This also boosts performance in practice [Titsias et al., 2020, Buzzega et al., 2020, Pan et al., 2020, Scannell et al., 2024]. The memory efficiency of all of these methods can be reduced by combining them with weight regularization, which brings us to the Knowledge- aadedddgtaattttioonnnnn prior of Khan and Swaroop [2021].
As discussed carlier, K- K-prior uses both 0, and memory set M, to approximate the full-batch gradi- ent over the past tasks. The memory set consists of K, vectors, that is, M, (uije> U2]r.... uk, ir) where Uk is the k'th memory vector. K K-prior uses combination of the weight- space and function- space regularizer to approximate the full-batch gradient. For instance, consider a regularized linear- regression problem with loss L(y;, ((0) 0)) _ [0; $ 0)2 2 for each (Φ).U;) pair, with regular- izer | For this model, we can use the following combination:
Σ [ ( u
K. (. = 1 5. 0,12
(5)
The first term ensures that new are close to old O, while the second term ensures the same for their predictions. We then use the following update for the next task,
0 0 < arg min (+ De+1) + (O,
e
Khan and Swaroop [2021] prove that, for generalized linear models, such an update can perfectly recover the exact gradient when M, Diat and 0, 0e (the batch solution over D1.)). Choosing a subset of Di. as the memory set yields an approximation of the gradient whose accuracy can be controlled by carefully choosing the memory set, as long as 0 is close enough to 0.
Choosing M, Di:t however not the best choice. Khan and Swaroop [2021, App. A] show that is also possible to construct a much more compact optimal memory set which can recover prefect gradients even when MM < (Dizl This can be done by setting Ugje to be the left singular vectors of the feature matrix. To be precise, denoting by oi: the feature matrix of containing features of inputs in D1s as columns, we compute the following singllarvalue-decompssiiii
(SVD).
+ 1at U.St (V?)T, T. where U," = (uije: u 22
u K
and V = (vī 22 V k (7)
3

---

The left and right singular vectors are denoted by uej and kit respectively. The rank is denoted by K and S is a diagonal matrix containing all K² non-zero singular values denoted by s Khan and Swaroop [2021, App. A] define the optimal memory to be M, U Then, they construct the following optimal K-prior, denoted by K. (0; ) for binary logistic regression,
RT
C (0: U = 2 0,12+ +
Σ W C ()((Et)) (.), (((u)))) 8))
(8)
and show that there exist scalars wxje for which it perfectly recovers the full-batch gradient at O. Here, ((f) denotes the prediction function which is the sigmoid function for binary logistic regression, that is. v(. of) = 1/(1 + exp(-f)). The result shows that the optimal memory size is equal to K and perfect gradient recovery is possible by simply comparing K² predictions at old and new models. This can be extremely cheap compared to the full batch gradient for long data streams whose intrinsic dimensionality much smaller.
The main issue with the method is that computing wije for each is infeasible in practice because requires access to all of the past data. Specifically, the optimal value is given by
2/l0 (V R/1
d
w A/je (
(u 0τ)
which require computation of the vector dg whose length equal to the data size [Diell with the j'th entry is equal to (Φ [ 0) o Φb 0,). Since we do not have access to old features Φy. is impossible to find the optimal w for all 0 values. In what follows, we propose practical method to estimate such compact memory for continual logistic regression, for both binary and multi-class cases. We do not aim to exactly obtain the optimal memory but still hope to keep the memory size small enough much larger than the rank of the feature matrix. Our method uses Hessian matching
so that it which is related to other theoretical works [Li et al., 2023, Ding et al., 2024, Li et al., 2025].
3 Compact Memory for K-prior
We now present practical alternative estimate optimal memory for continual logistic regression. Throughout, we assume quadratic regularizer 0 needed for training at task I; the method can easily handle other regularizers as well. Motivated by Eq. 8, we propose define the K K-prior by using a set of unit-norm memory vectors Ugee and their weights Wk|ps as shown below:
K.
K2(O; U,, w.) = 0 - 0.11² + Σ
Whie L (( ()(u)) O))
(10)
The set of memory and weight vectors are respectively defined as follows,
U² = (iije U2/4 (x, and
W, (wije Wait wk.())
(11)
and we denote by W, the diagomal matrix whose diagonal is the vector w,. Clearly, fixed we will not yield the optimal K-prior but we still hope that it will make the memory more compact. To estimate the memory and weights, we start with an initial value of (U,, wo) at t = 0 and use the following iterative procedure (illustrated in Fig. 1).
1. Estimate θe+1 by training over (0; D++1) + (0; U., w.).
2. Estimate (U/+1, W(+1) by using Hessian matching (with PPCA).
The first step is simply the standard K- -prior training as in Eq. 6. Next, we motivate the second step on a linear regression problem and then present extension for logistic regression.
3.1 Compact Memory for Linear Regression
We consider the following linear regression problem and show that estimating the second step can be realized via Hessian matching,
2,(0: = 1 2
Σ
Σ L (1, (V,(())))) DED;
where L(v. 0) = = ( 0) 2 (12)
0-0

---

Here, we denote the quadratic regularizer 5 by Do). For this problem, the K-prior can perfectly recover the joint loss if memory chosen optimally, as stated in the following theorem.
Theorem For the problem in Eq. 12, the K-prior shown in Eq. 10 is equivalent to the original loss and takes a quadratic form if we set O2 -- 0r, U, - U and W. W+= - St, that is, we have
Σ (,0,),)) KK, wi) -00--00 - 0,) H(O)) -0).
(13)
where H(O,) 1 T. 6,1 is the Hessian of the objective in Eq. 12.
proof is included in App. A.I.
This result gives clue on how to estimate the next (U++1. W141). Essentially, if 0,-1 <- 0 as well, then we should choose (U++1, We+1) such that the following holds for all A,
Ke+ UL+1, W = (0: Dz+1) + (o; U,, w.).
(14)
Such a choice will ensure that the batch training loss for tasks to + is perfectly reconstructed with the new K-prior K(+1- For linear regression, this condition is satisfied by using the second-order optimality condition, obtained by taking the second derivative of both sides at 0,+1. This is because the Hessian is independent of 0 and Hessian matching yields
U W U + 4411 क o T + U.W.U,,
(15)
where Ex-1 (61+1 62). We note that this is not possible by using the first-order optimality
condition because gradients are of the same size as 0, which is not enough to estimate a (larger) U, In contrast, the condition above is sufficient and is just another way formulate procedure. Essentially, we can concatenate (i++1. U, (.! and apply SVD to it to obtain U++1 and w The last term 6 represents the error made in this process which can be driven to zero by choosing K,-1 appropriately. SVD can however be numerically unstable and we will use an alternative based PPCA, which is easier to implement and also have intuitive interpretation.
online SVD
3.2 Hessian Matching via Probabilistic PCA (PPCA)
Hessian matching can be formulated as Probabilistic PCA [Tipping and Bishop, 1999a] and conve- niently solved using the Expectation Maximization (EM) algorithm. Essentially, aim to find U whose columns can accurately predict the columns of P.41 and U, More formally, we define the following two matrices,
T ( U. U.!!
(16)
where N K₁ + [De+1]. Then, we use the following PPCA model to predict the columns of T,
t
1/2 24 + Cs. where z, -N and ~ ~N(00
(17)
We now show that maximum- likellhood estimation on this model can find (U++1, We+1) that solve Eq. 15. As shown by Tipping and Bishop [1999a, Eq. 4], the log obtained by noting that t; follows Gaussian distribution too, which gives (details in App. A.2)
of this model is
log (TT W++1) = - log |C| + Tr (C-ITTT) /N)TTT/N]]]] + cnst.
(18)
where C U4+1 W W + II Then, taking the derivative and setting it to O, we can show that the optimality condition is to set TT /N which equivalent to Eq. 15; derivation included in App. A.2. Similarly to online SVD, maximum likelihood over PPCA also performs Hessian matching. PPCA also enables us to estimate 1,+1, , but for simplicity we fix to a constant value (denoted by from now onward).
The main advantage of this formulation is that leads to an casy-to-implement EM algorithm. Essentially, in the E-step, we find the posterior over z,, then in M-step, we update U. The two steps can be conveniently written just one line [Tipping and Bishop, 1999a, Eq. 29]. This is obtained by denoting U - U 1//2 and updating it as follows:
1+1
U - SÜ M ÜTSÜÜ-?
where M M-Ü- and = TT /N. (19)
5

---

Algorithm 1: Hessian Matching by PPCA
Algorithm 2: Extension for Logistic Regression
Require: 4 U,, Wt and
Require: Pt+1, Ut Wt. 6, and 0.+1
W Di) and U U, U/
I: W - Diag( w.) and U < U
2: A 5'( [+] 0r+1) and X2 00
2:
3: T - (ir-DDage DiaiDiaa))) U, U 1/2
3: T (i141, 3
S TT
4: S TT
while not converged do
5:
while not converged do
6: À - j'(U 0,41) and Ü - U(WA)
6:
7: M - Ü-ÜÜÜ+ +el
7: M U el
U SU(e++ M -ÜSÜÜ)
8:
Ü
Ü - UA UX-1/2 1/2
9:
9:
end while
10:
diag( Ú) and W Diag(w)
11: w - diag( Ü Ú) and W - Diag(w)
11: U UW-1/2
12: U U U-ÜW ÜW-1/2 1/2
12: end while
13: return U++1 - U and W++1 -
13: return U,+1 and W
The left side shows a Hessian mathhing algorithm for linear regression obtained using EM for PPCA. On the right, extension for logistic regression is shown where differences the regression case highlighted in red. The main differences are line 2, 3, 6, and 9, and they all mostly due the inclusion of terms that contain (J). For logistic regression, j"(() af) of))- When operated on a vector f, 0' (f) returns a vector of (ft). Diag( w) denotes a diagonal matrix with vector the diagonal, while diag(W) returns back the diagonal vector of W.
Figure 2:
Derivation included in App. A.2 and an algorithm to implement this procedure is outlined in Alg. 1, where in lines 11-12, we normalize columns of U to obtain (U,+1, We+1). The algorithm assumes that K2+1 Kg, but if memory size increased, then we need just one additional step to initialize the additional vectors (we can do this in line 1). In practice, we use subset of 4; to initialize the new memory vectors. The EM procedure implements Hessian matching by essentially finding new memory vectors that can predict old memories as well as the new features.
3.3 An Extension for Binary Logistic Regression
We now show an extension for binary logistic regression. The procedure also works for other generalized linear models but we omit it because the derivation is straightforward. We will however briefly discuss an extension to multi-class afterward.
For binary logistic regression, aiming for a recursion similar Eq. is challenging. As discussed earlier, this is because computing the optimal Wkia is infeasible in practice. To simplify the estimation, we relax this requirement and aim for Hessian matching 0.+1. That is, we reformulate the search for (U++1, W++1) the following Hessian-matching problem at Ox+1,
V (0++1 U--1, W++1) = vi D:+1) + ('K,(0 W.).
(20)
This condition leads to an equation shown below which takes form similar to Eq. 15 obtained for the linear regression case. A derivation is given in App. A.3, where we show that the only difference is in the diagonal matrices which are defined differently:
U W +1 U. + 4411 4 B ( + U.W.U.
(21)
where W t+1, W., and B,41 are the diagonal matrices defined as follows:
W 1+ W j (U.1)0+1)) W, W. (U. 0.t1), B Diag(i'(())..) 0++1)) (22)
Here, the function jf( denotes the first derivative of (({) with respect to f. For binary logistic regression, j(ʃ) = (f),, therefore y (f) a() When applied to vector f, (f) returns a vector j(J,), which is the case for all the quantities used above. last expression Diag(f) forms diagonal matrix whose diagonal is the vector f.

---

The new Hessian matching equation can also be solved using an algorithm similar to Alg. 1, shown in Alg. 2. We will now briefly describe how derive the algorithm. First, instead of using Eq. 16, we redefine T as shown below where we include some vectors computed using the function j(-).
T - (N(N Diag( 1, U.W 1/2 X/
(23)
where we use A1 - 5' " 10++1 and X2 - O'UU, 0++1). With this change, TT /N is equal to the matrix the right hand side Eq. 20. This is done in line 2 and of Alg. 2. The next change is in the update of U Essentially, we redefine
(24) where x - ((UU(+10+++ With this change, the left hand side in Eq. 20 becomes ÜÜ and then we can apply the update shown in Eq. 19. This is done in line 6-8. Finally, because we need to recompute À every time update U, regularly extract (U++1, We+1) from U1+1, which is done in line 9-12. One minor change is to initialize the procedure in line 1.
- W
1/2 (+1 1/2
The computational cost of the EM algorithm is dominated by the inversion in line 8 of a square matrix of size K++1- Therefore, the overall complexity For small K1+1 and I, this cost is manageable and does not add a huge overhead to overall training.
O( (1K341) where 7 the number of iterations.
3.4 Extensions for Multi Multi-Class Logistic Regression and Other Generalized Linear Models
Alg. 2 can be easily extended to estimate compact memory for other generalized linear models. The main change required is to appropriately define the function (f). We demonstrate this for multi-class logistic regression with number of classes C. For this case, define parameter vector g(s) for each class c, giving rise to the matrix @ = (0(1)
((0)T. T. The linear predictor for a feature & (of length P) is defined as f = OΦ and is of length C. Given such a linear predictor, the prediction function and its derivatives are defined through the softmax function as shown below,
cxp(f(o)) Σ exp(f(e)))
(()) = p and (f()) Diag(p) -pp", where e'th entry of p is (())
(25)
which are vector and matrix of size P, respectively. The gradient and Hessian are given by
oc(y, (()) = [b(pl() - (0)
22 @L((y.(f)) = [t'(F) (())) TØ (0(0)
(0(0)
where ((((())(e.e))) denotes the entry (c, of the matrix. As result, the size of the Hessian of the loss C(y, (ff) with respect to (vectorized) @ is a PC x PC matrix where P is the length of each g(e) Since a P P matrix is sufficient estimate the memory vectors of size P Ke+1, we choose To that is, we use the following,
to perform Hessian matching over the sum > d'
> ViKCH1 (1 U.+1 wx+1) U
> 10 ".
(26)
This condition ignores the cross-derivatives between all pairs e + d', but the Hessian matching takes a similar form to Eq. 21. Essentially, we can push the sum with respect to e inside because the only quantity that depends on is (()))). Due to this change, we need to redefine the diagonal entries of the matrices W W1+ W., and B by using the trace of g/ (f), as shown below (details in App. A.4):
((( = w Tr (O. ((44
w((k)) = waye Tr [f((0:1
(27)
B B Tr (0(+)++))
where & runs from 1 to K2 (or K(+1) and j runs from to (Dital. The trace function is denoted by (r)) = Σ., pfo p(e)) which takes a similar form to the binary case. Essentially, large value of this quantity indicates that the corresponding feature is important for at least one class. To improve numerical stability, we clip the vector p away from or 0 by small amount (we use 10-1). Overall, this example demonstrates how to extend our approach to other generalized linear models.
7

---

adactation for task-1
adaptation for task-2
adaptation for task-3
adaptation for task-1
W
Figure 3: Binary logistic regression on a toy 'four-moon' datasets with four tasks (red - blue black- green). The gray line shows the batch training, while our compact-memory method's boundary are shown with colors. We see that estimated boundary closely tracks the batch solution and ultimately recovers it at the end of training. We also show with circles the points in the input space that are closed to the memories in terms of the loss gradient.
0.95
0
0.90
accuracy
2
3
T2
0.85
batch
45
our
T3
k-prior
replay
6
5
10
20
memory (% past data)
(a) USPS even
(b) Representation of learned memories
Figure 4: Panel (a) shows results for binary classification USPS-odd-vs-even dataset. For each E 256 the 1-order polynomial feature-map is used, that is, ((x) [1. x] E R 267 is used. Our method clearly beats both replay and K-prior (no learned memories) and obtains performance close to batch. Panel (b) visualizes the learned memories when using memory size of 1%. Each row shows the learned memory images) sorted according to their weights Wik (left to right descending order). We clearly see that new features emerge as new tasks arrive.
Experimental Results
We show the following results: (i) multi-output linear regression for sanity check, (ūi) two binary logistic regression on toy data and USPS dataset, and (iii) multi-class logistic regression on four datasets based on CIFAR-10, CIFAR- 100, Tiny TintlmageNet-200, and ImageNet-100. The multi-output regression result is included in App. B.I where we compare to the vanilla SVD and show that our method is numerically more robust. The rest of the results are described in the next section. More details of the experiments are also included in App. B.
4.1 Binary Logistic Regression
We show that for continual logistic regression our method improves over the replay and vanilla K- K-prior methods. We start with the toy example the 'four-moon dataset and then show results for the USPS dataset. Both results follow the experimental setting used by Khan and Swaroop [2021].
Four-moon dataset. We split the four-moon dataset into four binary tasks and perform continual logistic regression over them. The training set of each task consists of 500 data points and we test over all inputs obtained on 2-D grid (-3.2,3.2) x (-1.2 1.2) in the input space (a total of 158,632). We use a polynomial feature map of degree 5. The memory size set to the half feature dimensions per task and increased in this fashion with every new task. Fig. 3 shows the changes in the decision boundary as the model trains on tasks, where we see that the classifier closely follows the decision boundary of the batch training (shown in gray), ultimately recovering them after 4 tasks.

---

0.8
0.92
accuracy
accuracy
0.7
0.90
0.88
batch
0.6
batch
our
k- S-prior
0.86
k-prior replay
replay
our
0.84
0.4
2
5
10
20
2
5
10
20
memory size (% of past data)
memory size (% of past data)
(a) Split-CIFAR CIFAR-I0
(b) Split-CIFAR-100
0.8
0.8
Almost batch
0.7
accuracy at 2%
0.7
0.6
accuracy
accuracy
+30%
0.5
improvement
0.6
batch
0.4
batch
0.5
k-prior
0.3
k-prior
replay
replay
0.2
our
our
0.4
1
2
10
20
0.5 memory size (% of past data)
0.3
memory size (% of past data)
(c) Split- Split-TlmageNet-200
(d) Spli-ImageNet-1000
features extracted using a pre-trained Vit-B/32 and the rest use features of a pre-trained Vit-L/14. Similarly to binary classification, method get much better accuracy compared to replay and K-prior for the same memory size. For instance, on Split-ImageNet- 1000, we get 30% improvement over replay when using memory size equivalent 0.3% of the data size. With 2% memory, obtain 74% accuracy which is close to the batch accuracy of 77.6% on this dataset.
Figure 5: Results for multi-class logistic regression. For panel (a),
To understand the nature of the newly learned memories, also plot the data examples that closest to the learned memories (shown with circles in Fig. 3). To be precise, we find examples x, in the input space whose gradient is closest to memory vector, that is, for given ugjt. look for x with the smallest V learned memories cover areas in the input space close to the decision boundary. This confirms that the memories are reasonable representation of the decision boundary.
VoL(y, ö ((fo(x)))l at 0 O,, We see that
datasets. The USPS dataset is 16x 16 image dataset of digits from 0 to 9. We relabel each digit based on whether is even or odd and consider a continual logistic regression with 5 task sequences given (0, 1) (2, (4, (8. 9). The training set of each task consists of 1000 data points, and the test set of each task consists of 300 data points. We use degree-1 polynomial feature-map yielding 256-dimensional features.
USPS odd
We consider the following baselines: K-prior [Khan and Swaroop, 2021] and experience replay [Rol- nick et al., 2019] that store partial datasets of past tasks as memory. The replay set of K-prior and experience replay are initialized by random data subsets as done Daxberger et al. [2023]. We also compare the batch training.
Fig. 4a shows the averaged accuracies on the test set over five tasks when the varying memory sizes are considered. For each experiment use five different random seeds. We plot results for memory sizes equivalent to 2, and 5% of dataset for memory. The result shows that our method outperforms other basclines significantly when using small memory sizes. Fig. 4b shows the visualization of the

---

ten trained memories when memory size is set to 1% of the data. The memory vectors are sorted according their weights Waje (left right in descending order) and visualized in an image form. With new tasks arriving, memories with new features are learned.
4.2 Multi-Class Logistic Regression using Pre-Trained Neural Network Features
We now show results following those done by Carta et al. [2023]. We use the feature map Φ(x) of Vision Transformer (Vit) [Dosovitskiy et al., 2021] which is pre-trained using CLIP (available at https::///// https://github. com/ htt///penai///i /CIPP. These are known to yield effective features for classify- ing various image datasets [Radford et al., 2021]. We construct the features { (o(x); (zz, y) E Dr) for t-th task D, while freezing the parameters of the feature extractor and then them for multi-class classification with logistic regression. We use the same baselines as the previous experiment.
We compare the performance on the following benchmarks:
1. Split-CIFAR-10 is sequence of 5 tasks constructed by dividing 50,000 examples of CIFAR-10 into non-overlapping subsets. Each task contains 2consecutive classes.
2. Split-CIFAR-I00 is sequence of 20 tasks constructed by dividing 50,000 examples of CIFAR- 100 into non-overlapping subsets. Each task contains 5 consecutive classes.
3. Spiit-TnnylmageNee-200 is sequence of 20 tasks constructed by dividing 100,000 examples of TinyImageNeet-200 into non-overlapping subsets. Each task contains 10 consecutive classes.
4. Split-ImageNet-1000 1000 is sequence of 10 tasks constructed by dividing 1,281,167 examples of ImageNet- 1000 into non-overlapping subsets. Each task contains 100 consecutive classes.
Results. Fig. 5 shows the test accuracy on each benchmark over 5 random seeds. In each figure, the r-axis shows the memory size and the y-axis shows the average accuracy over all tasks. Notably, when only 1% of the dataset is used for memory, our method significantly outperforms both replay and K-prior that use randomly selected memory. For instance, on AplittImageNet-IOO0 1000, we get 30% improvement by just using 0.3% memory size, and obtain close to batch performance by just using 2% of the data (74% vs 77.6% obtained by batch). These results clearly show the memory efficiency of our method.
Fig. 6 illustrates the computational overhead of our method com- pared to experience replay. Under the same setting as the Split- ImageNet- 1000 experiment, we consider 5 and 10 EM iterations per task; the result of 10 iterations are reported in Fig. 5. The ratio of running times, ours to replay, is reported to directly compare computational overhead. The result shows that the computational overhead of our method increases only marginally when using a smaller amount of memory. We believe that this level of com- putational overhead is acceptable for learning compact memory, which is the primary target of our research.
time ratio [our / replay)
4-
(T=10)
our (T=5)
0.1
0.3
0.5
1
2
memory size (% of past data)
Figure 6: Running time.
Discussion
As large pre-trained neural networks show remarkable performance, adaptation of these models new environments is becoming increasingly important. However, even in the case of continual learning with shallow neural network, the performance tends to degrade due to catastrophic forgetting issues. To remedy this issue, models require storing substantial memory as the learning progresses, which makes continual learning less practical.
In this work, we present Hessian matching method to obtain compact memory for continual logistic regression. We demonstrate that, this compact memory, seeking for accurate gradient reconstructions of past tasks' losses, significantly boosts memory efficiency. We further show that the memory efficiency holds even when the compact memory is used jointly with the pre-trained neural network as feature extractor. Although our algorithm is currently limited to the use of only the last-layer of a neural network with a frozen feature extractor, to ensure the principle of gradient reconstruction, we expect that this simple solution opens new path to finding the compact memory for large-scale deep neural networks such as a foundation model, and that this compact memory enables these models to continually adapt to new tasks in memory-efficien way.
10

---

Acknowledgments and Disclosure of Funding
Y. J., T. M., and M. E. K. are supported by the Bayes duality project, JST CREST Grant Number JPMJCR2112. H. L. is supported by the Institute of Information Communications Technology Plan- ning& & Evaluation (IITP) grant funded by the Korea government (MSIT) Star Fellowship (Kookmin University)). H. also supported by the Institute of Information & Com- munications Technology Planning & Evaluation(ITTP) grant funded by the Korea ( (No.RS-2025-0226377 Numan-Centtr Embodied AI Agents with Autonomous Decision- Making)).
AI
References
Mohammad Emtiyaz Khan and Siddharth Swaroop. Knowledgg--dappttiion priors. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plastichi dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
Richard S Sutton.
Two problems with backpropagation and other steepest-descent learning procedures for networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, 1986.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Demis Hassabis,
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 114(3))3521-3526 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(12):2935-299) -9447, 2017.
Sang-Woo Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung-Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), 2017.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guin continual learning in Bayesian neural networks. In International Conference on Learning Repre- sentations (ICLR), 2020.
Hippolyt Ritter, Alcksandar Botev, and David Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Cuong V Nguyen, Yingzhen Li, Thang D) Bui, and Richard E T'Trnrr. Variational continual learning. In International Conference on Learning Representations (ICLR), 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. 2019.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference Learning Representations (ICLR), 2019.
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with Gaussian processes. In International Conference Learning Representations (ICLR), 2020.

---

Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Tumeer, and Moham- mad Emtiyaz Khan. Continual deep learning by functional regularisation of memorable past. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Aidan Scannell, Riccardo Mereu, Paul Chang, Ella Tamir, Joni Pajarinen, and Amo Solin. Function- space parameterization of neural networks for sequential learning. In International Conference on Learning Representations (ICLR), 2024.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. In International Conference on Learning Representations (ICLR), 2022.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. information-theorette online memory selection for continual learning. In International Conference on Learning Repre- sentations (ICLR), 2022.
Zalán Borsos, Mojmir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Erik Daxberger, Siddharth Swaroop, Kazuki Osawa, Rio Yokota, Richard E Turner. José Miguel Hernández-Lobaan and Mohammad Emtiyaz Khan. Improving continual learning by accurate gradient reconstructions of the past. Transact. Mach. Learn. Res. (TMLRR), 2023.
Michael Tipping and Christopher M Bishop. Probabilistic principal component analysis. J. R. Stat. Soc. Ser. B Methodol., (183:611-622,, 1999a.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Fixed design analysis of regularization-base continual learning. In Conference on Lifelong Learning Agents (CoLLAs), 2023.
Meng Ding, Kaiyi Ji, Di Wang, and Jinhui Xu. Understanding forgetting in continual learning with
linear regression. In International Conference on Machine Learning (ICML), 2024.
Haoran Li, Jingfeng Wu, and Vladimir Braverman. Memory-statistics tradeoff in continual learning with structural regularization. arXiv preprint 1iXXi:2254400033, 2025.
Antonio Carta, Lorenzo Pellegrini, Andrea Cossu, Hamed Hemati, and Vincenzo Lomonaco. Avalanche: A PyTorch library for deep continual learning. J. Mach. Learn. Res. (JMLR), 24 (363):1-6, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Michael E Tipping and Christopher M Bishop. Mixtures of probabilistic principal component analyzers. Neural Comput., 11(2):443442, 1999b.
12

---

A Method Details
A.1 Proof of Theorem
We first write the quadratic form by noting that 1 4 1:t
U.W.U..
K,
O0; U. wr)
$6:110 0,11² +
2 (uz 1)
(0 - 0.) - - 0,) 1 + U, U.W.U
(28)
1 - 0,) 6, + 1 11 (0 - -0))
( 01) (0) O2)
Next, we note that 0, H()))
2 P therefore, we can write
K, U., wx)
[ 0.) (H)O)) 0.)
(Q) 0 (H + enst.
14.1012 + 1000 11 2 1.1
0 + PiVi:e cnst.
(29)
10.00 + Σ Σ
[
x 0) + enst.
J=1 DED
Σ D,)
This completes the proof.
A.2 Details of PPCA and EM algorithm
The marginal likelihood in Eq. 18 obtained by noting that, if we marginalize out z, and e, then t, ~ (O, C). Since all t; are identically distributed, the likelihood is simply a sum:
log (T W411 log (t, C) = Σ 1 log CI 2
1t ct,
| + cnst.
(30)
Nlog CI + Tr (C-ITT)]] cnst.
The stationarity condition can be derived by taking derivative with respect to Ú U W!/
++1=
(C-ITT - NC-
TTTC-U- NU,
C TTTTT (31)
where the last expression holds when Ü has full rank.
The EM algorithm is obtained by noting that the posterior of z; is also a Gaussian
z:/t: ~ NMMM) Ü. t,, MM.l)),
(32)
where M++1 U Ū.+1 + cI; see Tipping and Bishop [1999b, Eq. 6). Using the mean and covariance of this Gaussian in the following M-step, get
2+1
Σ E )
Σ E 2,2 1
U
Σ t.t U
Σ
(M
M M. tst U
(33)
SU (eMEL + M Ü SÜ,TSÜ
SÜ (I M Ü M U
This is the update shown in Eq. 19.
13

---

A.3 Details of Hessian Matching for Binary Logistic Regression
We derive the Hessian Matching equation given in Eq. 21. For binary logistic regression, the Hessian with respect is of size Px P. We start by writing the expression for the Hessian of (41 Dz+1),
8 Gete (0 Det1)
Σ JED
0(10))))))0
(34)
0000
where ('()= - The sum can be written conveniently in a P x P matrix form by defining diagonal matrix B as shown below:
1' 0
0
2 0
4 B E+1
de+1
$
r++1
0c+1)
where we denote the number of examples in De+1 by N. Similarly, the Hessian of the K-prior is
22 (0 U. wi)
Kr
I00I +
Σ
wxyed (uln"u(uu (u 0c+1 u
2000
Wi
0c+1)
(35)
W2 (u Oc+1)
0
= =600I + U.
UT.
WK.eD (u (ux,), 0.-1)
We will denote the diagonal matrix by W.. The Hessian for the K-prior task is obtained in similar fashion. Using these Hessians, we get the condition given in Eq. 21.
A.4 Details of Hessian Matching for Multi-Class Logistic Regression
For multi-class logistic regression, the Hessian with respect vectorized @ is of size PC x PC. To simplify this, we will write the expression in terms of a block corresponding to the second derivative with respect to g(e) and g(c') which is of size P P. In total, there are C2 such blocks for all pairs of (c, d'), using which the full Hessian can be obtained.
As before, start by writing the expression for the Hessian of (0;
(0: (ole
Σ W'(06))(6 jEDI+1
(36)
where (+ [ denotes the entry (c, of the P x P matrix jf (uug). This can be written
conveniently in a P P matrix form by defining diagonal matrix B (ee)) at O as shown below:
[ (1+11 (())
Lo' 1(04110))
0
(c,e) B
((dd) p H+ 2+1
Φ1+1
T
t+1
((
Next, we write the K-prior task
C
K.
K W.)
Σ
8,
(c)
-(())
1
+ Σ
whyee L (i(e,u..). ((ug))) (37)
14

---

1.0
/
2
3
0
7
6
Accuracy
T2
0.8
4
S
U
4
7
9
/
G
4
J
#mem=160
T3
0.7
betch
DVC
our
#mem=10
74
7
3
0.6
T1
T2
T3
T4 T5
(a) EM (our) SVD
(b) Representation of learned memories
Figure 7: In (a) we compare our method with the SVD approach and batch training (oracle) on multi-label regression task under two memory scenarios: small (10) and large (160). Our method achieves nearly the same accuracy as the SVD approach in both memory scenarios. However, for Tasks and 5, our method with small memory size outperforms the SVD approach, suggesting superior numerical robustness to gradient reconstruction errors caused by limited memory. In (b) we visualize the memories as more tasks are added. Each row shows the eigenimages sorted in decreasing eigenvalues from left to right. The first 2 images are consistent, and the left images with smaller eigenvalues (3rd image onward) show the merged features of digits as new task is learned.
The Hessian is given by following (we denote the indicator by I[(c c') which when = c¹),
(0; w.) = ]l(cc= d') +
K
Wh (uu ()
Up U up
= _l(c= d'6pIP
0
0 U.
+ U,
wk. (uk,i)))
(38)
We will denote the diagonal matrix at 0.+1 by wled', The K-prior at task + 1 is defined similar fashion. Now, applying Hessian matching, get the following condition for all (c, c') pair.
U W (ad) t+1 V T(c c.) 1et11 I D B
(add LI + U, U.Wrle)) U.
(39)
There are C2 such conditions, each involving We choose to match the over C conditions for the pair (), c). that is, when c. The advantage of this choice is that the form of Hessian matching is similar to the binary case. The only modification needed compute the diagonal matrices by summing of the matrix (f), which gives rise the matrices shown in Eq. 27.
P matrix. Since one such condition is sufficient,
B Experiment Details
Our code is available at tttpps://gitube com/team wttpa///aaaappppooooaaaaaa-e-eee aaproo--bayes/compaat. memory_cooe For all experiments, we use NVIDIA RTX 6000 Ada for experiments on Permuted-MNiST and TiyllmeiiymmmeeNee. For other experiments, use NVIDIA RTX-3090.
B.1 Multi-Output Regression on Split-MNIST
Experiment setting. We consider the following hyperparameters:
For feature map Φ(x), we use the identity map where the raw pixel values of the image serve directly features.
15

---

Table 1: The result on the Split-MNIST task: the performance of batch training is 0.849 + 0.008.
# +Memory
10
20
40
160
320
Our
0.697 + 0.004
0.744 + 0.001
0.798 + 0.000
0.820 + 0.000
0.815 + 0.001
SVD
0.639 + 0.002
0.685 + 0.003
0.745 + 0.001
0.815 + 0.002
0.828 + 0.002
For hyperparameters of learning model parameter O,, we use Adam optimizer with learning
Ir 10-3 3, We use 100 epochs for each task. For the weight-space regularization hyperparameter 8. we use 5 = 0.01.
For hyperparameters of learning memory Uz+1, we run 10, 000 iterations for each task. For the noise hyperparameter for the EM algorithm, we use =003
Multi-output linear regression We show that our EM algorithm for Hessian matching achieves compact memory. We conduct a continual multi-label regression task on Split-MNIST [Zenke al., 2017], where the MNIST dataset is divided into five subsets, each corresponding to binary classification task with label pairs (0.1), (2,3), (4.5), (6,7), and (8,9). To formulate the binary classification task as a regression problem, we map each label A E {0....;9 9) into a vector 1; To where 1g E R 10 is a one-hot vector with only the kth component set to one and all others set to zero, and 1 E R.00 is vector of ones. For the feature map, we use o(x) z 7844, that is, the vectorized pixel intensities of an image. For the memory, we use constant memory size per task and accumulate its size as the model trains new task. After training the model, measure the classification accuracy using the entire test dataset for evaluation.
Table 1 shows the averaged accuracy of our method with the SVD approach over three seeds across memory size {10, 20, 40, 160, 320) to investigate the memory efficiency of our method. The SVD approach obtains memory Ur+1 by applying SVD to the right side of (q,() Additionally, we report the result of the batch training, meaning that all tasks are trained without sequentially updating the memory and thus is regarded as the oracle performance. This result shows that our EM method is more effective than the SVD approach when using a smaller memory size (10, 20, and 40).
Fig. 7a compares our method with the SVD approach using memory sizes 10, 160, evaluated after each new task, to investigate its effectiveness with smaller memory. The x-axis denotes the task index, and the y-axis represents the average accuracy over Tasks 1 to t for t = ....., 5; for example, the result T-3 indicates the average accuracy over Tasks 1-3 after training on Task 3. The results show that our method achieves nearly the same accuracy as the SVD approach up to Task 3 and outperforms on Tasks and 5. This demonstrates the robustness of our method training later tasks, where gradient reconstruction errors are more likely to accumulate, making it harder to retain the performance on previously learned tasks. This claim further supported by the additional results in Table 2, which show that the performance of SVD past Tasks I and 2, evaluated after learning the final task, decreases substantially when a smaller memory size is used.
Additional results. Table 2 compares the performances of our method and the SVD approach, which regarded as the oracle method. For memory size K1+1, the SVD approach obtains the optimal memory
U UIKK4 diag( 2/2 di
sequentially where the columns of cigenvectors Ue I 11 x with its rank R and the corresponding cigenvalues d Ref are given by eigh . 1 (+1 ) Udiag(d) for the feature matrix
L(+1 R N and the previous memory U, E R # The AMem denotes the memory size allocated per task; for example, if AMem 10 is used, the memory increases by 10 for each task.
This result demonstrates that our method is effective in mitigating catastrophic forgetting, especially when using small amount of memory, because the performance on previous tasks, achieved by our method, tends be superior to that of the SVD approach.
B.2 Binary Logistic Regression on the Four-Moon Data Set
Experiment setting. We split all data points of Four-moon datasets into 4 tasks of [500, 500, 500, 500] where the task proceeds as shown in Fig. 8.
16

---

Table Multi-output regression on Split-MNIST over runs. The performance of batch training obtained as 0.839, comparable to the Avg of each method.
AMem
Method
Task 1
Task
Task 3
Task 4
Task
Avg
Ours
0,920 + 0.001
0,474 + 0.012
0,616 + 0.00000000 0.003
0,602
0.001
0.874
0.001
0,697 + 0.004
10
SVD
0.776 + 0.011
0.337 +: 0.005
0.592 = 0.006
0.587 0.004
0,905
0.002
0.639 : 0.002
Ours
0.941 + 0.000
0,566 :: 0.001
0.695 + 0.002
0.680
0.003
0.837
0.000
0.744 : 0.001
20
SVD
+:
0.006
0.632
0.641
0.004
0.685 1:
40
Ours
0.959 +: 0.000
0.689
0.001
0.740 + 0.001
0.786
0.001
0.816
0.001
0.798 + 0.000
SVD>
0.877 + 0.006
0.535 0.003
0.688 0.004
0.733
0.004
0.889
0.001
0.745 : 0.001
160
Ours
0,966 + 0.000
0,764
0.002
0.738
0.001
0.848
0.001
0.785
0.000
0.820 +: 0.000
SVD
0.895 i 0.006
0.687 +: 0.006
0,744 + 0.002
0,862
0.003
0.001 0.886
0.815 1 0.002
Ours
0.965 + 0.000
0.775
0.002
0.720 = 0.001
0.842
0.001
0.773
0.001
0.815 : 0.001
320
SVD
0.901 + 0.006
0.730 1: 0.005
0.751 + 0.004
0.874
0.003
0.887
0.002
0.828 + 0.002
1.0-
0.5
0,0
-3.0
-25
-20
-1.
-1.0
2
Figure 8: Sequences of four-moon classification task
We consider the following hyperparameters:
For feature map Φ(x), we use polynomial feature poly(5) yielding 21 feature dimension. We use the scikit-learn package.
For hyperparameters of learning model parameter 0,, we use Adam optimizer with learning Ir 0.01. We set the number of iterations across (10000, 20000) to learn the model parameter O, for each task; we confirm the convergence of the training loss. For weight-space regularization hyperparameter 8, we set 10
For hyperparameters of learning memory U++1 we set the number of iterations across (50, 100, 2003 for each task. For the memory size, we consider {7, 14, 21} memory for each task and accumulate the memory as the task proceeds. For the noise hyperparameter e, we conduct grid search over {10 10-4,,1000 1,10 7,105) and use 10
Fig. shows the results using the and 21 memory per task. The result using 14 memory is reported in Section 4 of the main manuscript. This confirms that our method can match the decision boundary of batch training closely even when using small amount of memory, as shown in Fig. 9a. As more memory used, our decision boundary becomes more equal to that of the batch training, as shown in Fig. 9b.
Additional results across varying memory size.
B.3 Binary logistic regression USPS odd
dataset
Experiment setting. We consider the following hyperparameters:
For feature map (xx), we polynomial feature poly(1) yielding 256 feature dimension. We use the scikit-learn package.
For hyperparameters of learning model parameter 0,, we use Adam optimizer with learning Ir 0.1. We 5000 iterations learn the model parameter O, for each task. For the weight- space regularization hyperparameter 6, we conduct grid search over 5 (0.5, 0.1, 0.05) and use 0.5 for K-prior and our method.
For hyperparameters of learning memory U we use 5000 iterations for each task. For the noise hyperparameter conduct grid search over {10-3 10-4 10-5)
17

---

daptaiion for task-1
adaptaion for task-2
ada on for task-3
adactaion for task-4
M
W
memory per task
adaptaion for task-1
adaptaion for task-2
adaptaion for task-3
adaptaion for task-4
W
W
(b) 21 memory per task
Figure 9: Investigation on varying memory size used for K-prior.
Investigation on varying weight-space regularization hyperparameter for K-prior. Fig. 10 compares K-prior and method across the varying hyperparameters 5 {0.5, 0.1, 0.05) that control the importance of weight-space regularization hyperparameter term for K-prior. This result confirms that method improves the memory efficiency in general and is more effective at {0.1, 0.5) where the K- K-prior can match the result of batch training more closely.
0.55
0.85
0.55
accuracy
accuracy
accuracy
0.85
0.85
batch
batch k-prio
batch
0.80
k-prior
0.80
0.8
k k-pror
10
20
10
20
10
20
memory size (36 past data)
memory size (% past data)
memory size (% of past data)
(a) 8 0.05
(b) 8 0.1
(c) 5 0.5
Figure 10: Investigation on varying hyperparameter 5 (0.05, 0.5) for K-prior
Investigation varying noise hyperparameter EM algorithm for PPCA. Table 3 compares the performances of our method across varying noise hyperparameter e in the EM algorithm for PPCA where we run 5 experiments with different random seeds. This confirms that our method is consistent over the noise hyperparameter e.
B.4 Multi-label Logistic Regression Split CIFAR-10, CIFAR-100, and Tiny TinyImageNet
Experiment setting.
We consider the following hyperparameters:
For feature map Φ(x), we use the feature extractor of ResNet-50 and Vision Transformer ( (Vit) that are pretrained by CLIP. For Split CIFAR-10, we use Vit-B/32, meaning the base-scale model with 32 patch size. For Split CIFAR- 100 and Split Tiny-ImageNNt, we use Vit-I/14, meaning the large-scale model size with 14 patch size. The feature extractors of ResNet-50 and Vit yield 2048 features and 768 features, respectively.
18

---

Table 3: Investigation on the effect of noise hyperparameter e {10-3, 10- 100-).
AMem
Task 1
Task 2
Task
Task
Task
Avg
10
0.769 + 0.046
0.840 + 0.012
0.566 0.033
0.982 :: 0.005
0.946 +: 0.002
0.820 + 0.013
1%
10-
0.791 + 0.029
0.855 + 0.014
0,623
0.022
0.983 :: 0.003
0.952 +: 0.005
0.840 + 0.007
10
0.772 + 0.043
0.872 0.010
0.627
0.032
0.980
0.003
0.954 + 0.002
0.841 i 0.008
a
0.860 + 0.010
0.913 + 0.002
0,689
0.020
0.979 + 0.003
0.934 + 0.003
0.875 + 0.003
2%
10
0.885 0.012
0.885
0.018
0.724
0.008
0.981
0.003
0.933 + 0.005
0.881 + 0.001
10-5
0.891 = 0.006
0.012 0.893
0.715 0.010
0,982 0.003
0.937 + 0.002
0.883 : 0.003
10 3
0,932 +: 0.002
0.931
0.003
0.810
0.011
0.980
0.003
0.920 1: 0.003
0.915 + 0.002
5%
10
0.950 + 0.002
0.921
0.003
0.819
0.012
0.983 + 0.001
0.915 + 0.005
0.918 + 0.002
10
0.948 + 0.002
0.921 + 0.007
0.798
0.009
0.984 t: 0.002
0.915 +: 0.004
0.913 + 0.002
10
0,933 + 0.002
0.944 + 0.005
0.876
0.008
0.972 +: 0.004
0.878 +: 0.003
0.921 + 0.002
10%
10
0.962 + 0.002
0.937 + 0.003
0.853
0.009
0.976 :: 0.003
0.884 +: 0.003
0.923 + 0.002
5
0.969 + 0.003
0.934 + 0.002
0.855
0.009
0.977 :: 0.003
0.889 + 0.003
0.925 + 0.001
10
0.949 + 0.003
0.946 + 0.002
0.891
0.005
0.971 +: 0.002
0.859 + 0.002
0.923 + 0.001
20%
0.979 + 0.001
0.949 + 0.002
0.877
0.006
0.965 + 0.002
0.872 + 0.003
0.929 + 0.002
10
0.981 + 0.001
0.940 0.001
0.868
0.006
0.967 0.002
0.876 + 0.001
0.927 + 0.002
For hyperparameters of learning model parameter 0z, we use Adam optimizer with learning Ir 0.1. We use 100 iterations for each task with 1024 batch size. For the weight-space regularization hyperparameter d we conduct a grid search over ò 00.1,
For hyperparameters of learning memory U we set 10 iterations for each task. For the noise hyperparameter e, use
10-1
0.97
0.85
0.80
accuracy
aceuracy
0.95
0.75
batch
batch
0.94
batch
our
our
our
0.70
k-prior
k-prior replay
k-prio
replay
replay
0.65
10
10
0.92
10
20
memory size (% past datal
memory size (% past data)
memory size (% of past data)
(a) ResNet-55
(b) Vit-B/32
(c) Vit-L/14
Figure 11: Investigation on varying feature extractor with Split- AplitCCIFAR-1O..
0.8
0.8
0.7
0.7
accuracy
accuracy
accuracy
0.7
0.5
0.6
batch k-prior replay
batch k-prior replay
batch
0.4
0.5
k-rrior
0.3
replay
our
our
our
0.4
2
10
20
10
20
10
20
memory size (% of past data)
memory size 1% of dasa)
memory size (% of past data)
(a) ResNet-50
(b) Vit-B/32
(c) Vit-L/14
Figure 12: Investigation on varying feature extractor with Split-CIFAR-000.
Investigation on varying feature extractor. Fig. 11 shows the results on Cplit-CIFAR 10 across varying feature extractors where the feature extractors of ResNet-50, Vit-B/32, and Vit-L/14 are used in Figures 1 la to 1c, respectively. Similarly, Figures 12 and 13 show the corresponding results on Split-CIFAR- 100 and Aplit-Tinylmaagen respectively. These results confirm that our method improves memory efficiency consistently regardless of the feature extractor.
19

---

0.7
0.7
3.6
0.7
accuracy
0.5
accuracy
accuracy
0.5
86
0.4
0.3
prirr replay
0.4
k-prirr
replay
0.5
k proor
replay
+ our
+ our
0.2
0.4
2
10
20
10
20
10
20
memory size (% of past data)
memory size (% of past data)
memory size (% of past data)
(a) ResNet-50
(b) Vit-B/32
(c) Vit-L/14
Figure 13: Investigation on varying feature extractor with Split- TinylmageNet.
B.5 Additional Results on Permuted MNIST
Following the benchmark experiment setting in [Carta et al., 2023], we also consider Permuted-MNIST consisting of sequence of 5 tasks constructed by permuting pixels of 60,000 training samples of MNIST for each task.
Experiment setting,
We consider the following hyperparameters:
For feature map Φfxx (x)), we use the feature extractor of ResNet-50 and Vision Transformer (Vit) that are pretrained by CLIP. The feature extractor of ResNet-50 and ViT yield 2048 features and 768 features, respectively
For hyperparameters of learning model parameter O,, we use Adam optimizer with learning Ir 0.1. We use 200 iterations for each task. For the weight-space regularization hyperparameter 5, we conduct a grid search over å E {0.01,0.001}.
For hyperparameters of learning memory UL+1 we set 10 iterations for each task. For the noise hyperparameter consider
Additional results. Table 4 shows that our method outperforms other baselines, especially when 1, 2, and percent of data points are used memory. This result confirms that method also improves memory efficiency on a domain incremental task.
Table 4: Performance on Permuted-MNIST across varying amount of memory.
Feature
Method
1 1%
2%
5%
10 %
Replay
0.474 + 0.002
0.508 + 0.002
0.554 + 0.001
0.592 + 0.002
ResNet-50
K-prior
0.465 0.003
0.493 + 0.002
0.534 + 0.002
0.572 0.003
Our
0.490 + 0.001
0.521 + 0.001
0.561 + 0.002
0.592 + 0.003
Replay
0.490 + 0.003
0.524 + 0.002
0.578 + 0.004
0.623 + 0.003
Vit-B/32
K-prior
0.453 + 0.001
0.489 + 0.003
0.549 = 0.005
0.594 + 0.004
Our
0.522 + 0.006
0.545 + 0.007
0.584 0.003
0.613 + 0.004
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our work focuses finding compact memory for continual logistic regression and showing its effectiveness.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention method of our approach section of Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification: Our work does not have theoretical result and focuses on showing empirical improvement of the proposed algorithm.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe experiment configuration in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We will attach our implementation and share it using anonymous Github,
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ Seepi//cuuddes/ooeSSaeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /niis...... wwwpcpppccggpuucceeececceooooooa /tte:/guuddeesoeSuuuiiiii /Ct////oooeisssseeeeeee wwt.suubmissionni.. ieriiisssooooliiii onssiiic) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes
Justification:
We describe experiment configuration in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [Yes]
Question:
Justification: We obtain statistically significant results after multiple runs with different random seeds for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
23

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe experiment configuration in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pettps://neurr http///nnuriip.eeeee cttppbbic/Etiiiccsteeeeeeeeee
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is expected advance continual adaptation procedure effectively, but it seems to have limited relevance to broader societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: this paper does not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: this paper does release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for formatting purposes, including figure and tables.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https:/ /ttpas/////eus..//eeee /ct.///oeeeeeencee///////11 for what should or should not be described.
27


Reviews:
[{"review_id": "B8aRSijqlX", "reviewer": "Reviewer_CdwX", "summary": "This paper proposes an algorithm on utilizing a compact memory for continual linear and logistic regression, which can be used in the last layer of neural networks.\nThe design of the compact memory is based on the theoretical properties of continual linear and logistic regression, and the efficiency of the algorithm is improved by the PPCA method.\nThe algorithm is evaluated on datasets including split MNIST, CIFAR and TinyImageNet.", "questions": "Q1: What is the computational complexity of each step of the algorithm of PPCA_EM_for_logistic? What is the complexity improvement compared to using PCA? Could you please use some specific examples (e.g., for the last layer of NN, there is a 2048 -> 10 classification) to illustrate how significant the complexity improvement is?\n\nQ2: In experiments, I see that you compare with K-prior and standard replay with random sample selection. Could you please also compare with other sample selection methods, like [Aljundi, 2019]?\n\nQ3: Is there any time analysis or ablation study on the time cost and performance comparison of your PPCA-based method and the one with PCA?\n\nQ4: The algorithm is tested on datasets including TinyImageNet, with a similar number of images compared to MNIST and CIFAR. What is the reason for setting the experiment scale to this level? Is it possible to run it on the standard ImageNet?\n\n[1] Aljundi, R., Lin, M., Goujaud, B., & Bengio, Y. (2019). Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32.", "limitations": "N/A on societal impact; for technical see above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I think this looks like a good paper. I have some questions on the algorithm and the experiments in the following Question part; if they are properly clarified I am happy to set the score to accept.\n\nThe proposed algorithm is quite simple in theory yet effective in practice, as illustrated in Sections 3 and 4. The writing is clear. The proposed algorithm is validated by comparing it with several prominent replay-based CL algorithms on several widely-used CL datasets.\n\nI would like to seek some clarification on the computational complexity and the experimental setting. The authors claim that the computation via PPCA is faster than the PCA, but it is not very clear 1) what is the theoretical computational complexity and 2) what is the time cost in running the experiment.", "was_revised": true, "final_justification": "Thank the author for their rebuttal. My concerns are addressed. I will increase my score from 4 to 5 and increase my confidence from 3 to 4.\n\nI agree with most of Reviewer zWmk's suggestions. In particular, you might want to cite some theoretical CL work to justify the Hessian matching, and I leave some suggested papers:\n\n[1] Evron et al. How catastrophic can catastrophic forgetting be in linear regression? COLT'22\n\n[2] Li et al.  Fixed Design Analysis of Regularization-Based Continual Learning. CoLLAs'23\n\n[3] Evron et al. Continual Learning in Linear Classification on Separable Data. ICML'23\n\n[4] Li et al. Memory-statistics tradeoff in continual learning with structural regularization. arXiv:2504.04039.", "rebuttal": ">Q1: What is the computational complexity of each step of the algorithm of PPCA_EM_for_logistic? What is the complexity improvement compared to using PCA? Could you please use some specific examples (e.g., for the last layer of NN, there is a 2048 -> 10 classification) to illustrate how significant the complexity improvement is?\n\nPCA is based on SVD which is costly compared to the EM iterations (both aim to solve Eq. 7). In our problem, SVD is applied to the concatenated matrix $[ \\Phi_{t+1} , U_t ] \\in R^\\{P \\times (T+K)\\} $ with $P$ being the number of parameters, $T$ being the size of new data, and $K$ being the size of memory $U_t$; for example, we use $T=12000$, $P=784$, and memory size $K\\in \\\\{10,20,30,40\\\\}$ (10 memory per task) in our experiment of Figure 2. \n \nThe computational complexity of SVD is $\\mathcal{O}( (T+K) P^{2} )$ for full SVD and $\\mathcal{O}( (T+K) P K )$ for top-$K$ SVD.\n\nThe computational cost of the EM algorithm is $\\mathcal{O}(K^{3} )$ per iteration and is dominated by inverse of $K$-size square matrices $S$ and $(S^{-1}+ MM^{\\top})$; see Algorithm 1. Since $K\\ll T$ and $P$ both, this is faster if the number of iterations is reasonable.\n\nWe note that the main reason to use EM is not always the speed but mainly its numerical stability compared to the version of SVD we used. It is possible to use other variants of SVD that can handle the numerical errors piling up as the new tasks are observed.\n\n\n---\n> Q2: Is there any time analysis or ablation study on the time cost and performance comparison of your PPCA-based method and the one with PCA? Is experiment necessary?\n\nThe performance of SVD was much worse (see Fig 2) which is why we did not conduct the time-analysis but we are happy to add it in the final version of the paper. EM was generally faster but not always, but it was definitely more numerically stable which is the reason why it worked better.\n\n \n---\n> Q3: In experiments, I see that you compare with K-prior and standard replay with random sample selection. Could you please also compare with other sample selection methods, like [Aljundi, 2019]?\n\nThanks for the suggestion. Below we report some comparison to Aljundi (2019)‘s method called Gradient-based Sample Selection (GSS) and find it to be worse than our method. We adapt the implementation provided by the authors into our setup of Figure 5. We consider several memory sizes from 1 to 20% and compared it to Experience Replay using GSS (ER-GSS) and Random sample selection (ER-RN). Additionally, we compare the ER using  class-wise random sample selection (ER-RN-C) that balances the label within the memory buffer.\n\nOn CIFAR-10, the performance of GSS is slightly worse than that of random selection, but on CIFAR-100, it performs significantly worse, especially when the memory size is small (1% or 2%).\n\n| Dataset     |              |  1 %         | 2 %         | 5 %          | 10%        | 20%        |\n|-------------   |-------------|---------------|---------------|---------------|---------------|---------------|\n|                  | **Ours**     | **89.8 ± 0.2**    | **91.6 ± 0.2**    | **93.2 ± 0.1**    | **93.7 ± 0.1**    | **93.5 ± 0.1**    |\n| CIFAR-10  |**ER-RN-C**   | 87.9 ± 0.3    | 89.5 ± 0.2    | 91.2 ± 0.2    | 92.0 ± 0.3    | 92.3 ± 0.1    |\n|                  |**ER-RN**    | 87.9 ± 0.6    | 89.8 ± 0.5    | 91.4 ± 0.2    | 91.9 ± 0.2    | 92.3 ± 0.2    |                    \n|                  |**ER-GSS**    | 86.9 ± 0.4    | 88.8 ± 0.4    | 91.0 ± 0.2    | 91.8 ± 0.3    | 92.0 ± 0.2    |\n\n\n| Dataset     |              |  1 %         | 2 %         | 5 %          | 10%        | 20%        |\n|---------------|-------------|---------------|---------------|---------------|---------------|---------------|\n|                     | **Ours**      | **78.0 ± 0.4**    | **79.3 ± 0.2**    | **81.1 ± 0.2**    | **81.9 ± 0.1**    | **81.8 ± 0.2**    |\n| CIFAR-100  | **ER-RN-C**   | 56.6 ± 2.3    | 66.2 ± 0.6    | 72.9 ± 0.3    | 78.2 ± 0.2    | 81.8 ± 0.2    |\n|                     | **ER-RN**      | 51.3 ± 1.5    | 64.6 ± 1.3    | 72.9 ± 0.8    | 78.3 ± 0.2    | 81.5 ± 0.2    |\n|                     | **ER-GSS**      | 33.7 ± 2.5    | 54.8 ± 1.0    | 66.7 ± 1.0    | 73.3 ± 0.4    | 78.3 ± 0.2    |\n\nWe hypothesize that this is because GSS fails to preserve label balance within the memory buffer because GSS  is known to select gradient-diverse samples. This issue is likely exacerbated when each task contains a large number of classes (e.g., 2 classes per task for CIFAR-10 vs. 10 classes per task for CIFAR-100). Indeed, the comparison between ER-RN-C and ER-RN on CIFAR-100 shows that the performance of ER-RN, which also does not consider label balance, was worse than that of ER-RN-C with 1% and 2% memory. This result supports our hypothesis on GSS’ performance degradation.\n\n\n---\n> Q4: The algorithm is tested on datasets including TinyImageNet, with a similar number of images compared to MNIST and CIFAR. What is the reason for setting the experiment scale to this level? Is it possible to run it on the standard ImageNet?\n\nThanks again for the suggestion. We used TinyImageNet-200 as the larger dataset, but now have also done experiments on ImageNet-1k where we find our method to again perform better. Following [1], we split the dataset into 10 tasks each with 100 consecutive classes and consider memory size of 0.5% (600 per task), 1% (1200 per task), and  2% (2400 per task) respectively. Below are the results:\n\n| #Memory | Replay        | K-Prior       | Ours             | Batch (Oracle) |\n|:-------:|:-------------:|:-------------:|:----------------:|:------:|\n| 0.5%    | 41.7 ± 0.2    | 36.3 ± 0.6    | **65.2 ± 0.2**    | 77.6   |\n| 1%      | 54.0 ± 0.4    | 49.4 ± 0.1    | **70.4 ± 0.0**    | 77.6   |\n| 2%      | 62.3 ± 0.1    | 59.9 ± 0.5    | **73.9 ± 0.2**    | 77.6   |\n\nWe hope that our explanation and new experiments address your concerns. We are happy to further discuss these with you.\n\n[1] Improving Continual Learning by Accurate Gradient Reconstructions of the Past - TMLR 23"}, {"review_id": "juYWSZfgV9", "reviewer": "Reviewer_ebAG", "summary": "This paper presents a method to construct compact memory representations for continual learning with logistic regression, based on the K-prior framework. The key idea is to perform gradient reconstruction via Hessian matching, utilizing probabilistic PCA (PPCA) to iteratively extract compact memory vectors and their associated weights. The method generalizes from linear to logistic and multi-label regression and is shown to work effectively when using pretrained ViT feature extractors on datasets like CIFAR and TinyImageNet. The approach achieves significant memory efficiency while maintaining or exceeding baseline performance, especially under small memory constraints.", "questions": "Questions:\n1. Why does SVD outperform EM at large data sizes (e.g., in Figure 2(a) at 320 samples)? A brief comment on this would help clarify the method’s relative strengths.\n\nSuggestions:\n1. The summation in Equation (1) uses index j, but the summand still depends on t, not j. This seems inconsistent and could benefit from correction.\n2. On Line 135, … and 𝑓_𝑘^𝜃, defined similarly using 𝜃_𝑡, which should be: “ 𝑓_𝑘^(𝜃_𝑡), defined similarly using 𝜃_𝑡 ”\nIt would be helpful to cite the sources of the datasets used, such as USPS or Four-moon, for completeness and reproducibility.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper provides a principled and scalable solution for learning compact memory in continual learning, grounded in gradient/Hessian matching and the K-prior framework.\n\nThe use of Hessian matching and connection to the optimal gradient reconstruction theory is well-motivated and extended to logistic regression and multi-label classification.\n\nThe experimental results show that the proposed method is more effective than the previous approach, k-prior, as well as the random selection replay method, on CIFAR-10, CIFAR-100, and Tiny ImageNet-200.\n\nWeaknesses:\n\nThe method currently applies only to last-layer adaptation using frozen feature extractors.", "was_revised": true, "final_justification": "Thanks for the rebuttal. My questions have been mostly addressed well. Overall, I believe the paper is technically sound. I maintain my original rating.", "rebuttal": "We thank the reviewer for their encouraging comments. \n\n> Q1. Why does SVD outperform EM at large data sizes (e.g., in Figure 2(a) at 320 samples)? A brief comment on this would help clarify the method’s relative strengths.\n\nThanks for the question. The difference in performance is mostly due to numerical issues (see [our response of Question 1 (Q1) to Reviewer zWmk](https://openreview.net/forum?id=XLa5Puhqzg&noteId=29gUNDP2KA)). For larger memory, SVD can get close to near perfect reconstruction and have less numerical issues.\n\nThanks also for pointing out all the mistakes. We will fix them in the next version of our paper."}, {"review_id": "zKgdhGHlsF", "reviewer": "Reviewer_zWmk", "summary": "The authors develop a method for constructing a set of unlabeled pseudo datapoints (not a subset of the training data) using an EM algorithm for Probabilistic PCA (PPCA) to represent data from past tasks in a continual learning regression setting. They show that using this learned memory for regularization results in improved memory efficiency compared to prior methods which use a subset of the training data for regularization.", "questions": "**Questions**\n- Why is SVD worse in Figure 2 (a)? I might be missing something, but SVD should result in the _optimal_ low-rank representation of the current task's data combined with the compact memory. How can PPCA be better for a small number of data points?\n- How does the cost of PPCA EM compare to other numerical linear algebra methods to compute top $K$ singular vectors and values, e.g. randomized SVD? Can you add more discussion around the choice of PPCA EM and its computational cost?\n- How is convergence of EM determined? How many steps of EM are run in practice?\n- Is experience replay defined anywhere?\n- Are there any other relevant works on continual regression besides Khan and Swaroop [2021]? What about methods for selecting subsets of training data and other works using pseudo-data for continual learning?\n\n**Suggestions**\n\nI will increase my score if these suggestions are followed.\n- It would be very informative to add baselines with non-random subsets of training data. This would provide some insight in how much you gain from using pseudo data points compared to a curated subset of the training data. Also, it is a fair comparison since you have some additional overhead of constructing your compact memory with PPCA EM, so you could use the same amount of compute for the selection of a subset of the training data.\n- Please answer my questions above and add additional discussion in the paper where necessary.\n\n**Minor**\n- Abstract:\n> Our approach can drastically reduce memory size without sacrificing performance, for instance, we can obtain 30% improvement in accuracy over Experience Replay that uses 1% of the data for memory when adapting it with the the pre-trained Vision transformer as a feature extractor.\n\nThe second half of the sentence is hard to parse and not completely clear: “that uses 1% of the data for memory” -> does this mean Experience Replay uses 1% of the training data? If yes, how much less memory does the proposed method need? “when adapting it with the the pre-trained Vision transformer as a feature extractor” -> it’s unclear how that relates to the proposed method.\n- Please add references to the appendix.\n- Line 45/46: “(…) the memory size can be large because $\\mathcal{M}_t$ in practice is chosen to be a subset of the data.” Is the point that the cardinality of the subset is a fixed percentage of the data, so for large datasets the subset might be large as well? The same can be said about compact memory (of course the goal is the the fixed percentage per task is smaller).\n- $K_t$ in line 106/107 is the rank of the input matrix, in line 117 it is the memory size which should be is probably < the rank of the input matrix?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- Well-written and mostly clear.\n- Conceptually elegant and simple approach (using unlabeled pseudo data memory) to an important problem.\n- Sufficient details about the method (see the appendix).\n- Well-designed experiments and visualizations.\n\n**Weaknesses**\n- Some more discussion of the SVD result in Figure 2 (a) (see questions) and the choice of using PPCA instead of SVD would be helpful.\n- Discussion of literature about (1) other approaches that construct pseudo data (for continual learning), (2) continual learning in regression settings, and (3) training data subset selection methods is missing.\n- Comparing against baselines with non-randomly chosen subsets of the training data would be more informative.\n- Discussion of overall computational cost compared to the other methods would be useful.", "was_revised": true, "final_justification": "The authors followed the suggestions I detailed in my review (non-random training data subset selection experiment and addressing my questions). The other reviewers had similar questions (e.g. PPCA EM vs. SVD) which the authors addressed. Hence, as promised I increased my score and recommend to accept the paper.", "rebuttal": ">  Q1. Why is SVD worse in Figure 2 (a)? I might be missing something, but SVD should result in the optimal low-rank representation of the current task's data combined with the compact memory. How can PPCA be better for a small number of data points?\n\nThanks for the question. This is most likely due to the piling up of numerical errors in SVD. This can be clearly seen by looking at the averaged test accuracy over Task $1$ to $t$ for $t=1,..,5$ during training. The following table shows that SVD performance gets suddenly worse after Task $3$; for example, the column of Task $3$ shows the averaged test accuracy over Task $1$ to $3$. This holds for all memory sizes 10, 20, 40, and 160, especially when the smaller memory is used.\n\n| Method         | Task 1       | Task 2       | Task 3       | Task 4       | Task 5       |\n|----------------|--------------|--------------|--------------|--------------|--------------|\n| **EM-10**       | 99.8 ± 0.00  | 93.7 ± 0.00  | 86.8 ± 0.20  | 77.1 ± 0.30  | 69.7 ± 0.60  |\n| **SVD-10**      | 99.7 ± 0.00  | 93.5 ± 0.10  | 86.1 ± 0.30  | 74.8 ± 0.00  | 63.9 ± 0.40  |\n| *Gap (EM − SVD)*| **+0.1**     | **+0.2**     | **+0.7**     | **+2.3**     | **+5.8**     |\n| **EM-20**       | 99.7 ± 0.00  | 94.7 ± 0.00  | 87.8 ± 0.20  | 80.6 ± 0.20  | 74.3 ± 0.10  |\n| **SVD-20**      | 99.8 ± 0.00  | 94.7 ± 0.00  | 86.7 ± 0.30  | 77.4 ± 0.20  | 68.5 ± 0.50  |\n| *Gap (EM − SVD)*| **−0.1**     | **+0.0**     | **+1.1**     | **+3.2**     | **+5.8**     |\n| **EM-40**       | 99.8 ± 0.00  | 95.2 ± 0.00  | 89.9 ± 0.10  | 85.9 ± 0.10  | 79.8 ± 0.10  |\n| **SVD-40**      | 99.7 ± 0.00  | 95.2 ± 0.00  | 88.6 ± 0.40  | 81.7 ± 0.20  | 74.5 ± 0.10  |\n| *Gap (EM − SVD)*| **+0.1**     | **+0.0**     | **+1.3**     | **+4.2**     | **+5.3**     |\n| **EM-160**      | 99.8 ± 0.00  | 95.2 ± 0.00  | 91.9 ± 0.00  | 88.0 ± 0.00  | 81.8 ± 0.30  |\n| **SVD-160**     | 99.8 ± 0.00  | 95.7 ± 0.10  | 91.4 ± 0.20  | 88.1 ± 0.30  | 81.4 ± 0.30  |\n| *Gap (EM − SVD)*| **+0.0**     | **−0.5**     | **+0.5**     | **−0.1**     | **+0.4**     |\n\nThe EM algorithm is more stable because it relies on matrix inversion where an identity matrix is added (see update of $S$ in Algorithm 1). It is possible that these issues will be fixed by using a better numerical algorithm for SVD or also by tuning the delta in Eq. 3, although the latter is tricky for the sequential setting in continual learning.\n\n---\n>  Q2. How does the cost of PPCA EM compare to other numerical linear algebra methods to compute top  singular vectors and values, e.g. randomized SVD? Can you add more discussion around the choice of PPCA EM and its computational cost? \n\n**computation cost of PPCA EM compared to top-K SVD**: \nIn our problem, SVD is applied to the concatenated matrix $[ \\Phi_{t+1} , U_t ] \\in R^\\{P \\times (T+K)\\} $ with $P$ being the number of parameters, $T$ being the size of new data, and $K$ being the size of memory $U_t$. The computational complexity of SVD is $\\mathcal{O}( (T+K) P K )$ for top-$K$ SVD.\n\nThe computational cost of the EM algorithm is $\\mathcal{O}(K^{3} )$ per iteration and is dominated by inverse of $K$-size square matrices $S$ and $(S^{-1}+ MM^{\\top})$; see Algorithm 1. Since $K\\ll T$ and $P$ both, this is faster if the number of iterations is reasonable. \n\nIn our experiments, sometimes EM took a long number of iterations to converge in which cases the cost saving compared to SVD are not obtained, but the method was still more numerically stable. It is also true that there might be other versions of SVD that could compete with the EM but for our purpose EM worked well and we did not try to find another alternative and only compared to the standard SVD. \n\n**Discussion regarding choice of PPCA EM**: Thanks for the suggestion. We will highlight the numerically stability under accumulation of error as new tasks as the main reason for choosing EM over SVD.\n\n---\n> Q3. How is convergence of EM determined? How many steps of EM are run in practice?\n\nWe keep track the norm of $|| U_{t+1} U_{t+1}^T - U_{t+1}^{old} (U_{t+1}^{old})^T ||$ and stop when it is small enough. However, we sometimes found EM to use maximum iterations (see Appendix for the max iterations for each experiment).\n\n---\n> Q4. Is experience replay defined anywhere?\n\nYes, the equation in the right hand side of Eq. 2 is experience replay.\n\n---\n> Q5. Are there any other relevant works on regression besides Khan and Swaroop [2021]? What about methods for selecting subsets of training data and other works using pseudo-data for continual learning?....... Comparing baselines with non-randomly chosen subsets of the training data would be more informative.\n\n**Regression tasks**: to the best of our knowledge, we have not found relevant work that takes a similar approach to ours, as most continual learning research has focused on image classification task.\n\n**Existing sample selection methods**: [1], one of the representative works, uses gradients to maintain the gradient diversity within the memory. [2] further devises a gradient-based scoring function that also considers adaptation to new tasks. [3] chooses memory samples that are most interfered with by the current task. [4] uses an information-theoretic metric within a Bayesian framework to balance information gain and learnability based on the estimated uncertainty. [5] samples memory per class using prototype vectors defined in the feature space.\n\nIn contrast to these existing approaches, our method directly extracts memory without requiring labels and therefore leads to a more compact representation of past data. We will include a discussion in the main text to clarify these points and add citations to relevant works.\n\nReferences:\n\n[1] Gradient-based Sample Selection for Online Continual Learning – NeurIPS 2019\n \n[2] Online Coreset Selection for Rehearsal-based Continual Learning – ICLR 2022\n \n[3] Online Continual Learning with Maximally Interfered Retrieval – NeurIPS 2019\n \n[4] Information-theoretic Online Memory Selection for Continual Learning – ICLR 2022\n\n[5] iCaRL: Incremental Classifier and Representation Learning – CVPR 2017\n\n**Suggestion**: we have added a comparison to the method of [1] Aljundi (2019). Please refer to our response to [Question 3 (Q3) from Reviewer CdwX](https://openreview.net/forum?id=XLa5Puhqzg&noteId=z9EjRAahrW)."}, {"review_id": "GDfPkRqlSk", "reviewer": "Reviewer_qXrR", "summary": "This paper introduces a novel approach for learning compact memory in continual logistic regression. It employs a Hessian matching-based Expectation-Maximization (EM) algorithm that leverages Probabilistic Principal Component Analysis (PPCA) for K-prior regularization. This method significantly enhances memory efficiency by precisely reconstructing past task gradients. Empirical evaluations across diverse benchmarks, including those with frozen pre-trained neural network features, demonstrate superior memory efficiency and competitive accuracy against baselines, highlighting its potential for scalable continual adaptation.", "questions": "See weaknesses.", "limitations": "The limitations have been correctly discussed in Section 5.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n1. The paper introduces a novel, theoretically sound framework for K-prior regularization. It learns compact memory through a Hessian matching principle derived from gradient matching, solved by an EM-based PPCA algorithm.\n2. A core contribution is the demonstrable improvement in K-prior's memory efficiency by learning compact memory rather than storing raw data. This is a crucial practical advancement for scaling continual learning, particularly for large-scale models.\n3. The paper provides comprehensive experimental validation and extensive ablations.\n4. The code is available\n\n### **Weaknesses**\n\n1. It is better to include larger databases such as imagenet-1k [1] into the experiments to make the results more convincing.\n2. Please clarify what \"compact memory\" entails. Does it mean representing more sample information within the same buffer size? If so, what is the quantitative gain (e.g., how many times more information) compared to storing raw images?\n3. In comparisons with Experience Replay (ER) [2], was the buffer size held constant (allowing for more represented sample information), or was the number of stored samples kept constant? This distinction is crucial for assessing the fairness of the comparison.\n4. Since Nearest Class Mean (NCM) classifiers are effective and widely used for continual learning with frozen feature extractors [3,4,5,6], please justify why storing compact memory representations of *samples* (features) is preferred over maintaining single *class prototypes*, especially if NCM yields comparable or superior performance.\n5. Equation (1) seems problematic. Additionally, some equations (7, 8, 9) are missing punctuation (e.g., periods or commas) at their conclusion.\n\n[1] Daxberger, Erik, et al. \"Improving continual learning by accurate gradient reconstructions of the past.\" *TMLR. 2023.*\n\n[2] Rolnick, David, et al. \"Experience replay for continual learning.\" *NeurIPS. 2019.*\n\n[3] Zhou, Da-Wei, et al. \"Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need.\" *IJCV. 2025.*\n\n[4] Zhou, Da-Wei, et al. \"Expandable subspace ensemble for pre-trained model-based class-incremental learning.\" *CVPR. 2024.*\n\n[5] Prabhu, Ameya, et al. \"Random Representations Outperform Online Continually Learned Representations.\" *NeurIPS. 2024.*\n\n[6] Goswami, Dipam, et al. \"Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning.\" *NeurIPS. 2023.*", "was_revised": true, "final_justification": "I appreciate the detailed rebuttal, which has effectively addressed my primary concerns. The new experiments on ImageNet-1k, in particular, compellingly demonstrate the method’s significant performance gains under strict memory constraints. Accordingly, I have raised my score to 5.", "rebuttal": "We thank the reviewer for their comments. We have tried to address them below and hope to discuss more with the reviewer. If the reviewer's concerns are addressed appropriately, it will be great if they can consider increasing their score. Thanks for reading our work!\n\n> Q1. It is better to include larger databases such as imagenet-1k [1] into the experiments to make the results more convincing.\n\nThanks for the suggestion. We have now included ImageNet-1k where we find our method to again perform better. Following [1], we split the dataset into 10 tasks each with 100 consecutive classes and consider memory size of 0.5% (600 per task), 1% (1200 per task), and 2% (2400 per task) respectively. Below are the results:\n\n| #Memory | Replay        | K-Prior       | Ours             | Batch (Oracle) |\n|:-------:|:-------------:|:-------------:|:----------------:|:------:|\n| 0.5%    | 41.7 ± 0.2    | 36.3 ± 0.6    | **65.2 ± 0.2**    | 77.6   |\n| 1%      | 54.0 ± 0.4    | 49.4 ± 0.1    | **70.4 ± 0.0**    | 77.6   |\n| 2%      | 62.3 ± 0.1    | 59.9 ± 0.5    | **73.9 ± 0.2**    | 77.6   |\n\n---\n\n>  Q2. Please clarify what \"compact memory\" entails. Does it mean representing more sample information within the same buffer size? If so, what is the quantitative gain (e.g., how many times more information) compared to storing raw images?\n\nYes, you are absolutely right. Quantitative gain can be seen in Fig. 4 and 5, for example, for tinyImagNet, to match the performance of our compact memory of 1% we need around 5% of the data for replay and K-prior. Similar results hold for other datasets.\n\n---\n> Q3. In comparisons with Experience Replay (ER) [2], was the buffer size held constant (allowing for more represented sample information), or was the number of stored samples kept constant? This distinction is crucial for assessing the fairness of the comparison.\n\nWe set the memory size (buffer size) per task and increase it whenever the model learns a new task. For example, if a continual learning experiment consists of 5 tasks and the memory size is set to 10 per task, the total memory size increases as [10,20,30,40]. This configuration of the memory size is applied equally to both ER and our method.\n\n\n\n\n\n\n---\n> Q4. Since Nearest Class Mean (NCM) classifiers are effective and widely used for continual learning with frozen feature extractors [3,4,5,6], please justify why storing compact memory representations of samples (features) is preferred over maintaining single class prototypes, especially if NCM yields comparable or superior performance.\n\nThough NCM has been empirically effective for continual learning, to the best of our knowledge, the rationale for how NCM yields superior performance, is not entirely clear. Our work follows the more usual trend to represent the past features in a principled way that aims to recover the gradient of past task loss using a K-prior regularizer, and this could also be considered complimentary to NCM. We are happy to learn more about reviewers point of view and add discussion on such methods in the paper.\n\n---\n> Q5. Equation (1) seems problematic. Additionally, some equations (7, 8, 9) are missing punctuation (e.g., periods or commas) at their conclusion.\n\nThanks for pointing this out. We will fix these mistakes in the next versions."}]

Author rebuttal:

Meta-review: [Overview]
This paper addresses the critical problem of compact memory for continual learning, specifically within the context of logistic regression and the K-prior regularization framework. It introduces a novel approach that formulates memory learning as a Hessian matching problem and solves it using a Probabilistic Principal Component Analysis (PPCA)-based Expectation Maximization (EM) algorithm. The reviews acknowledged the paper's interesting problem setting and the empirical results, with authors providing clarifications and commitments in the rebuttal that satisfactorily addressed key concerns.

[Strengths]
The paper presents a principled and novel methodology for generating compact memory representations by formulating the problem as Hessian matching and solving it via a PPCA-EM algorithm. This offers a theoretically grounded alternative to heuristic memory selection strategies.
Empirical evaluations demonstrate significant improvements in memory efficiency and performance over existing K-prior and experience replay baselines, especially in challenging multi-label classification tasks using pre-trained deep feature extractors (e.g., Split-CIFAR, Split-TinyImageNet), showcasing the method's practical utility.

[Weaknesses]
While the combination of techniques is novel, the conceptual justification for why Hessian matching is the optimal strategy for compact memory could be elaborated further to provide deeper theoretical insights.
The initial submission had some clarity issues in presentation and lacked complete experimental details.

[Recommendation]
The proposed Hessian matching approach for compact memory is a valuable contribution to continual learning, demonstrating strong empirical performance, particularly when integrated with modern pre-trained feature extractors. Although some clarity issues were noted, the authors' rebuttal adequately addressed the major concerns, making the paper a useful contribution to the field. Therefore, we recommend acceptance.
Decision: Accept (poster)