F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning
Authors: Huiping Zhuang
Yuchen Liu
Run He
Kai Tong
Ziqian Zeng
Cen Chen
Yi Wang
Lap-Pui Chau
Venue: NeurIPS 2024 poster (2024)
Keywords: Class incremental learning
closed-form solution
exemplar-free
continual learning
online continual learning
Primary area: online_learning

Abstract: Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resource friendly but often lack accuracy. In this paper, we propose an exemplar-free approach—Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on bench mark datasets demonstrate F-OAL’s robust performance in OCIL scenarios. Code is available at: https://github.com/liuyuchen-cz/F-OAL

F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning
Huiping Zhuang'; Yuchen Liu Run He', Kai Tong', Ziqian Zeng',
Cen Chen 1,4 Yi Wang'. Lap-Pui Chau
South China University of Technology, China
The University of Hong Kong, Hong Kong SAR
'The Hong Kong Polytechnic University, Hong Kong SAR 4Pazhou Lab, Guangzhou, China
(hpzhuang, , runhe, kai Maaitong. zqzeng, cenchen) @scut edu, en, liuyuchen@connectt hku. hk,
fyi-eie. wang, lap-pui chauh edu hk
Abstract
Online Class Incremental Learning (OCIL.) aims to train models incrementally, where data arrive mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-fie free (i.e., data need be stored for replay in production) methods are resource- friendly but often lack accuracy. In this paper, we propose an exemplar-fre free Fortvaddo-- Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAI. does not rely on hack-propagation and is forward-only, sig- nificantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on bench- mark datasets demonstrate F-OAL's robust performance in OCIL scenarios. Code com is available at: httts://gitthut http:/oooooomlliuuaa O cZ/FFOAL
Introduction
Class Incremental Learning (CIL) updates the model incrementally a task-by-task manner with new classes in the task. Traditional CIL most plans for static offline datasets which historical data accessible. However, with the rapid increase of social media and mobile devices, massive amount of image data have been produced online in streaming fashion, and render training models on static data less applicable. To address this, Online Class Incremental Learning (OCIL) is developed taking an online constraint in addition the existing CIL setting. OCIL is a more challenging CIL setting in which data come in mini-bate mini-bbtcches, and the model is trained only in one epoch (i.c., learning from one pass data stream) [14]. The model required achieve high accuracy, fast training time, and
low resource consumption 25
However, CIL techniques (including OCIL) suffer from Catastrophic Forgetting (CF) 27 also known as the erosion of previous knowledge when new data are introduced. The problem becomes
"Equal contribution.
Conresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

more severe in online scenarios since the model can only see data once. Two major factors contribute CF: (I) Using the loss function update the whole network leads to uncompleted feature capturing to adjust linear classifier results in recency bias, which a significantly imbalanced weight distribution, showing preference only current learning data [15].
and diminished global representation [[I]. (2) Using
To address CF in an online setting, replay-basee methods 9 21 are the mainstream solution by preserving old exemplars and revisiting them in new tasks. This strategy has strong performance but resource consuming, while exemplar-free methods show less competitive results.
20 have lower resource consumption but
exemplar-fre branch, delivering encouraging outcomes. ACL. methods pinpoint the iterative pack-propagation as the main factor behind catastrophic forgetring and seek to address it through linear recursive strategies. Remarkably, for the first time, these methods achieve outcomes comparable to those utilizing replay- based techniques.
Recently, Analytic Continual Learning (ACL) 49 methods emerged
There are two limitations in existing ACL methods: (1) Multiple iterations of base training are needed when the model is applied. Subsequently, the acquired knowledge is encoded into regularized feature autocorrelation matrix by analytic re-alignment. The incremental learning phase then unfolds, utilizing the recursive least squares method for updates. This pattern is repeated when the dataset is switched, significantly elevating the temporal cost in an online scenario. (2) Classic ACL methods demand data aggregation from single task, facilitating analytic learning in one fell swoop. This process increases GPU memory footprint and is unsuitable for online contexts where data for each task is presented mini-batcher
To address those limitations, we propose Forward-only Online Analytic Learning (F-OAL) that learns online batch-wise data streams. The F-OAL consists of frozen pre-trained encoder and an Analytic Classifier (AC). The frozen encoder is capable of countering the uncompleted feature representation caused by using the loss function to update and replace the time- tine-consuming base training. With Feature Fusion and Smooth Projection, the encoder provide informative representation for analytic learning. The AC is updated by recursive least square rather than back propaaropagttion to solve recency bias and decrease calculation. Therefore, F-OAL is an exemplar-fie countermeasure to CF and reduces resource consumption since the encoder is frozen and only the AC is updated.
Our main contributions can be concluded as follows:
We present the F-OAL, an exemplar-free technique that achieves high accuracy and low resource consumption together for the OCIL.
F-OAL redefines the OCIL problem into a recursive least square manner and is updated a mini-batch manner.
F-OAL introduces a framework of frozen pre-trained encoder with Feature Fusion to generate representative features and Smooth Projection for recursively updating AC to counter CF.
We conduct massive experiments benchmark datasets with other OCIL baselines. The results demonstrate that F-OAL achieves competitive results with fasting training speed and low GPU footprint.
2 Related Works
Online Class Incremental Learning focuses on extracting knowledge from one pass data stream with new classes in the new task. Time and memory consumption requirements are particularly critical in OCIL, given the fast and large nature of online data streams [13].
are mainstream solutions for OCIL problems by preserving historical exemplars and using them in new tasks. The accuracy is better than exemplar-fi free methods', but the training time and memory consumption are higher.
Replay-based methods 33 25
37 36
38
39
35
Analytic Learning (AL), also referred to pseudo-inverse learning [10 emerges as a solution to the pitfalls of back-propagation by recursive least square. Analytic Learning's computational intensity is demanding since the entire dataset is processed. The obstacle is solved by the block-wise recursive

---

Idkenity
(a) F-OAL leaming agenda of task
R
Matrix
Initialiiaation
Pmo tranne Prozen Encoder
nalytic Classrtter
Task
Exuariie
Mini-batehes
VIT
uudate
R
Task
Online
Information
Data Stream
Storing
F-OAL leaminn agenda of task k
Pro tnined Prozen Encoder
A nalytic Claseitter
R
Task #
Tesk K.
update
R
Task
Figure I: This diagram illustrates the learning agenda of F-OAL. the encoder, features from each block of the ViT are extracted, summed, and averaged to form a composite feature. This feature is then randomly projected into a higher-dimensions space and normalized using the sigmoid function, serving as the activation for updating the classifier. All parameters in the encoder remain frozen. In the analytic classifier section, we introduce to retain historical information and update the linear classifier using recursive least squares. This process is forward-only with no gradients.
Moore-Penrose algorithm 47). which achieves equivalent precision with joint-learning (i.c., training with all data). AL has already been used in online reinforcement learning 22 which shows the potential of AL in OCIL.
Exemplar-free methods can be categorized into regularization--asel prototype-based, and recently emerged ACL methods. Regularization-bane based methods [20] 17
g
44 apply constraints on
the loss function change the gradients preserve knowledge. These solutions reduce resource consumption but commonly can not outperform replay-based methods. Prototype-basee methods
7 (28) mitigate CF by augmenting the prototypes features classify old classes or generating psudo-fouee psudo-features of old classes from new representations. ACL methods represent new branch of the CIL community and show great potential in the OCIL scenario. Analytic Class Incremental Learning (ACIL) [49 is the first approach that applies AL to the CII. problem. The ACIL achieves competitive performance the offline CIL scenario. Gaussian Kernel Embedded Analytic Learning (GKEAL) [48]. following the ACIL, focuses on solving CF in the few-shot scenario by adopting the kernel method. DS-AL 46 overcomes the under- under-fitting problem of AL.
43
45
29 40
32
3 Methodology
3.1 Proposed Framework
Our F-OAL framework consists of two modules:
Encoder. Encoder has two parts: backbone with Feature Fusion and Smooth Projection. The perfor- mance of analytic learning is highly dependent on the quality of the extracted features. Therefore, employ Vision Transformer (VIT) as the backbone instead of CNN, because ViT generates more comprehensive feature representation 30). To further enhance the representativeness of the features, we utilize Feature Fusion. Specifically, we extract the outputs from each block the ViT, sum them, and take their average form the image feature.
Subsequently, we expand this feature into a higher-dimensional space using random projection and apply the sigmoid function for smoothing (i.e.. Smooth Projection). This results in the final activation

---

used to update the classifier. Therefore, the encoder x(-) is defined as:
<(x) = LP (1(x) (,(B(()) + BB Bn(Ba-1 (.....((.)))) (xx)))
(1)
Y2
Smooth Expansion
where B1(-). ... B-(-) are blocks in VIT, LP(:) linear projection and a(-) is sigmoid activation function.
Backbone pre-trained and frozen and weight matrix of projection is sampled from normal distri- bution and is frozen. Freezing the encoder avoids selective learning caused by loss function, which prioritizes features that are easiest to learn rather than the most representative and significantly reduces the number of trainable parameters.
Analytic Classifier. Unlike back- back-propase propagation, we employ the least squares method to obtain the analytic solution for the linear mapping from the activation to the one-hot label. We then recursively update the weight matrix of the linear mapping. This approach is forward-only and does require the use of gradients, resulting in low memory usage and fast computation speed.
3.2
Analytic Solution
Name
Description
Dimension
((X)
Activation of all images.
V x D
One-hot label of all images.
V X M
W
Joint-learning result of classifier's weight matrix.
M x D
Activation matrix of the i-th batch of the k-th task.
S D
yuain
One-hot label matrix of the i-th batch of the k-th task.
S C,
xle)
Activation matrix from the start to the i-th batch of the k-th task.
V, X D
Yuain (k..))
One-hot label matrix from the start.to the i-th batch of the k-th task.
V, X C.
Classifier of the i-th batch of the k-th task.
C2 X
Rx.i
Regularized feature autocorrelation matrix to the n-th batch of the k-th task.
D x
Table 1: Description of notations and their dimensions. Here, V is the number of all images, 7) is the encoder output dimension, M is the number of all classes, S' is the batch size, C2 is the number of classes seen so far, and is the number of images seen far.
The overview shown in Figure | To derive our solution, the notations needed are listed in Table Unlike other methods that treat model training as a back-propagation process, our method formulates the problem using linear regression, which allows for direct computation of the optimal parameters in a closed-fome foreed-fomm solution:
Y = ((X),,
(2)
where <6-) is the pre-trained and forzen encoder and W is the learnable parameter of linear classifier. The learning problem can be rewritten into an optimization form:
argmin Y
o(X)
W||
where || ||r represents the Frobenius norm and y the regularization term. The optimal solution is defined as:
W (6(x)
(6(x))T(XX)) 6(X) + +11) 6(x)TY.
(4)

---

3.3 Learning Agenda
Given dataset {X& " J1"h be the training set of task k (k= = 2, m). Every task is divided into mini-batches. We denote { R,a
k, I. as the mini-batch (ž=1, 2, .... n) of training set of task k. At task 1 batch k, the model aims to optimize
argmin w(..)
ytrain 1,15
(x (x
1,19 |F +>
w(. 2
(5)
F
where
X 1,1 & 1(11,1
(6)
The optimal solution to parameter ((1,)) is given as:
(1.))
(x
Tx 1,1 L.i +iict)))
+1) " (x 1,1:) 1,15*
(x
Regarding x(.)
i,1 and yarain stacks of row activation vectors and one-hot label vectors respectively, observe that (x(A)
1,1:1) (1 1,1:)) (x( and (x 1,2s) yT ytrain are both sums of outer products w.r.t feature vectors. Notice that the solution contains no data correlation. Thus we can further derive Equation into
wh (COVELAN) (x(:))]
t71
[x [x(
yenin 0
x
I.
(8)
(xx1)))-1) T( F ((x(.)) +n -1 (xrfe) yiri
(x(())
Let
R1,-1
(x (x1 T (x
(x1--11)
+
be the regularized feature antocorrelation matrix batch i-1 of task 1, where both historical and current information is encoded in.
Therefore, calculate both R and W in recursive manner by the following theorems when the scrial numbers tasks and batches are given:
Theorem 3.1 For the batch 1 of task k Let the null matrix. Let w (k)
W (h - 1,a)
0
and (k,1) can be calculated via:
(1.1)) (-1 + RR
Xta/T
ytrain
x(0) (e) l,ny"
(10)
k.l
where
v(iTT) -1
Rg,1 R - X
MaaT k.l
(I+ x 1
X(i) A R 1,n-
(11)
Theorem 3.2 For the batch i (i> 1 of task k, (kr))))
can be calculated via:
(hi)) W (hi--1)
R X
T
(yuain
x(a) w ((i 1)
(12)
where
RR RE Rk
TajT
War Re R
R
(13)
Proof. See Appendix A
Thus, we achieve absolute memorization in an exemplar-fre free way with all data used only once. The learning agenda of F-OAL is summarised in Algorithm I
5

---

Algorithm 1 Forward-Only Analytic Learning
Input: Mini-batches (A k,s /1.1
y ma.
Initialization: Identity matrix Ro; Null matrix w for = to m do for to n do if' = then # Theorem 3.1: Load W mak--1n I,n)
and R
Expand W W
I,))
to W (M
Obtain activation X based on x
Update Rx.1 by x-aa k.1 and Rg-
Update ((k.1) by k. k,I else
pdrain w(k-1,m) and Rg:
# Theorem 3.2: Load w(k.i-1) and R
Obtain activation X based on x
Update RA by X and RR,i-1;
Update ((ki))) by x(a). ytain w(k.i-1) and R
end if
end for
end for
3.4 Complexity Analysis
In terms of space complexity, our trainable parameters are only R and W. The R matrix of size D x D, where D is the output dimension of the encoder. In paper, the encoder output dimension 1000. Therefore, according to Equation 9 the size of the R matrix is 1, 000 1, 000. The W matrix has dimensions of C x D, where C' is the number of classes in the target dataset. For example, with CIFAR-100, its size is 100 x 1, 000. The total number of trainable parameters is relatively small and does not require gradients. This results in method using less than 2GB of GPU memory.
In terms of computational complexity, we denote the batch size as S, encoder's output size as D, and class number as C. Therefore, the dimensions of X, Y, R and W are S D, S x C, D x D, and C' x D, respectively. Thus, the calculation is shown below:
The computational complexity for updating R is dominated by the matrix multiplications, thus:
((S))
(14)
The computational complexity for updating W dominated by the matrix multiplications and the matrix inversion:
o(s)) mxx(O ou.K00EE...osshh...55...c....,,, (0s o) max(0(s)),0(0')) (15)
In the OCIL setting, the batch size relatively smaller. Therefore, the overall computational complexity is primarily controlled by D.
3.5 Overhead Analysis
In terms of space overhead, compared to the conventional backbone classifier structure, F-OAL introduces an additional linear projection control the output dimension D of the encoder, and matrix R. where only R is trainable. According to Equation the dimension of R remains fixed size D D. Other methods require more extra space. For instance, LwF 20) employs knowledge distillation, necessitating the storage of additional models, while replay-based methods require extra storage to retain historical samples. In contrast, the overhead introduced by F-OAL,, consisting of an additional matrix and linear layer, is smaller.

---

Melhod
Replay?
CIFAR-100
FGCVAireraft
Tiny-ImageNet
Country211
iCaRLICVPR 2017 3
91.6
95.6
36.4
74.1
91.3
12.2
[2
ASERIAAAT 2021
13
87.2
87.1
25.2
57.4
85.8
13.2
2021)
91,9
DVC(CVPR 9
92.4
97.1
31.7
67.3
91.5
16.1
2023)
LWE(TPAMI 2018
X
69.3
47.0
14.2
40.2
82.5
14
EWCIPNAS 2017)
49,9
27.6
6.1
EASE/VVPR 2024]
91.1
85.0
38.2
76.0
92.0
15.9
LABXCCC [
79.1
73.3
13.5
63.5
86.7
14.5
SLCA/ICV 2023) 40
90.4
93.7
34.3
70.9
88.6
17.8
F-OLL
91.1
96.3
62.2
82.8
91.2
24.4
[C&RL/CVPR 2017)
87.5
93.2
29.8
66.3
87.8
6.5
ERICRA 2019)
84,6
92.1
28.6
54.3
81.6
6.8
ASERIAAAI 2021)
82.0
82.1
14.8
49.4
80.0
6.7
SCRICVPR 2021)
87.7
93.6
50.3
68.7
75.8
8.0
DVCICVPR 2022)
87,8
96.0
27.0
57.2
87.2
9.2
(x)t
PCRICVPR 2023)
81.4
93.9
9.0
34.6
86.1
6.1
TPAM 2018)
64,8
26.3
EWC/PNAS 2017)
X
25.2
21.1
3.0
13.3
44,6
1.9
LAB/ICCV 2023)
X
75.6
67.1
6.3
53.6
82.4
9.3
SLCAICCY 2023)
F-OAL
86.5
92.5
54.0
75.9
87.3
17,5
(CaRLICVPR 2017)
ERACRA 2019)
20.7
4,3
34.0
29.8
13.3
21.0
ASERIAAAI
SCRICVPR 2021)
6.2
38
14.5
11.6
7.7
644
DVC(CVPR 2022)
8.2
2.3
29.7
21.7
P(%)
PCRICVPR 2023)
9/2
4.2
2.7
14
1.7
IwF(TPAMI 2018)
1.3
3.1
4.5
EWCIPNAS 2017)
67.4
81.0
38.8
68.3
20.7
51.5
EASE/CVPR 2024)
6.1
10.7
19.2
12.5
16.8
LAB1CCV 2023)
11.8
12.2
25.0
16.7
SI CA(CCV 2023)
X
7.1
3.4
10.2
12.7
4.2
14.9
F-OAL
5.5
10.1
Table 2: This table shows the comparison results of method with other baselines six datasets. We select three metrics: average accuracy (Aoog), last task accuracy (Ataet). and forgetting rate (F)). Higher values for Aang and Atast indicate better performance, while lower values for F indicate better performance. In Replay? column, replay-based methods are marked with v. Conversely, exemplar- free methods are marked with x. Data Bold are the best within exemplar- free methods, and data underlined are the best considering both categories.
In terms of time overhead, our method primarily consists of forward pass and matrix multiplication, which determined by the output dimension of the encoder. By changing the output dimension of the encoder, we can balance the accuracy and time. According to [16], the backward pass in back- prok-aropagttion (forward pass backward pass) accounts for 70% of the time. Therefore, our method's time overhead is also relatively small.
4 Experiment
To show the effectiveness of F-OAL, we conduct extensive experiments to compare our approach with baseline methods. We build code and reproduce relevant results based on 24 34 All baselines are with pre-trained VIT as the backbone.
4.1 Datasets
We focus coarse-grained data scenarios, such as CIFAR-100, Tiny-ImageNet, and Core50, as well as fine-grained data scenarios, including DTD, FGVCAircraft, and Country211. All these open-source benchmark datasets. However, there other data scenarios, such long-tail distributions, where unbalanced data distribution will make it harder to achieve good performance by only training the classifier. We will study this case in future work.
CIFAR- 1-00 [8] constructed into 10 tasks with disjoint classes, and each task has classes.
Each task has 5,000 images for training and 1,000 for testing.
7

---

CORe50 [23] is benchmark designed for class incremental learning with 9 tasks and 50 classes: 10 classes in the first task and 5 classes in the subsequent 8 tasks. Each class has 2,398 training images and 900 testing images on average.
FGVCAircraft 266 contains 102 different classes of aireraft models. 100 classes are selected and divided into 10 tasks. Each class has 33 training images and 33 testing images on average.
DTD 3 is texture database, organized into 47 classes, 40 classes are selected and divided into 10 tasks, Each class has 40 training images and 40 testing images.
Tiny-ImageNet is subset of ImageNet with 200 classes for training. Each class has 500 images. The test set contains 10,000 images. The dataset evenly divided into 10 tasks.
Country211 contains 211 classes of country images, with 150 train and test images per class. The dataset is evenly divided into 10 tasks with 210 classes chosen.
4.2 Evaluation Metrics
We define as as the accuracy evaluated on the test set of task j after training the network from task 1 through and the average accuracy is defined as
A;
Σ D15
(16)
When i (i.e., the total number of tasks), Am represents the average accuracy by the end of training.
Forgetting rate at task i is defined as Equation 17 Ji represents how much the model has forgot about task j after being trained on task Specifically, y, max (at,)) denotes the best test accuracy tE(1....k-1)
the model has ever achieved on task before learning task k, and ag.j is the test accuracy task after learning task k.
F
fig
(17)
where
fa. = max (ar.) - ak.jt Vj < k.
(18)
le (e(1....k-1)
4.3
Implementation Details
ViT-B (5), pre-trained ImageNet- IK [4), serves as the backbone network for all methods. The data stream is kept identical across all experiments to ensure a fair comparison. The learning rate and batch size to 0.001 and 10, respectively. The optimizer is SGD. We assign 5,000 memory buffer sizes for replay-based methods. In F-OAÍ,, the expansion size is 1,000 (i.e., the output activation size to update AC is 1,000). The regularization term is be All experiments are conducted on single RTX 4070ti GPU 12GB, and an average of 3 runs is reported.
4.4 Result Comparison
We tabulate the average accuracy (Aang), last task accuracy (Atast), and the forgetting rate (F) from the compared methods in Table
For fine-grained datasets such as FGCVAircraft, DTD, and Country211, F-OAL achieves the best performance, with average accuracies of 66.2%, 82.8%, and 24.4%, respectively. The second-best results are 55.6%, 76.0%, and 17.8%, respectively. Similarly, F-OAL also outperforms in last task accuracy. This demonstrates the excellent transferability of our method the OCIL setting, effectively leveraging the feature extraction capabilities of the pre-trained encoder.
On coarse-grained datasets such as CIFAR-100, CORe50, and ImageNe, F-OAL still demon- strates excellent performance, achieving the highest accuracy among all exemplar-free methods, except on Imag- where 0.8% behind EASE average accuracy. Compared the best replay-based methods, it only lags by 1.3%, 0.8%, and 0.3%, respectively.

---

Methods
CIFAR-100
CORe50
FGVCAireraft
DTD
Tiny-ImageNet
Country211
LwF
412
877
135
841
256
EWC
451
922
115
62
1,190
249
iCaRL
832
1,716
53
1671
513
ER
652
1,433
40
21
1,315
404
ASER
5,608
7,700
91
18,597
20,611
SCR
2,843
5,939
88
42
62,996
810
DVC
4,191
9,351
287
130
10,940
2,622
PCR
1,624
3,742
113
53
3,274
1,028
EASE
383
760
147
139
638
304
LAE
252
458
156
140
500
355
SLCA
726
1,416
289
278
1,185
551
F-OAL
261
570
16
507
157
Table 3: Training time including feature extraction is reported in seconds where replay-based methods with 5,000 buffer size. Data in Bold show the fastest time.
Typically, a lower forgetting rate is better, but forgetting is based on accuracy. Therefore, when comparing forgetting rates, it is important consider models with similar accuracy levels. When comparing F-OAL to the well-performing DVC, F-OAL exhibits lower forgetting rate. On CIFAR- 100, F-OAL's forgetting rate is 5.5%, while DVC's is 8.2%. This indicates that F-OAL not only maintains high level of accuracy but also effectively manages forgetting.
2.03
L2 Norm of Weights
Peak GPU Memory Usage (G8)
1.25
1.00
0.53
0.25
10
30
F.38L DWC
SLSA
ADO1
12A
DVC
Curret Tact
Classes of DTD
Comp Tadcs
Figure 3: Visualization of the weights of a lin- ear classifier. The result comes from F-OAL DTD. Based on [15], when recency bias happens, L2 norm of current task significantly larger. In F-OAL, the L2 norm of current task is in a aver- age level.
Figure 2: Peak GPU memory footprint in GB with batch size on CIFAR-100. Replay-based methods with 5,000 buffer size. F-OAL has low GPU footprint since it does not require gra- dients.
4.5 Resource Consumption
GPU. Peak GPU memory footprint on CIFAR- 100 is shown in Figure 2 without using gradient and extra space for exemplars, F-OAL requires a low memory footprint while having good performance (i.e., higher accuracy than LAE and EASE on most datasets).
As we state in Section 3.5,
Training Time. Table illustrates training time including feature extraction. F-OAL is fast with competitive accuracy (i.e., higher accuracy than LAE). Only the classifier and regularized feature autocorrelation matrix are updated, leading to fewer of trainable parameters and fast training speed.
4.6 Countering Recency Bias
As demonstrated Figure 3 the linear classifier of AC obtained through F-OAL training does not exhibit recency bias. Notably, the weights corresponding to the classes in the most recent tasks do not significantly surpass those of the earlier classes. The frozen encoder and the recursive least square updating manner ensure equal treatment for all samples.

---

4.7
Ablation Study
We design the ablation study in Table to verify the effectiveness of the Feature Fusion and Smooth Projection modules. Without these components, the accuracy of F-OAL drops, especially fine-grained datasets.
FF
SP
CIFAR-100
CORe50
FGVCAireraft
DTD
Tiny-ImageNet
Country211
91.1
96.3
62.2
82.8
91.2
24,4
V
90.6
95.3
60,9
80.5
91.4
21.3
V
X
90.7
95.4
58.7
79.3
91.2
22.8
X
90.6
95.4
56.0
71.2
91.4
21.1
Table 4: Average accuracy comparison with Feature Fusion (FF) and Smooth Projection (SP) modules across various datasets. means with and means without.
As Table 5 (See appendix [ shows, we prove analytic classifier is key to high accuracy. Herein, define the Fully Connected Classifier (FCC) as having the identical structure to the AC, but it is updated through
rather than utilizing the and recursive least square. Without AC, the accuracy drops from 91. % 32.4% CIFAR-100.
Regularization Term. Table (See See appendix B shows the effects varying 7. For large volume dataset, F-OAL gives robust performance in wide range of y value (e.g.. 10 10- while small datasets need larger value (e.g.. 102 1). The comprehensive results show that y= is the suitable choice.
Projection Size. Figure 4 (See appendix B) demonstrates the influence of different random projection sizes. The results suggest that the setting of a 1.000-dimensional projection is appropriate.
4.8 Potential Positive and Negative Societal Impacts
The key advantage of our approach is its ability achieve exemplar-fre OCIL with fast training and low memory footprint, offering environmentally friendly and efficient solution for this research track. However, the main limitation of our method lies its reliance on powerful pre-trained encoder. As result, it is crucial for to leverage open-source pre-trained backbone networks from deep learning community, rather than training our own, which will otherwise lead to higher GPU usage and increased resource consumption.
Conclusion
In this paper, propose Forward-only Online Analytic Learning (F-OAL), exemplar-free method for addressing several limitations of the Online Class Incremental Learning scenario. We use Analytic Learning to acquire the optimal solution of the classifier directly instead of training for dozens Leveraging the frozen pre-trained encoder with Feature Fusion
of epochs by back-propagation and Smooth Projection and the Analytic Classifier updated by recursive least square, approach achieves the identical solution to joint-learingg on the whole dataset without preserving any historical exemplars, achieving high accuracy and reducing the resource consumption. Our experiments show the competitive performance F-OAL.
6
Acknowledgment
This research was supported by the National Natural Science Foundation of China (62306117) the Guangzhou Basic and Applied Basic Research Foundation (2024A04J368) 2023 the South China University of Technology-TCL Technology Innovation Fund, the Fundamental Research Funds for the Central Universities (2023ZYXZZ0023, 2024ZYGXZR074) the Guangdong Basic and Applied Basic Research Foundation (2024A151501002) and the CAAI- MindSpore Open Fund developed on Openl Community.
10

---

References
[1] Rahaf Aljundi, Fugene Belilovsky, Timne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances Neural Information Processing Systems, pages 11849-11860) 2019.
[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pages 11816-11822 2019.
[3] Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. pages 3606-3613, 2014.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[6] Arthur Douillard, Matthicu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. Computer vision ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pages 86-102, 2020.
[7] Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guider representation learning for data-free class incremental learning. In European Conference Computer Vision, pages 423-439, 2022.
[8] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-effiieent tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11483-1 1493, 2023.
[9] Yanan Gu, Xu Yang, Kun Wei, and Cheng Deng. Not just selection, but exploration: Online class- incremental continual learning via dual view consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7442-7451, 2022.
[10] Ping Guo, Michael Lyu, and NE Mastorakis, Pscudoinverse Iearning algorithm for feedforward neural networks. Advances in Neural Networks and Applications, pages 321-326 2001.
[11] Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maxi- mization. In International Conference Machine Learning, pages 8109-8126. PMLR, 2022.
[12] Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming lcarning. In 2019 International Conference on Robotics and Automation, pages 9769-9776, 2019.
[13] Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning online scenario. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13926- 13935, 2020.
[14] Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backpropagation. 2018.
[15] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8311-39,,410 2019.
[16] Zhouyuan Huo, Bin Gu, Heng Huang, et al. Decoupled parallel backpropagation with convergence guarantee. In International Conference Machine Learning, pages
PMLR, 2018
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences, pages 3521-3526, 2017.
[18] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.

---

[19] Byung Hyun Lee, Okchul Jung, Jonghyun Choi, and Se Young Chun. Online continual learning on hierarchical label expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11761-11770, 2023.
[20] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1935-2947, 2017.
[21] Huiwei Lin, Baoquan Zhang. Shanshan Feng, Xutao Li, and Yunming Ye. Per: Proxy- broxy-basedooo contrastive replay for online class incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24246 24255, 2023.
[22] Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for lcarning world models online. In International Conference on Learning Representations, 2024.
[23] Vincenzo Lomonaco and Davide Maltoni. Core50: new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17-26, 2017.
[24] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, pages 28-51, 2022.
[25] Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 3589-3599, 2021.
[26] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1306.5:3066.15,, 2013.
[27] Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. 1989.
[28] Zichong Meng. Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, and Yanzhi WAng. Diffclass: Diffusion- based class incremental learning. European Conference on Computer Vision, 2024.
[29] Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Fea- ture translation for exemplar- free innsemincremttal learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3911-3920, 2023.
[30] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, pages 12116 12122,22811111 2021.
[31] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[32] Grzegorz Rypeśé, Sebastian Cygert, Valeriya Khan, Tomasz Trzcinski, Bartosz Michal Zieliński, and Bartlomiej Twardowski. Divide and not forget: Ensemble of selectively trained experts in continual learning. In International Conference on Learning Representations, 2024.
[33] Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongscong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9630-9638, 2021.
[34] Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De- Ce-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprini arXiv: 2023.
[35] Fu- Yun Wang, Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for intremnaatemetalll learning. In European conference on computer vision, pages 398 414, 2022.
[36] Quanziang Wang, Renzhen Wang. Yichen Wu, Xixi Jia, and Deyu Meng. Cba: Improving online continual learning via continual bias adaptor. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19082- 19092, 2023.
[37] Zhen Wang, Liu Liu, Yajing Kong. Jiaxian Guo, and Dacheng Tao. Online continual learning with contrastive vision transformer. In European Conference on Computer Vision, pages 6-550 2022.
[38] Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18764-188 18774, 2023.
12

---

[39] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3014-3023, 2021.
[40] Gengwei Zhang, Liyuan Wang. Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow leamer with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[41] Yaqian Zhang. Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jía. simple but strong baseline for online continual learning: Repeated augmented rehearsal. In Advances in Neural Information Processing Systems, pages 14771 117783,, 2022.
[42] Da-Wes Wei Zhou, Hai-Long Sun, Han-Jia Ye. and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554-2355555 23564,, 2024.
[43] Fei Zhu, Xu-Yas Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu, Prototype augmentation and self- supervision for incremental learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 5871-5880, 2021.
[44] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self sustainnt representation expansion non non-exemper conexxrmlaar llass-incremental learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9296-9305, 2022.
[45] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Sclf-sustaining representation expansion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9296-9305, 2022.
[46] Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. Ds-al: dual-stream analytic learning for exemplar-free class-incremental Icarning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17237-17244, 2024.
[47] Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Blockwise recursive moore-penrose inverse for network learning. IEEE Transactions on Systems, Man, and Cyberneties: Systems, 52(5):3337733255 2021.
[48] Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7746-7755, 2023.
[49] Huiping Zhuang. Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances Neural Information Processing Systems, pages 11602-11614, 2022.
13

---

A Appendix
Proof of Theorem 3.1 and 3.2
For Theorem 3.1, we start with proving the case in batch I of task &.
According to Equation g of task & I, we can expand the stacked activation and label matrixes:
yauin
x mi m. + 71
(19)
parT 1
drain
Hence, batch task k, have
X yirii
X p(a) m.
m.d + 7
X(aiT x(a)
(20)
X y
k-in
x yuma
We have defined regularized feature autocorrelation matrix R:-1.n via
11
ΣΣ i=1
p x m mi
+
7
(21)
m=
To facilitate subsequent calculations, here we also define cross-corrclation matrix Q 1m11,
2 1,-
x ptrai
POTT
drain
(22)
Thus, we can rewrite Equation 20 as
(8-1m)) -R--1nn
(23)
Therefore, at batch task have
(K.1) 1R,11
(24)
From Equation 21 we can recursively calculate Rk,1 from R
R RT-L.a
x k,
Xx,1
(25)
According the Woodbury matrix identity, we
(A+ --UV A A UC
1 V/
(26)
Let 1 U-
X E1: I, V ( in Equation 26 have
R
R R
R 1
x
R
pOrT
R
(27)
Hence, R3 can be recursively updated using last task counterpart R and current data (i.c., xx.1). This proves the recursive calculation in Equation TT
Next, we derive the recursive formulation of W (k.1) QK. at the batch of task k:
To this end, we also recuse the cross-correlation matrix
ex11
T pinin 1.1
porT k 1,1
ptnin
1mm1n
pOOT yimin
k,
1'k-1,n +
X ydrin
k,
(28)
where
Q'x-1.- Qe-1,
(29)
where daa the dimension of activation and dyy is dimension of total classes of task k. Note that the concatena- tion in Equation 29 is due to that Y k.2 of task contains more data classes (hence more columns) than Y k.m
Let K F..1 [ (((a k, Rs -.-
X
Since I K K-1 k, KF.1 (I+ X(( R 1.m
X
have KF.1
1 Kx.1 (0) Rs X IoT
Therefore.
k
14

---

R X (0)T
R.1 (I x.
Rs 1 x xx1)
RR,1 X
GTT k,1 K1
k,l
RA,1 X x (I K Xb, Ra- X xx.1)
(30)
(Ri-1.n R. X 1 X. R X i
RR,, X
As W (M-ll (W((--bo))
0] has expanded its dimension similar to what Q' ki--,,n does, we
W (k-Lm) RR Q'k Q'k-1.0
(31)
Hence, W (k,1)
can rewritten as
W (k.1) Ra.1 Qx,1 = Rx.1 Q k-1.1 + X x.1) = Ru,1 Q'n-1a + RA.1 X
k. YK.
(32)
By substituting Equation 27 into 1 1, have
Ru.1 Ve-1,-- RE-1.n I 1.n X +I+ X R X - x R Qk-
(33)
(k-1 R k, (I Xt,1
X (k-
k.
According Equation 30 Equation 33 be rewritten
R Q k-1,n
(k-l,y)
R X (alT
X W (1-11,))
(34)
By inserting Equation
33 into Equation
32
we have
W (4:1) W (,n)
R
X kil
X (
+ Rx.1 X y
(35)
w (k=
X k. VYTT X (u-1)
R
which proves the case in the hatch of task k.
For Theorem 3.2, we consider the case at rest batches of task &. In the rest of batches, the number of classes maintain unchanged compared with batch which means column expansion is required. According
Equation 35 we substitute W (k-I,n) by W (k,t-1) Similar substitution is applicd to Ra-1,n with R according to Equation [7 since the shape of regularized feature autocorrelation matrix is remained through whole learning agenda. Then have
(1
Recc T (rKin x x(o)
Rr,
(36)
RR RE RE xlle
(1+ X
RK, Rk,i
plasT
Xei Reii-1,
(37)
which complete the proof.
15

---

Average Accuracy (%)
60
1,000
20
2,000
4,000
6,000
8,000
10,000
Projection Size
CIFAR-100
FGVCAircraft Country211
CORe50 Tiny-lmageNet
DTD
Figure 4: Average accuracy measured in different projection sizes. Due to the time complexity in computing Equation[[] higher projection sizes escalate training time while increase little performance. On small datasets, higher projection sizes may even lead to reduction.
B Appendix
influence different random projection sizes. These experiments reveal higher- dimensional projections may not always improve accuracy, suggesting that projection size must fit the dataset's s volume. If the volume of the dataset is small, high-dimensional projection results in over-fitting the training The maximum accuracy DTD is 1,000 projection size, while the accuracy slightly increases after 1,000. When projection size 2,000, the results FGVCAirerft and CORe50 are little higher than the results on 1,000, but time consumption is twice higher due to the cubic time complexity in the matrix inverse. Thus, we limit the projection size measurement to 2,000 for CORe50 and Tiny- ImageNtt due to the significant increase in time consumption.
Figure demonstrates digurregdmoonstratts
Table 1 justify the contribution of AC. The reduction from 91.1% % 32.4% without AL suggesting that epoch simple pack-propagation
unable to handle the OCIL.
Table varifies the choice as regularization term is suitable for OCIL. Fine-grainee saaeegranned datasets suffer lot from the wrong choice y. The average accuracy on FGVCAircraft drops from 61.1% 22.4% when y is tuned from to 10-3
Dataset
10²
10'
10"
10"
10"3
AC
FCC
Result
CIFAR-100
91.1
91.1
91.1
90.8
90.3
87.2
CORc50
95.3
95.3
FGVCAireraft
55.1
59.5
61.1
47.1
34,9
22.4
91.1
DID
81.5
X
32.4
Country211
Imageee
91.2
23.2
91.2
91.2
90.9
22.6
90.3
89.8
Table 5: Average accuracy of CIFAR- 100 with updating manner in classifier.
Table 6: Average accuracy measured with dif-
ferent regularization terms. 7=1 shows robust
results across all datasets.
16

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The most important idea of this paper is claimed in substract(i.e. forward-only lcarning).
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Main limitation F-OAI. needs powerful pre-trained backbone.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have detailed proof in Apendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
17

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The results can be reproduced in released code.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer:
[Yes]
Justification:
We release the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/public/
quides/Codesuumi ssiopooiicii for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nniss//..//////////e:::eeee cc/public/ qu..des//ooeeSubblesssoolo...... for more details.
18

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
We have detailed information in Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix. supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification:We do not include statistical significance.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is verified.
For asymmetric distributions, the authors should be careful symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
show in tables figures
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: have this Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
19

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics httppe///eerri...cc/pibi///////a http://wwbceehheeeaa.iei.....a..
Answer: [Yes]
Justification:
We follow the rules.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have this content in section 4.8.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We algorithm will not generate harmful content directly.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
20

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: We cite the source code, which is allowed by code author.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that the procedures for this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
22


Reviews:
[{"review_id": "F3bB9xJWcN", "reviewer": "Reviewer_Pgch", "summary": "The paper introduces Foward-only Online Analytic Learning (F-OAL), an exemplar-free approach designed for Online Class Incremental Learning. The method addresses Catastrophic Forgetting by utilizing a pre-trained frozen encoder and a recursive least square updated linear classifier, which significantly reduces memory usage and computational time. The authors conducted extensive experiments to demonstrate the effectiveness of F-OAL on multiple benchmark datasets, showing its superior performance over existing exemplar-free methods and several replay-based methods.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses. I think providing a more comprehensive comparison with recent state-of-the-art works, a complexity analysis and an overhead comparison would help justify the contribution of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The F-OAL framework introduces a forward-only learning mechanism that avoids back-propagation, effectively reducing computational overhead and memory footprint.\n\nBy not relying on exemplar storage, F-OAL maintains data privacy, a crucial requirement in many real-world applications where data sensitivity is a concern.", "weaknesses": "While the paper is strong in many aspects, it lacks a detailed discussion on potential limitations of the proposed method, such as its dependence on the quality of the pre-trained encoder and the challenges that might arise in different data scenarios.\n\nAlthough the paper compares several baseline methods, including more recent and varied techniques could provide a more comprehensive evaluation of F-OAL’s relative performance. Some recent exemplar-free works could be easily generalized to OCIL setting and should be considered include:\n\n[1] Divide and not forget: Ensemble of selectively trained experts in continual learning, ICLR 2024\n\n[2] R-dfcil: Relation-guided representation learning for data-free class incremental learning, ECCV 2022\n\n[3] Self-sustaining representation expansion for non-exemplar class-incremental learning, CVPR 2022\n\n[4] DiffClass: Diffusion-Based Class Incremental Learning, ECCV 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Replies to Reviewer Pgch\n\nThank you for your constructive and detailed feedbacks. We provide detailed responses to your concerns below.\n\n## W1: Dependence on the quality of the pre-trained encoder.\n    \nThank you for the suggestion. Indeed, our method relies on well pre-trained backbones such as ViT and ResNet pre-trained on ImageNet. We acknowledge this as a limitation and will discussion this limitation in the manuscript so readers can fully understand our technique.\n\nOn the other hand, there are many open-source pre-trained backbones available within the deep learning community, which are relatively easy to obtain. Additionally, leveraging pre-trained models for fine-tuning downstream tasks has become a mainstream approach. Hence, although the need for pre-trained backbones is an existing limitation of F-OAL, it is feasible and in line with mainstream.\n\n## W2:  Challenges that might arise in different data scenarios.\n\nThank you for the suggestion. Indeed, we focus on coarse-grained data scenarios, such as CIFAR-100, Tiny ImageNet, and Core50, as well as fine-grained data scenarios, including DTD, FGVC Aircraft, and Country211. However, there are other data scenarios, such as **long-tail distributions**, which we have not addressed in this work. We shall include this discussion in the manuscript.\n\n## W3: Including some recent exemplar-free works.\n\nThank you for pointing out these baselines. However, they seem to be specially designed for non-pre-trained ResNet, and changing their backbones will compromise their performance. Therefore, we have included several alternatives, i.e., EASE [1], LAE [2] and SLCA [3], which are designed with pre-trained ViT and are the SOTA exemplar-free CIL approaches. **Please refer to the uploaded PDF in the general response.**\n\nIn addition, the mentioned baselines [4-7] are good references, and we shall include them in our literature review to complete our review.\n\n[1] Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning, CVPR 2024\n    \n[2] A Unified Continual Learning Framework with General Parameter-Efficient Tuning, ICCV 2023\n    \n[3] SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model, ICCV 2023\n\n[4] Divide and not forget: Ensemble of selectively trained experts in continual learning, ICLR 2024\n\n[5] R-dfcil: Relation-guided representation learning for data-free class incremental learning, ECCV 2022\n\n[6] Self-sustaining representation expansion for non-exemplar class-incremental learning, CVPR 2022\n\n[7] DiffClass: Diffusion-Based Class Incremental Learning, ECCV 2024\n\n\n## L2: A complexity analysis.\n\nThank you for the suggestion, we have included the complexity analysis as follows.\n\nIn terms of space complexity, our trainable parameters are only  _**R**_  and _**W**_. The _**R**_ matrix is of size  D × D , where D is the output dimension of the encoder. In our paper, the encoder output dimension is 1000. Therefore, according to Equation 9, the size of the _**R**_ matrix is  1000 × 1000 . The _**W**_ matrix has dimensions of C × D , where C  is the number of classes in the target dataset. For example, with CIFAR100, its size is 100 × 1000 . The total number of trainable parameters is relatively small and does not require gradients. This results in our method using less than 2GB of memory, as shown in Figure 2.\n\nIn terms of computational complexity, we denote the batch size as S, encoder’s output size as D, and class number as C. Therefore, the dimensions of $X$, $Y$, $R$ and $W$ are S×D, S×C, D×D and C×D, respectively. Thus, the calculation is shown below:\n\nThe computational complexity for updating _**R**_ is dominated by the matrix multiplications, thus:\n$\\text{max}${$\\mathcal{O}(SDC), \\mathcal{O}(SC), \\mathcal{O}(SDC), \\mathcal{O}(D^2C), \\mathcal{O}(DC)$}≈ $\\text{max}${$\\mathcal{O}(SDC), \\mathcal{O}(D^2C)$} \n\nThe computational complexity for updating _**W**_ is dominated by the matrix multiplications and the matrix inversion:\n $\\text{max}${$\\mathcal{O}(SD^2), \\mathcal{O}(S^2), \\mathcal{O}(S^3), \\mathcal{O}(DS^2), \\mathcal{O}(D^2S), \\mathcal{O}(D^2)$} ≈ $\\text{max}${$\\mathcal{O}(S^3), \\mathcal{O}(DS^2), \\mathcal{O}(D^2S)$} \n\n\nIn the OCIL setting, the batch size is relatively smaller. Therefore, the overall computational complexity is primally controlled by $D$.\n\n## L3: Overhead comparison.\n\nIn terms space overhead, compared to the conventional backbone + classifier structure, F-OAL introduces an additional linear projection to control the output dimension  D  of the encoder, and a matrix $R$ , where only $R$ is trainable. According to Equation 9, the dimension of  $R$  remains a fixed size of D × D. Other methods require more extra space. For instance, LwF employs knowledge distillation, necessitating the storage of additional models, while replay-based methods require extra storage to retain historical samples. In contrast, the overhead introduced by F-OAL, consisting of an additional matrix and a linear layer, is smaller.\n\nIn terms of time overhead, our method primarily consists of a forward pass and matrix multiplication, which is determined by output dimension of encoder. By changing the output dimension of encoder, we can balance the accuracy and time.  According to [8], the backward pass in backpropagation (forward pass + backward pass) accounts for 70% of the time. Therefore, our method's time overhead is also relatively small.\n\n[8] Decoupled Parallel Backpropagation with Convergence Guarantee. ICML 2018\n\nBased on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlBMo63vJe", "reviewer": "Reviewer_nHHU", "summary": "The authors address the problem of online class incremental learning (OCIL), where new tasks arrive periodically in a data stream and the trainer seeks to learn these new tasks without catastrophic forgetting of past performance. The paper presents two modes of OCIL; replay-based methods and exemplar-free methods. Replay-based methods offer strong performance but require storing some amount of replay data from the stream to include in incremental training. Exemplar-free methods lift this limitation, but have not thus far achieved comparable performance to replay-based methods. The authors method (forward-only online analytic learning, or F-OAL) uses projections of blocks from a frozen, pre-trained ViT encoder as input to a trainable linear classifier, and then perform recursive least-squares updates on the linear classifier to avoid the catastrophic forgetting that backpropagation would cause in the same setting. The authors extensively compare their method by accuracy, training time, and memory usage to other recently-SOTA methods for OCIL, both replay-based and exemplar-free. The comparisons show that F-OAL significantly improves over existing exemplar-free methods and achieves comparable performance with replay-based methods, while being significantly more efficient in terms of both training time and memory usage. Lastly, the authors ablate their solution and find that keeping the ViT encoder frozen and using their analytically-learned classifier (instead of learning it through backpropogation) are both key ingredients in F-OAL's accuracy.", "questions": "It seems clear to me that the authors method of fusing the ViT blocks $B_i(.)$ with random linear projection is valuable, but also not entirely necessary. The goal of this approach appears two-fold; (1) capture information from different levels of abstraction in the representation that their analytic classifier uses, and (2) be able to control the dimensionality of that representation, which will surely need to be tune-able at training time for different datasets (e.g. to avoid overfitting). The ablation in Appendix B reassures the reader that (2) is necessary, but none of the ablations suggest that (1) is necessary. A simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach with a simpler method that applies the smoothed projection to the last ViT block. I'd be curious to know why the authors chose this particular approach. The paper itself states that this feature fusion was implemented to \"further enhance the representativeness of the features\", but is there any work or experiment they can point to that suggests this?\n\nIn any case, this seems to be the only weak point of the paper, and a clarification or improvement would be nice.", "limitations": "The only concern I would have is that the paper (and its cited works) abuse the notion of \"data privacy\", which I have previously argued against in this review. Works that claim to \"improve data privacy\" without treatment of the staggering amount of literature that has gone into defining and proving what is and is not \"private\" in an information-theoretic, statistical, or engineering-focused sense are likely to muddy the waters for those fields. The paper's setting and solution are already worth publishing; in my opinion, the unsubstantiated data privacy claim is hurting more than it's helping.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is generally quite strong. The F-OAL method is natural and intuitive; overall, it seems like a significant improvement to the Pareto frontier of accurate and efficient OCIL. The comparisons to past SOTA in both replay-based and exemplar-free methods are extensive and compelling. The authors' exposition is clear and logical, and the experiments are largely informative and useful for the reader. The F-OAL method seems pragmatic, and a natural baseline to which all future work in this area should be compared.", "weaknesses": "This paper (and past papers in this vein) suggest that exemplar-free methods are \"good for data privacy\", but there is very little justification for this claim. I understand the basic premise as this: if replay-based methods force you to store some subset of the data stream, which is worse for users' data privacy than exemplar-free methods that don't require such storage. While it's worth noting as a design consideration/feature, I do not agree with this framing as relevant for \"data privacy\". There is no legitimate security model of privacy in the literature that would recognize this as \"higher privacy\". Once the model provider has seen/processed the data by running it through the model, any legitimate security model would view this as data that's been made public. Methods that enhance security (e.g. SMPC, HE, TEEs) and methods that provably reduce statistical privacy leakage (i.e. differential privacy) are orthogonal; these methods can be used interchangeably with both replay-based and exemplar-free methods! In a realistic setting, I can see how exemplar-free methods might help assuage compliance concerns or company-specific rules, but I know of no data privacy regulation or cryptographic threat model where this would be a relevant factor. I would suggest that the authors attempt to correct this misconception in the literature by simply stating the feature in terms of its utility: data need not be stored for replay in production. The reader can implicitly understand that this can have several benefits depending on the circumstances of the deployment.\n\nOtherwise, the only criticism I'd have for the paper is its ablation study. While it's a useful sanity check to see their result of ablating AC and FCC, the results with Frozen are obvious and unnecessary. The ViT-B model was developed for ImageNet-sized datasets, of course it will overfit CIFAR-100! I think there are more useful ablations that could be performed (more on that below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer nHHU\n\nThank you for your positive reviews and helpful suggestions. We provide detailed responses to your concerns below. \n   \n## W1: Abuse the notion of \"data privacy\".\n    \nThank you for raising this important concern. We agree that the use of “data privacy” is less appropriate in this paper. We will remove the use of \"data privacy\" and stick to “exemplar-free” (i.e., data need not be stored for replay in production).\n\n## Q1: Ablation study results with frozen are obvious and unnecessary.\n    \nThank you for the suggestion. We will remove this unnecessary experiment and add more important ablation experiments based on your question.\n\n## Q2: Simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach.\n    \nThank you for pointing out the missing items in our ablation study. We have included a new set of experiments as follows according to your suggestions. The results are shown below.\n    \n| Block-averaging| Smoothed Projection | CIFAR-100 | CORe50 | FGVCAircraft | DTD  | Tiny-ImageNet | Country211 |\n|--------|-----|------|--------|--------------|------|---------------|------------|\n| √   | √  | 91.1      | 96.3   | 62.2         | 82.8 | 91.2          | 24.4       |\n| ×   |   √| 90.6      | 95.3   | 60.9         | 80.5 | 91.4          | 21.3       |\n|√|×|90.7 |95.4 |\t58.7\t|79.3\t|91.2 |\t22.8 |\n|×|×|90.6|95.4|56.0|71.2|91.4|21.1|\n\nWe conduct ablation study to prove the contribution of  block-averaging fusion module and the smoothed projection module. The average accuracies are reported. As the table shows, without two modules, the results of F-OAL are already competitive. The two  modules further improve F-OAL's performance, especially on fine-grained datasets (e.g., DTD, FGVCAircraft and Country211).\n\n## Q3: Is there any work or experiment they can point to that suggests feature fusion works?\n    \nThe idea of feature fusion is inspired by DenseNet [1], which suggests that hidden features are still helpful for generating more representative final output. This approach can be considered equivalent to an ensemble of multiple backbones, which can provide feature diversity for F-OAL with a frozen encoder.\n\n[1] Densely Connected Convolutional Networks, CVPR 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jVlcbjVmPS", "reviewer": "Reviewer_UyXG", "summary": "This paper presents an analytic class incremental learning method that does not need backpropogation. The main idea is to use a pre-trained mode to extract features followed by random projection to higher dimensional space, and then use recursive least squares to update the linear regression weights. By doing so, the closed-form solution solves for all seen data thus guarantees no forgetting. The experiments in several class incremental image classification tasks show superior results than many continual learning baselines.", "questions": "- Does the time recorded in Table 2 for F-OAL include the ViT feature extraction time?", "limitations": "The limitations of the method are not discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The empirical results of the method is very strong compared to other continual learning baselines.", "weaknesses": "- Starting from section 3.2, it's better to give more precise definition for all notations, such as their dimensionality.\n- There are some confusing notations that can be improved. E.g., at line 121 you use $k$ to denote task, but at line 123 you use $k$ to denote batch, then in equation 9 you use change to $n$ to denote batch.\n- There are some writing issues like typos in the main, e.g. line 17, line 117, equation 4 (should be $\\phi(X)^\\top Y$?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer UyXG\n\nThank you for your valuable time in reviewing. We provide detailed information for your concerns below\n\n## W1: Give more precise definition for all notations, such as their dimensionality.\n\nThank you for your suggestion. We have included a following notation table.\n\n| Name          | Description                                                      | Dimension                                      |\n|---------------|------------------------------------------------------------------|------------------------------------------------|\n| $ϕ(X)$         | Activation of all images                                         | $V×D$ ($V$ is the number of all images, $D$ is the encoder output dimension)  |\n| $Y$             | One-hot label of all images                                      | $V×M$ ($M$ is the number of all classes)                               |\n| $\\hat{W}$       | Joint-learning result of classifier's weight matrix              | $M×D$                                           |\n| $X_{k,n}^{(a)}$ | Activation matrix of the n-th batch of the k-th task               | $S×D$ ($S$ is the batch size)                      |\n| $Y_{k,n}^{train}$| One-hot label matrix of the n-th batch of the k-th task            | $S×C_s$ ($C_s$ is the number of classes seen so far)|\n|$ X_{k,1:n}^{(a)}$| Activation matrix from the start to the n-th batch of the k-th task | $V_s×D$ ($V_s$ is the number of images seen so far)|\n| $Y_{k,1:n}^{train}$   | One-hot label matrix from the start to the nth batch of the k-th task | $V_s×C_s$ |\n| $\\hat{W}^{(k,n)}$  | Classifier of the nth batch of the k-th task           | $C_s×D$                                        |\n| $R_{k,n}$       | Regularized feature autocorrelation matrix to the nth batch of the k-th task | $D×D$                                           |\n\n\n## W2: There are some confusing notations that can be improved.\n\nThank you for your comments. We will revise the confusing notions as instructed. For instance, we will use $k$ to represent the task index and $n$ to represent the batch index respectively to avoid confusion.\n\n## W3: There are some writing issues like typos.\n\nThank you for pointing out these writing issues! We have corrected \"class\" to \"classes\" and \"plan\" to \"plans\" in line 17, changed \"a\" to \"an\" in line 117, and revised $ϕ(X)Y$ to $ϕ(X)^{T}Y$ in equation 4. Additionally, we will carefully review the paper to correct any other typos.\n\n## Q1: Does the time recorded in Table 2 for F-OAL include the ViT feature extraction time?\n\nYes, we have documented the entire training process, from the image entering the model to the completion of the model updates.\n\n## L1: The limitations of the method are not discussed.\n\nThank you for the suggestion. The major limitation is that, our method relies on well pre-trained backbones such as ViT and ResNet pre-trained on ImageNet. However, there are many open-source pre-trained backbones available within the deep learning community, which are relatively easy to obtain. Additionally, leveraging pre-trained models for fine-tuning downstream tasks has become a mainstream approach. Hence, although the need for backbones is an existing limitation of F-OAL, it is feasible and in line with mainstream.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z1Ze2kgLgb", "reviewer": "Reviewer_XB6P", "summary": "The author proposed a method called F-OAL for online class incremental learning, which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. In summarize, the contributions are as follows:\n1) The paper presents theF-OAL, which is an exemplar-free method.\n2) F-OAL can be updated in an mini-batch manner;\n3) The methods are evaluated in several benchmarks;", "questions": "See Limitations. \nThe author may want to addresses the questions prpopsed in Limitations.", "limitations": "1. The formula No.4 seems to be wrong. Please check the formula carefully and give the proof process to ensure that the equation is valid.\n2. Why does the formula No.4 show the optimal solution? For each mini-batch of data, the parameter W is the solution that makes the formula No.5 always equal to 0, in theory. But considering all batches of data, parameter W is too idealized and may even overfit the data of the current batch.\n3. It is difficult for the parameter W calculated using a mini-batch of data to have an effect on other batches of data, especially when the distribution of data for different tasks is significantly different. Even if all the parameters W of the mini-batch are combined, I don’t think it can exceed the back-propagation based method in terms of effect, because this combination is linear.\n4. Is the optimization process recursive on all batches of data? Why is the recursive method more efficient? Less computation? GPU parallel computing? Please explain the reason.\n5. The innovation of this paper is insufficient. It seems that the main contribution is to calculate the parameter W by using the forward process and the least squares method. In fact, this method faces many disadvantages, such as overfitting.\n6. Many models that appeared in the comparative experiments did not have annotated references, such as DVC in Table 1.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strengths:\nThe author proposed a method called F-OAL for online class incremental learning, which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time.", "weaknesses": "1) Some of the descriptions are unclear, such as Formula 4. The author may want to give a more vivid explanation;\n2) The innovation is limited, See Limitations No 5;\n3) The author may want to discuss more methods in evulation part;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer XB6P\n    \nThank you for the thorough review. We provide more detailed response below. Hope this can help you with your concerns.\n\n## W1&L1: The formula No.4 seems to be wrong.\n    \nThank you for pointing out the typo. We revise $ϕ(X)Y$ to $ϕ(X)^⊤Y$ in equation 4. \n\n## L2: Why does the formula No.4 show the optimal solution? Is W too idealized and may even overfit the data of the current batch?\n\nWe appologize that our way of delivering the analytical solution can be misleading. In fact, Eq. 4 is the optimal LS solution to loss function in Eq. 3 in the non-CIL sense (like a preliminary **in the case with all the data**). That is, $W$ here **does NOT indicates the weight for any batch**, but for the one computed using **all data**.\n\nOn the other hand, Eq. 5 indicates **the same loss function** as that of Eq. 3, but in the sense of breaking the dataset into $k$ segments (hence the \"$1:k$\" symbols). This leads to the least-squares solution in Eq. 7 (the same as Eq. 4 but in the sense of breaking the dataset into $k$ segments). Here the **optimal solution** $W$ does not guarantee Eq. 5 being 0. The \"optimal\" here indicates that the obtained $W$ allows the objective function in Eq. 5 to be the smallest (not necessarily 0 though).\n\nThe use of Eq. 5 (with many \"$1:k$\") prepares us for the subsequent recursive derivations (i.e., from \"$1:k-1$\" to \"$1:k$\"). Based on this, we are able to derive the F-OAL mainly indicated by Eq. 10 and Eq. 12 in **recursive form**. Note that the $W$ recursively computed using  Eq. 10 or Eq. 12, obtains **the identical result to that of Eq. 7**. That is, the update formula in Eq. 10 or Eq. 12 considers both the current-batch data (e.g., $k$) and the past-batch data (e.g., $1:k-1$). In a sense, the F-OAL does no suffer from catastrophic forgetting (CF) at all!\n\n## L3: Can recursive least square exceed the back-propagation?\n\nOur F-OAL beats BP because it does not suffer from CF (see analysis in L2). According to [1], using BP to update the model is the reason why CF happens, which leads to incomplete feature learning and recency bias [2]. Existing baselines are still based on BP and manage to alleviate CF. We redefine the OCIL into a recursive learning problem to avoid BP. According to [3], the recursive update paradigm could obtain results that are identical to their joint-learning counterparts. \n\nTo give further proof, we provide a quick experiment on MNIST to show that BP suffers from CF, while recursive methods do not. A two-layer MLP is trained on first 5 classes of MNIST (i.e., number 0 to 4) via BP and we test it on these 5 classes. Then the MLP is trained on rest of the 5 classes (i.e., number 5 to 9) via BP incrementally for each class per phase and we test it on the old 5 classes (i.e., number 0 to 4). We apply the identical experiment on our F-OAL using the same MLP. The results are reported below. Please check the source code of the quick experiment in **General Response**.\n\n|         | Acc on 0:4 after trained on 0:4 | Acc on 0:4 after trained on 0:9 |\n|---------|--------------------------|--------------------------|\n| BP      | 98.4                     | 43.2                     |\n| F-OAL     | 98.2                     | 96.4                     |\n\n[1] Online continual learning through mutual information maximization, ICML 2022\n\n[2] Learning a unified classifier incrementally via rebalancing, CVPR 2019\n\n[3] Blockwise recursive Moore–Penrose inverse for network learning. IEEE TMSC-S, 2021\n\n## L4: Is the optimization process recursive on all batches of data? Why is the recursive method more efficient?\n\nYes, the optimization process is  on all batches of data. For **space efficiency**, our method is forward-only and exemplar-free, eliminating the need for gradients and additional memory buffers, thereby significantly reducing the GPU footprint. \n\nFor **computational efficiency**, F-OAL does not have backward pass, resulting in a faster training. Accroding to [1], in BP, the backward pass spends 70% of the entire time.\n\n[1] Decoupled Parallel Backpropagation with Convergence Guarantee. ICML 2018\n\n## L5: The innovation of this paper is insufficient. This method faces many disadvantages, such as overfitting.\n\nOur main contributions are 1) pinpoint BP as the main cause of CF, 2) introducing F-OAL, the recursive method instead of BP that well handles OCIL problem, and 3) fusion module and smoothed projection that enhance the performance of F-OAL.\n\nRegarding the overfitting disadvantage, we must respectably disagree. The F-OAL is mainly linear regression, which usually invites \"under-fitting\" instead of \"over-fitting\" in nature. In addition, we avoid possible over-fitting (due to small data), we have introduced an $L_{2}$ regularization in Eq. 5. However, the F-OAL needs a fixed backbone. This is an existing disadvantage that will be discussed. \n\n\n## L6：Many models that appeared in the comparative experiments did not have annotated references, such as DVC in Table 1.\n\nThank you for pointing out the issue. We will add citations for each method in Table 1 and also review the rest of the paper to ensure there are no other similar problems. The table 1 will be revised in the following form with the right citation:\n\n| Metric | Method                 | CIFAR-100 | CORe50 | FGVCAircraft | DTD  | Tiny-ImageNet | Country211 |\n|--------|------------------------|-----------|--------|--------------|------|---------------|------------|\n| $A_{avg}$(↑)  | DVC(CVPR 2024) [1]     |92.4| 97.1 |33.7 |67.3 |91.5 |16.1|\n...\n\n[1] Not just selection, but exploration: Online class-incremental continual learning via dual view consistency. In CVPR 2022.\n\nBased on these additional results and clarifications, we hope you could consider increasing your score in support of this work. If not, could you kindly let us know what additionally needs to be done in your assessment to make this work ready for publication?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We thank all the reviewers for their time, insightful suggestions and valuable comments. In summary, Reviewer nHHU appreciates that our work is **natural**, **intuitive**, and overall **quite strong**.  The writing is **clear**, **logical**, **informative** and **useful** for the reader. Reviewer UyXG points out that our empirical results are **very strong**. Both Reviewer Pgch and Reviewer XB6P appreciate that our work effectively reduces computational overhead and memory footprint.

We provide point-by-point responses to all reviewers’ comments and concerns. On the other hand, reviewers also point out the recursive mechanism is relatively rare in OCIL, and could not fully understand the F-OAL purely through the derivations. To address this in the response, we have attached the source code for a **quick experiment** comparing the BP and the F-OAL on MNIST dataset as follows. These codes should be able to run freely on any platform such as Colab.

Also the **PDF contains the table for Reviewer Pgch**.


```
import torch

import torch.nn as nn

import torch.optim as optim

from torchvision import datasets, transforms

import torch.nn.init as init

import torch.nn.functional as F


class MLP(nn.Module):

        def __init__(self):

            super(MLP, self).__init__()

            self.fc1 = nn.Linear(28 * 28, 1000)

            self.fc2 = nn.Linear(1000, 10, bias=False)

        def forward(self, x):

            x = self.get_activation(x)

            return self.fc2(x)

        def get_activation(self, x):

            x = x.view(-1, 28 * 28)

            x = torch.relu(self.fc1(x))

            return x

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)

test_dataset = datasets.MNIST('./data', train=False, download=True, transform=transform)

train_dataset_5 = [(data, target) for data, target in train_dataset if target < 5]

test_dataset_5 = [(data, target) for data, target in test_dataset if target < 5]

train_loader_5 = torch.utils.data.DataLoader(train_dataset_5, batch_size=64, shuffle=True)

test_loader_5 = torch.utils.data.DataLoader(test_dataset_5, batch_size=1000, shuffle=False)

model = MLP()

criterion = nn.CrossEntropyLoss()

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)


def train(model, train_loader, criterion, optimizer, epochs=1):

       model.train().cuda()

       for epoch in range(epochs):

           for data, target in train_loader:

               optimizer.zero_grad()

               output = model(data.cuda())

               loss = criterion(output, target.cuda())

               loss.backward()

               optimizer.step()


def test(model, test_loader):

        model.eval().cuda()

        correct = 0

        with torch.no_grad():

            for data, target in test_loader:

                output = model(data.cuda())

                pred = output.argmax(dim=1, keepdim=True)

                correct += pred.eq(target.cuda().view_as(pred)).sum().item()

        print(f'\nTest set: Accuracy: {correct}/{len(test_loader.dataset)} ({100. * correct / len(test_loader.dataset):.1f}%)\n')

        return 100. * correct / len(test_loader.dataset)


print('We train the MLP on first 5 classes of MNIST via 1 epoch of BP (OCIL setting)')

train(model, train_loader_5, criterion, optimizer)

print('\nThe test accuracy on the first 5 classes is:')

old_acc = test(model, test_loader_5)

train_dataset_5to9 = [(data, target - 5) for data, target in train_dataset if target >= 5]

test_dataset_5to9 = [(data, target - 5) for data, target in test_dataset if target >= 5]

train_loader_5to9 = torch.utils.data.DataLoader(train_dataset_5to9, batch_size=64, shuffle=True)

test_loader_5to9 = torch.utils.data.DataLoader(test_dataset_5to9, batch_size=1000, shuffle=False)

print('Then we train the MLP on the rest of 5 classes')

train(model, train_loader_5to9, criterion, optimizer)

print('We test the model on the old 5 classes to see if it forgets the old knowledge')

new_acc = test(model, test_loader_5)

print(f'The accuracy drops from {old_acc:.1f}% to {new_acc:.1f}%, suggesting that BP suffers a lot from CF')

print('\nNow we verify that our approach tackles CF')

print('For this easy dataset, we even do not need a powerful pre-trained backbone, and use the same MLP')

new_mlp = MLP()

R = (torch.eye(1000).float()).cuda().double()

W = (init.zeros_(new_mlp.fc2.weight.t())).double().cuda()


def trainFOAL(new_model, train_loader):

        global R, W

        new_model.train().cuda()

        with torch.no_grad():

            for data, target in train_loader:

                data, target = data.cuda(), target.cuda()

                activation = new_model.get_activation(data).double().cuda()

                label_onehot = F.one_hot(target, 10).double().cuda()

                R = R - R @ activation.t() @ torch.pinverse(
                    torch.eye(data.size(0)).cuda() +
                    activation @ R @ activation.t()) @ activation @ R

                W = W + R @ activation.t() @ (label_onehot - activation @ W)

                new_model.fc2.weight = torch.nn.parameter.Parameter(torch.t(W.float()))


print('Similarly, we still train our model on first 5 classes, and the test accuracy is:')

trainFOAL(new_mlp, train_loader_5)

old_FOAL_acc = test(new_mlp, test_loader_5)

print('Then we train our model on rest 5 classes, and the test accuracy on old classes is:')

trainFOAL(new_mlp, train_loader_5to9)

new_FOAL_acc = test(new_mlp, test_loader_5)

print(f'The small gap between {old_FOAL_acc:.1f}% and {new_FOAL_acc:.1f}% suggests that the our model does not forget the old knowledge')
```

Meta-review: The reviewers were generally appreciative about the responses that authors gave to their concerns, and appreciated both the empirical results and computational efficiency of the proposed method. The authors are strongly encouraged to incorporate additional experiments and explanations they included in their rebuttal. These are, in particular:


* The extensive additional experiments against competitors included in the pdf of the rebuttal.
* The ablation w.r.t. capturing information from different levels of abstraction. 
* The discussion on both computational and space complexity, as well as the comparison of overheads.
* A table with notations.
* A clear statement that evaluations include the feature extraction time.
* A discussion of limitations.
* A discussion on long-tailed distributions.
* A thorough correction of all typos indicated by reviewers, and a careful proofreading.
Decision: Accept (poster)