Title: F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Class incremental learning

closed-form solution

exemplar-free

continual learning

online continual learning
Abstract: Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resource friendly but often lack accuracy. In this paper, we propose an exemplar-free approach—Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on bench mark datasets demonstrate F-OAL’s robust performance in OCIL scenarios. Code is available at: https://github.com/liuyuchen-cz/F-OAL

F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning

Abstract

Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resourcefriendly but often lack accuracy. In this paper, we propose an exemplar-free approach-Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on benchmark datasets demonstrate F-OAL's robust performance in OCIL scenarios. Code is available at: https://github.com/liuyuchen-cz/F-OAL

1 Introduction

Class Incremental Learning (CIL) updates the model incrementally in a task-by-task manner with new classes in the new task. Traditional CIL most plans for static offline datasets which historical data are accessible. However, with the rapid increase of social media and mobile devices, massive amount of image data have been produced online in a streaming fashion, and render training models on static data less applicable. To address this, Online Class Incremental Learning (OCIL) is developed taking an online constraint in addition to the existing CIL setting. OCIL is a more challenging CIL setting in which data come in mini-batches, and the model is trained only in one epoch (i.e., learning from one pass data stream) [14]. The model is required to achieve high accuracy, fast training time, and low resource consumption [25].

However, CIL techniques (including OCIL) suffer from Catastrophic Forgetting (CF) [27], also known as the erosion of previous knowledge when new data are introduced. The problem becomes

more severe in online scenarios since the model can only see data once. Two major factors contribute to CF: (1) Using the loss function to update the whole network leads to uncompleted feature capturing and diminished global representation [11]. (2) Using back-propagation to adjust linear classifier results in recency bias , which is a significantly imbalanced weight distribution, showing preference only on current learning data [15].

To address CF in an online setting, replay-based methods [9, 21] are the mainstream solution by preserving old exemplars and revisiting them in new tasks. This strategy has strong performance but is resource consuming, while exemplar-free methods [17, 20] have lower resource consumption but show less competitive results.

Recently, Analytic Continual Learning (ACL) [49] methods emerged as an exemplar-free branch, delivering encouraging outcomes. ACL methods pinpoint the iterative back-propagation as the main factor behind catastrophic forgetting and seek to address it through linear recursive strategies. Remarkably, for the first time, these methods achieve outcomes comparable to those utilizing replaybased techniques.

There are two limitations in existing ACL methods: (1) Multiple iterations of base training are needed when the model is applied. Subsequently, the acquired knowledge is encoded into a regularized feature autocorrelation matrix by analytic re-alignment. The incremental learning phase then unfolds, utilizing the recursive least squares method for updates. This pattern is repeated when the dataset is switched, significantly elevating the temporal cost in an online scenario. (2) Classic ACL methods demand data aggregation from a single task, facilitating analytic learning in one fell swoop. This process increases GPU memory footprint and is unsuitable for online contexts where data for each task is presented as mini-batches.

To address those limitations, we propose Forward-only Online Analytic Learning (F-OAL) that learns online batch-wise data streams. The F-OAL consists of a frozen pre-trained encoder and an Analytic Classifier (AC). The frozen encoder is capable of countering the uncompleted feature representation caused by using the loss function to update and replace the time-consuming base training. With Feature Fusion and Smooth Projection, the encoder provide informative representation for analytic learning. The AC is updated by recursive least square rather than back-propagation to solve recency bias and decrease calculation. Therefore, F-OAL is an exemplar-free countermeasure to CF and reduces resource consumption since the encoder is frozen and only the AC is updated.

Our main contributions can be concluded as follows:

· We present the F-OAL, an exemplar-free technique that achieves high accuracy and low resource consumption together for the OCIL.

· F-OAL redefines the OCIL problem into a recursive least square manner and is updated in a mini-batch manner.

· F-OAL introduces a framework of frozen pre-trained encoder with Feature Fusion to generate representative features and Smooth Projection for recursively updating AC to counter CF.

· We conduct massive experiments on benchmark datasets with other OCIL baselines. The results demonstrate that F-OAL achieves competitive results with fasting training speed and low GPU footprint.

2 Related Works

Online Class Incremental Learning focuses on extracting knowledge from one pass data stream with new classes in the new task. Time and memory consumption requirements are particularly critical in OCIL, given the fast and large nature of online data streams [13].

Replay-based methods [33, 25, 21, 9, 31, 12, 2, 11, 15, 41, 37, 36, 1, 38, 19, 6, 39, 35, 9] are mainstream solutions for OCIL problems by preserving historical exemplars and using them in new tasks. The accuracy is better than exemplar-free methods', but the training time and memory consumption are higher.

Analytic Learning (AL), also referred to as pseudo-inverse learning [10], emerges as a solution to the pitfalls of back-propagation by recursive least square. Analytic Learning's computational intensity is demanding since the entire dataset is processed. The obstacle is solved by the block-wise recursive

Task 1

(a) F-OAL learning agenda of task 1

Rk-1,n

· · ·

· · ·

R1,n

R0

Rk,n

Identity

Matrix

Initialization

Task  #1

Task  #k-1

Task  #k

Task k

Online

Data

Stream

(b) F-OAL learning agenda of task k

· · ·

· · ·

Information

Storing

· · ·

· · ·

· · ·

· · ·

Mini-batches

Mini-batches

ViT Block

ViT Block

ViT Block

···

Expansion

Analytic Classifier

Linear

ViT Block

ViT Block

ViT Block

···

Expansion

Analytic Classifier

Linear

Moore-Penrose algorithm [47], which achieves equivalent precision with joint-learning (i.e., training with all data). AL has already been used in online reinforcement learning [22], which shows the potential of AL in OCIL.

Exemplar-free methods can be categorized into regularization-based, prototype-based, and recently emerged ACL methods. Regularization-based methods [20, 17, 42, 8, 44] apply constraints on the loss function or change the gradients to preserve knowledge. These solutions reduce resource consumption but commonly can not outperform replay-based methods. Prototype-based methods [43, 45, 29, 40, 32, 7, 28] mitigate CF by augmenting the prototypes of features to classify old classes or generating psudo-features of old classes from new representations. ACL methods represent a new branch of the CIL community and show great potential in the OCIL scenario. Analytic Class Incremental Learning (ACIL) [49] is the first approach that applies AL to the CIL problem. The ACIL achieves a competitive performance in the offline CIL scenario. Gaussian Kernel Embedded Analytic Learning (GKEAL) [48], following the ACIL, focuses on solving CF in the few-shot scenario by adopting the kernel method. DS-AL [46] overcomes the under-fitting problem of AL.

3 Methodology

3.1 Proposed Framework

Our F-OAL framework consists of two modules:

Encoder . Encoder has two parts: backbone with Feature Fusion and Smooth Projection. The performance of analytic learning is highly dependent on the quality of the extracted features. Therefore, we employ Vision Transformer (ViT) [5] as the backbone instead of CNN, because ViT generates more comprehensive feature representation [30]. To further enhance the representativeness of the features, we utilize Feature Fusion. Specifically, we extract the outputs from each block of the ViT, sum them, and take their average to form the image feature.

Subsequently, we expand this feature into a higher-dimensional space using random projection and apply the sigmoid function for smoothing (i.e., Smooth Projection). This results in the final activation

used to update the classifier. Therefore, the encoder ϕ ( · ) is defined as:

where B 1 ( · ) · · · B n ( · ) are blocks in ViT, LP ( · ) is linear projection and σ ( · ) is sigmoid activation function.

Backbone is pre-trained and frozen and weight matrix of projection is sampled from normal distribution and is frozen. Freezing the encoder avoids selective learning caused by loss function, which prioritizes features that are easiest to learn rather than the most representative [11] and significantly reduces the number of trainable parameters.

Analytic Classifier . Unlike back-propagation, we employ the least squares method to obtain the analytic solution for the linear mapping from the activation to the one-hot label. We then recursively update the weight matrix of the linear mapping. This approach is forward-only and does not require the use of gradients, resulting in low memory usage and fast computation speed.

3.2 Analytic Solution

Table 1: Description of notations and their dimensions. Here, V is the number of all images, D is the encoder output dimension, M is the number of all classes, S is the batch size, C s is the number of classes seen so far, and V s is the number of images seen so far.

The overview is shown in Figure 1. To derive our solution, the notations needed are listed in Table 1. Unlike other methods that treat model training as a back-propagation process, our method formulates the problem using linear regression, which allows for direct computation of the optimal parameters in a closed-form solution:

where ϕ ( · ) is the pre-trained and forzen encoder and W is the learnable parameter of linear classifier. The learning problem can be rewritten into an optimization form:

where ∥ · ∥ F represents the Frobenius norm and γ is the regularization term. The optimal solution is defined as:

3.3 Learning Agenda

Given dataset { X train k , Y train k } m 1 be the training set of task k ( k = 1, 2, ... , m ). Every task is divided into n mini-batches. We denote { X train k,i , Y train k,i } , as the i mini-batch ( i =1, 2, ..., n ) of training set of task k .

At task 1 batch k , the model aims to optimize

where

The optimal solution to parameter W (1 ,i ) is given as:

Regarding X ( a ) 1 , 1: i and Y train 1 , 1: i as stacks of row activation vectors and one-hot label vectors respectively, we observe that ( X ( a ) 1 , 1: i ) T ( X ( a ) 1 , 1: i ) and ( X ( a ) 1 , 1: i ) T Y train 1 , 1: i are both sums of outer products w.r.t feature vectors. Notice that the solution contains no data correlation. Thus we can further derive Equation 7 into

Let

be the regularized feature autocorrelation matrix at batch i -1 of task 1, where both historical and current information is encoded in.

Therefore, we can calculate both R and W in a recursive manner by the following theorems when the serial numbers of tasks and batches are given:

Theorem 3.1 For the batch 1 of task k , Let ˆ W (0) be the null matrix. Let ˆ W ( k -1 ,n ) ′ = [ ˆ W ( k -1 ,n ) 0 ] and ˆ W ( k, 1) can be calculated via:

where

Theorem 3.2 For the batch i ( i > 1 ) of task k , ˆ W ( k,i ) can be calculated via:

where

Thus, we achieve absolute memorization in an exemplar-free way with all data used only once. The learning agenda of F-OAL is summarised in Algorithm 1.

Algorithm 1 Forward-Only Analytic Learning

3.4 Complexity Analysis

In terms of space complexity, our trainable parameters are only R and W . The R matrix is of size D × D , where D is the output dimension of the encoder. In our paper, the encoder output dimension is 1000. Therefore, according to Equation 9, the size of the R matrix is 1 , 000 × 1 , 000 . The W matrix has dimensions of C × D , where C is the number of classes in the target dataset. For example, with CIFAR-100, its size is 100 × 1 , 000 . The total number of trainable parameters is relatively small and does not require gradients. This results in our method using less than 2GB of GPU memory.

In terms of computational complexity, we denote the batch size as S , encoder's output size as D , and class number as C . Therefore, the dimensions of X , Y , R and W are S × D , S × C , D × D , and C × D , respectively. Thus, the calculation is shown below:

The computational complexity for updating R is dominated by the matrix multiplications, thus:

The computational complexity for updating W is dominated by the matrix multiplications and the matrix inversion:

In the OCIL setting, the batch size is relatively smaller. Therefore, the overall computational complexity is primarily controlled by D .

3.5 Overhead Analysis

In terms of space overhead, compared to the conventional backbone + classifier structure, F-OAL introduces an additional linear projection to control the output dimension D of the encoder, and a matrix R , where only R is trainable. According to Equation 9, the dimension of R remains a fixed size of D × D . Other methods require more extra space. For instance, LwF [20] employs knowledge distillation, necessitating the storage of additional models, while replay-based methods require extra storage to retain historical samples. In contrast, the overhead introduced by F-OAL, consisting of an additional matrix and a linear layer, is smaller.

In terms of time overhead, our method primarily consists of a forward pass and matrix multiplication, which is determined by the output dimension of the encoder. By changing the output dimension of the encoder, we can balance the accuracy and time. According to [16], the backward pass in back-propagation (forward pass + backward pass) accounts for 70% of the time. Therefore, our method's time overhead is also relatively small.

4 Experiment

To show the effectiveness of F-OAL, we conduct extensive experiments to compare our approach with baseline methods. We build our code and reproduce relevant results based on [24, 34]. All baselines are with pre-trained ViT as the backbone.

4.1 Datasets

We focus on coarse-grained data scenarios, such as CIFAR-100, Tiny-ImageNet, and Core50, as well as fine-grained data scenarios, including DTD, FGVCAircraft, and Country211. All of these are open-source benchmark datasets. However, there are other data scenarios, such as long-tail distributions, where unbalanced data distribution will make it harder to achieve good performance by only training the classifier. We will study this case in future work.

CIFAR-100 [18] is constructed into 10 tasks with disjoint classes, and each task has 10 classes. Each task has 5,000 images for training and 1,000 for testing.

CORe50 [23] is a benchmark designed for class incremental learning with 9 tasks and 50 classes: 10 classes in the first task and 5 classes in the subsequent 8 tasks. Each class has 2,398 training images and 900 testing images on average.

FGVCAircraft [26] contains 102 different classes of aircraft models . 100 classes are selected and divided into 10 tasks. Each class has 33 training images and 33 testing images on average.

DTD [3] is a texture database , organized into 47 classes. 40 classes are selected and divided into 10 tasks. Each class has 40 training images and 40 testing images.

Tiny-ImageNet is a subset of ImageNet with 200 classes for training. Each class has 500 images. The test set contains 10,000 images. The dataset is evenly divided into 10 tasks.

Country211 contains 211 classes of country images, with 150 train and test images per class. The dataset is evenly divided into 10 tasks with 210 classes chosen.

4.2 Evaluation Metrics

We define a i,j as the accuracy evaluated on the test set of task j after training the network from task 1 through to i , and the average accuracy is defined as

When i = m (i.e., the total number of tasks), A m represents the average accuracy by the end of training.

Forgetting rate at task i is defined as Equation 17. f i,j represents how much the model has forgot about task j after being trained on task i . Specifically, max l ∈{ 1 ,...,k -1 } ( a l,j ) denotes the best test accuracy the model has ever achieved on task j before learning task k , and a k,j is the test accuracy on task j after learning task k .

4.3 Implementation Details

ViT-B [5], pre-trained on ImageNet-1K [4], serves as the backbone network for all methods. The data stream is kept identical across all experiments to ensure a fair comparison. The learning rate and batch size are set to 0.001 and 10, respectively. The optimizer is SGD. We assign 5,000 memory buffer sizes for replay-based methods. In F-OAL, the expansion size is 1,000 (i.e., the output activation size to update AC is 1,000). The regularization term is set to be 1. All experiments are conducted on a single RTX 4070ti GPU 12GB, and an average of 3 runs is reported.

4.4 Result Comparison

We tabulate the average accuracy ( A avg ), last task accuracy ( A last ), and the forgetting rate ( F ) from the compared methods in Table 2.

For fine-grained datasets such as FGCVAircraft, DTD, and Country211, F-OAL achieves the best performance, with average accuracies of 66.2%, 82.8%, and 24.4%, respectively. The second-best results are 55.6%, 76.0%, and 17.8%, respectively. Similarly, F-OAL also outperforms in last task accuracy. This demonstrates the excellent transferability of our method in the OCIL setting, effectively leveraging the feature extraction capabilities of the pre-trained encoder.

On coarse-grained datasets such as CIFAR-100, CORe50, and Tiny-ImageNet, F-OAL still demonstrates excellent performance, achieving the highest accuracy among all exemplar-free methods, except on Tiny-ImageNet where it is 0.8% behind EASE in average accuracy. Compared to the best replay-based methods, it only lags by 1.3%, 0.8%, and 0.3%, respectively.

where

Typically, a lower forgetting rate is better, but forgetting is based on accuracy. Therefore, when comparing forgetting rates, it is important to consider models with similar accuracy levels. When comparing F-OAL to the well-performing DVC, F-OAL exhibits a lower forgetting rate. On CIFAR100, F-OAL's forgetting rate is 5.5%, while DVC's is 8.2%. This indicates that F-OAL not only maintains a high level of accuracy but also effectively manages forgetting.

F-OAL

LwF

EWC EASE

LAE

SLCA iCaRL

ER

ASER

PCR

DVC

SCR

0

2

4

6

8

10

Peak GPU Memory Usage (GB)

1.9

3.4

4.1

1.5

0.8

2.6

7.5

6.3

7.8

9.0

9.1

9.8

4.5 Resource Consumption

GPU . Peak GPU memory footprint on CIFAR-100 is shown in Figure 2. As we state in Section 3.5, without using gradient and extra space for exemplars, F-OAL requires a low memory footprint while having good performance (i.e., higher accuracy than LAE and EASE on most datasets).

Training Time . Table 3 illustrates training time including feature extraction. F-OAL is fast with competitive accuracy (i.e., higher accuracy than LAE). Only the classifier and regularized feature autocorrelation matrix are updated, leading to fewer of trainable parameters and fast training speed.

4.6 Countering Recency Bias

As demonstrated in Figure 3, the linear classifier of AC obtained through F-OAL training does not exhibit recency bias. Notably, the weights corresponding to the classes in the most recent tasks do not significantly surpass those of the earlier classes. The frozen encoder and the recursive least square updating manner ensure equal treatment for all samples.

0

5

10

15

20

25

30

35

40

Classes of DTD

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

L2 Norm of Weights

Current Task

Completed Tasks

4.7 Ablation Study

We design the ablation study in Table 4 to verify the effectiveness of the Feature Fusion and Smooth Projection modules. Without these two components, the accuracy of F-OAL drops, especially on fine-grained datasets.

As Table 5 (See appendix B) shows, we prove analytic classifier is key to high accuracy. Herein, we define the Fully Connected Classifier (FCC) as having the identical structure to the AC, but it is updated through back-propagation rather than utilizing the R and recursive least square. Without AC, the accuracy drops from 91.1% to 32.4% on CIFAR-100.

Regularization Term. Table 6 (See appendix B) shows the effects of varying γ . For large volume dataset, F-OAL gives a robust performance in wide range of γ value (e.g., 10 2 -10 -3 ), while small datasets need larger value (e.g., 10 2 -1 ). The comprehensive results show that γ =1 is the suitable choice.

Projection Size . Figure 4 (See appendix B) demonstrates the influence of different random projection sizes. The results suggest that the setting of a 1,000-dimensional projection is appropriate.

4.8 Potential Positive and Negative Societal Impacts

The key advantage of our approach is its ability to achieve exemplar-free OCIL with fast training and low memory footprint, offering an environmentally friendly and efficient solution for this research track. However, the main limitation of our method lies in its reliance on a powerful pre-trained encoder. As a result, it is crucial for us to leverage open-source pre-trained backbone networks from the deep learning community, rather than training our own, which will otherwise lead to higher GPU usage and increased resource consumption.

5 Conclusion

In this paper, we propose Forward-only Online Analytic Learning (F-OAL), an exemplar-free method for addressing several limitations of the Online Class Incremental Learning scenario. We use Analytic Learning to acquire the optimal solution of the classifier directly instead of training for dozens of epochs by back-propagation. Leveraging the frozen pre-trained encoder with Feature Fusion and Smooth Projection and the Analytic Classifier updated by recursive least square, our approach achieves the identical solution to joint-learning on the whole dataset without preserving any historical exemplars, achieving high accuracy and reducing the resource consumption. Our experiments show the competitive performance of F-OAL.

6 Acknowledgment

This research was supported by the National Natural Science Foundation of China (62306117), the Guangzhou Basic and Applied Basic Research Foundation (2024A04J3681, 2023A04J1687), the South China University of Technology-TCL Technology Innovation Fund, the Fundamental Research Funds for the Central Universities (2023ZYGXZR023, 2024ZYGXZR074), the Guangdong Basic and Applied Basic Research Foundation (2024A1515010220), and the CAAI- MindSpore Open Fund developed on Openl Community.

References

Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems , pages 11849-11860, 2019.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems , pages 11816-11825, 2019.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3606-3613, 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16 , pages 86-102, 2020.

Qiankun Gao, Chen Zhao, Bernard Ghanem, and Jian Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. In European Conference on Computer Vision , pages 423-439, 2022.

Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11483-11493, 2023.

Yanan Gu, Xu Yang, Kun Wei, and Cheng Deng. Not just selection, but exploration: Online classincremental continual learning via dual view consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7442-7451, 2022.

Ping Guo, Michael R Lyu, and NE Mastorakis. Pseudoinverse learning algorithm for feedforward neural networks. In Advances in Neural Networks and Applications , pages 321-326, 2001.

Yiduo Guo, Bing Liu, and Dongyan Zhao. Online continual learning through mutual information maximization. In International Conference on Machine Learning , pages 8109-8126. PMLR, 2022.

Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation , pages 9769-9776, 2019.

Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning in online scenario. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1392613935, 2020.

Xu He and Herbert Jaeger. Overcoming catastrophic interference using conceptor-aided backpropagation. 2018.

Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 831-839, 2019.

Zhouyuan Huo, Bin Gu, Heng Huang, et al. Decoupled parallel backpropagation with convergence guarantee. In International Conference on Machine Learning , pages 2098-2106. PMLR, 2018.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. In Proceedings of the National Academy of Sciences , pages 3521-3526, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Byung Hyun Lee, Okchul Jung, Jonghyun Choi, and Se Young Chun. Online continual learning on hierarchical label expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11761-11770, 2023.

Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 2935-2947, 2017.

Huiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li, and Yunming Ye. Pcr: Proxy-based contrastive replay for online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24246-24255, 2023.

Zichen Liu, Chao Du, Wee Sun Lee, and Min Lin. Locality sensitive sparse encoding for learning world models online. In International Conference on Learning Representations , 2024.

Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning , pages 17-26, 2017.

Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing , pages 28-51, 2022.

Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3589-3599, 2021.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. 1989.

Zichong Meng, Jie Zhang, Changdi Yang, Zheng Zhan, Pu Zhao, and Yanzhi WAng. Diffclass: Diffusionbased class incremental learning. European Conference on Computer Vision , 2024.

Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free class-incremental learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3911-3920, 2023.

Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems , pages 12116-12128, 2021.

Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2001-2010, 2017.

Grzegorz Rype´ s´ c, Sebastian Cygert, Valeriya Khan, Tomasz Trzcinski, Bartosz Michał Zieli´ nski, and Bartłomiej Twardowski. Divide and not forget: Ensemble of selectively trained experts in continual learning. In International Conference on Learning Representations , 2024.

Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 9630-9638, 2021.

Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-based continual learning toolbox. arXiv preprint arXiv:2309.07117 , 2023.

Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for class-incremental learning. In European conference on computer vision , pages 398-414, 2022.

Quanziang Wang, Renzhen Wang, Yichen Wu, Xixi Jia, and Deyu Meng. Cba: Improving online continual learning via continual bias adaptor. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19082-19092, 2023.

Zhen Wang, Liu Liu, Yajing Kong, Jiaxian Guo, and Dacheng Tao. Online continual learning with contrastive vision transformer. In European Conference on Computer Vision , pages 631-650, 2022.

Yujie Wei, Jiaxin Ye, Zhizhong Huang, Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18764-18774, 2023.

Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3014-3023, 2021.

Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19148-19158, 2023.

Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jia. A simple but strong baseline for online continual learning: Repeated augmented rehearsal. In Advances in Neural Information Processing Systems , pages 14771-14783, 2022.

Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23554-23564, 2024.

Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and selfsupervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5871-5880, 2021.

Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9296-9305, 2022.

Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9296-9305, 2022.

Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. Ds-al: A dual-stream analytic learning for exemplar-free class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17237-17244, 2024.

Huiping Zhuang, Zhiping Lin, and Kar-Ann Toh. Blockwise recursive moore-penrose inverse for network learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 52(5):3237-3250, 2021.

Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. Gkeal: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7746-7755, 2023.

Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. In Advances in Neural Information Processing Systems , pages 11602-11614, 2022.

A Appendix

Proof of Theorem 3.1 and 3.2

For Theorem 3.1 , we start with proving the case in batch 1 of task k .

According to Equation 8, of task k -1 , we can expand the stacked activation and label matrixes:

Hence, at batch 1 of task k , we have

We have defined regularized feature autocorrelation matrix R k -1 ,n via

To facilitate subsequent calculations, here we also define a cross-correlation matrix Q k -1 ,n

Thus, we can rewrite Equation 20 as

Therefore, at batch 1 of task k we have

From Equation 21 we can recursively calculate R k, 1 from R k -1 ,n

According to the Woodbury matrix identity, we have

Let A = R -1 k -1 ,n , U = X (a) T k, 1 , C = I , V = X (a) k, 1 in Equation 26, we have

Hence, R k can be recursively updated using its last task counterpart R k -1 ,n and current data (i.e., X k, 1 ). This proves the recursive calculation in Equation 11.

Next, we derive the recursive formulation of ˆ W ( k, 1) . To this end, we also recuse the cross-correlation matrix Q k, 1 at the batch 1 of task k :

where

where d (a) is the dimension of activation and d yk is dimension of total classes of task k . Note that the concatenation in Equation 29 is due to fact that Y k, 1 of task k contains more data classes (hence more columns) than Y k -1 ,n

Hence, ˆ W ( k, 1) can be rewritten as

By substituting Equation 27 into R k, 1 Q ′ k -1 ,n , we have

According to Equation 30, Equation 33 can be rewritten as

By inserting Equation 33 into Equation 32, we have

which proves the case in the batch 1 of task k .

For Theorem 3.2 , we consider the case at rest batches of task k . In the rest of batches, the number of classes maintain unchanged compared with batch 1, which means no column expansion is required. According to Equation 35, we substitute ˆ W ( k -1 ,n ) ′ by ˆ W ( k,i -1) . Similar substitution is applied to R k -1 ,n with R k,i -1 according to Equation 27 since the shape of regularized feature autocorrelation matrix is remained through whole learning agenda. Then we have

which complete the proof.

0

2,000

4,000

6,000

8,000

10,000

Projection Size

20

40

60

80

Average Accuracy (%)

1,000

CIFAR-100

DTD

FGVCAircraft

Country211

CORe50

Tiny-ImageNet

B Appendix

Figure 4 demonstrates the influence of different random projection sizes. These experiments reveal that higherdimensional projections may not always improve accuracy, suggesting that projection size must fit the dataset's volume. If the volume of the dataset is small, a high-dimensional projection results in over-fitting to the training set. The maximum accuracy on DTD is at 1,000 projection size, while the accuracy slightly increases after 1,000. When the projection size is 2,000, the results of FGVCAircrft and CORe50 are a little higher than the results on 1,000, but time consumption is twice higher due to the cubic time complexity in the matrix inverse. Thus, we limit the projection size measurement to 2,000 for CORe50 and Tiny-ImageNet due to the significant increase in time consumption.

Table 5 justify the contribution of AC. The reduction from 91.1% to 32.4% without AL suggesting that one epoch of simple back-propagation is unable to handle the OCIL.

Table 6 varifies the choice of 1 as regularization term is suitable for OCIL. Fine-grained datasets suffer a lot from the wrong choice of γ . The average accuracy on FGVCAircraft drops from 61.1% to 22.4% when γ is tuned from 1 to 10 -3 .

AC

FCC

Result

✓

✗

91.1

✗

✓

32.4

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The most important idea of this paper is claimed in substract(i.e., forward-only learning). Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Main limitation is F-OAL needs a powerful pre-trained backbone.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have detailed proof in Apendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The results can be reproduced in released code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have detailed information in Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:We do not include statistical significance.

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have this in Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the rules.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have this content in section 4.8.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We algorithm will not generate harmful content directly.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the source code, which is allowed by code author.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposed a method called F-OAL for online class incremental learning, which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. In summarize, the contributions are as follows:\n1) The paper presents theF-OAL, which is an exemplar-free method.\n2) F-OAL can be updated in an mini-batch manner;\n3) The methods are evaluated in several benchmarks;\n\nSTRENGTHS:\nStrengths:\nThe author proposed a method called F-OAL for online class incremental learning, which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time.\n\nWEAKNESSES:\n1) Some of the descriptions are unclear, such as Formula 4. The author may want to give a more vivid explanation;\n2) The innovation is limited, See Limitations No 5;\n3) The author may want to discuss more methods in evulation part;\n\nQUESTIONS:\nSee Limitations. \nThe author may want to addresses the questions prpopsed in Limitations.\n\nLIMITATIONS:\n1. The formula No.4 seems to be wrong. Please check the formula carefully and give the proof process to ensure that the equation is valid.\n2. Why does the formula No.4 show the optimal solution? For each mini-batch of data, the parameter W is the solution that makes the formula No.5 always equal to 0, in theory. But considering all batches of data, parameter W is too idealized and may even overfit the data of the current batch.\n3. It is difficult for the parameter W calculated using a mini-batch of data to have an effect on other batches of data, especially when the distribution of data for different tasks is significantly different. Even if all the parameters W of the mini-batch are combined, I don’t think it can exceed the back-propagation based method in terms of effect, because this combination is linear.\n4. Is the optimization process recursive on all batches of data? Why is the recursive method more efficient? Less computation? GPU parallel computing? Please explain the reason.\n5. The innovation of this paper is insufficient. It seems that the main contribution is to calculate the parameter W by using the forward process and the least squares method. In fact, this method faces many disadvantages, such as overfitting.\n6. Many models that appeared in the comparative experiments did not have annotated references, such as DVC in Table 1.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper presents an analytic class incremental learning method that does not need backpropogation. The main idea is to use a pre-trained mode to extract features followed by random projection to higher dimensional space, and then use recursive least squares to update the linear regression weights. By doing so, the closed-form solution solves for all seen data thus guarantees no forgetting. The experiments in several class incremental image classification tasks show superior results than many continual learning baselines.

STRENGTHS:
- The empirical results of the method is very strong compared to other continual learning baselines.

WEAKNESSES:
- Starting from section 3.2, it's better to give more precise definition for all notations, such as their dimensionality.
- There are some confusing notations that can be improved. E.g., at line 121 you use $k$ to denote task, but at line 123 you use $k$ to denote batch, then in equation 9 you use change to $n$ to denote batch.
- There are some writing issues like typos in the main, e.g. line 17, line 117, equation 4 (should be $\phi(X)^\top Y$?)

QUESTIONS:
- Does the time recorded in Table 2 for F-OAL include the ViT feature extraction time?

LIMITATIONS:
The limitations of the method are not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of online class incremental learning (OCIL), where new tasks arrive periodically in a data stream and the trainer seeks to learn these new tasks without catastrophic forgetting of past performance. The paper presents two modes of OCIL; replay-based methods and exemplar-free methods. Replay-based methods offer strong performance but require storing some amount of replay data from the stream to include in incremental training. Exemplar-free methods lift this limitation, but have not thus far achieved comparable performance to replay-based methods. The authors method (forward-only online analytic learning, or F-OAL) uses projections of blocks from a frozen, pre-trained ViT encoder as input to a trainable linear classifier, and then perform recursive least-squares updates on the linear classifier to avoid the catastrophic forgetting that backpropagation would cause in the same setting. The authors extensively compare their method by accuracy, training time, and memory usage to other recently-SOTA methods for OCIL, both replay-based and exemplar-free. The comparisons show that F-OAL significantly improves over existing exemplar-free methods and achieves comparable performance with replay-based methods, while being significantly more efficient in terms of both training time and memory usage. Lastly, the authors ablate their solution and find that keeping the ViT encoder frozen and using their analytically-learned classifier (instead of learning it through backpropogation) are both key ingredients in F-OAL's accuracy.\n\nSTRENGTHS:\nThe paper is generally quite strong. The F-OAL method is natural and intuitive; overall, it seems like a significant improvement to the Pareto frontier of accurate and efficient OCIL. The comparisons to past SOTA in both replay-based and exemplar-free methods are extensive and compelling. The authors' exposition is clear and logical, and the experiments are largely informative and useful for the reader. The F-OAL method seems pragmatic, and a natural baseline to which all future work in this area should be compared.\n\nWEAKNESSES:\nThis paper (and past papers in this vein) suggest that exemplar-free methods are \"good for data privacy\", but there is very little justification for this claim. I understand the basic premise as this: if replay-based methods force you to store some subset of the data stream, which is worse for users' data privacy than exemplar-free methods that don't require such storage. While it's worth noting as a design consideration/feature, I do not agree with this framing as relevant for \"data privacy\". There is no legitimate security model of privacy in the literature that would recognize this as \"higher privacy\". Once the model provider has seen/processed the data by running it through the model, any legitimate security model would view this as data that's been made public. Methods that enhance security (e.g. SMPC, HE, TEEs) and methods that provably reduce statistical privacy leakage (i.e. differential privacy) are orthogonal; these methods can be used interchangeably with both replay-based and exemplar-free methods! In a realistic setting, I can see how exemplar-free methods might help assuage compliance concerns or company-specific rules, but I know of no data privacy regulation or cryptographic threat model where this would be a relevant factor. I would suggest that the authors attempt to correct this misconception in the literature by simply stating the feature in terms of its utility: data need not be stored for replay in production. The reader can implicitly understand that this can have several benefits depending on the circumstances of the deployment.\n\nOtherwise, the only criticism I'd have for the paper is its ablation study. While it's a useful sanity check to see their result of ablating AC and FCC, the results with Frozen are obvious and unnecessary. The ViT-B model was developed for ImageNet-sized datasets, of course it will overfit CIFAR-100! I think there are more useful ablations that could be performed (more on that below).\n\nQUESTIONS:\nIt seems clear to me that the authors method of fusing the ViT blocks $B_i(.)$ with random linear projection is valuable, but also not entirely necessary. The goal of this approach appears two-fold; (1) capture information from different levels of abstraction in the representation that their analytic classifier uses, and (2) be able to control the dimensionality of that representation, which will surely need to be tune-able at training time for different datasets (e.g. to avoid overfitting). The ablation in Appendix B reassures the reader that (2) is necessary, but none of the ablations suggest that (1) is necessary. A simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach with a simpler method that applies the smoothed projection to the last ViT block. I'd be curious to know why the authors chose this particular approach. The paper itself states that this feature fusion was implemented to \"further enhance the representativeness of the features\", but is there any work or experiment they can point to that suggests this?\n\nIn any case, this seems to be the only weak point of the paper, and a clarification or improvement would be nice.\n\nLIMITATIONS:\nThe only concern I would have is that the paper (and its cited works) abuse the notion of \"data privacy\", which I have previously argued against in this review. Works that claim to \"improve data privacy\" without treatment of the staggering amount of literature that has gone into defining and proving what is and is not \"private\" in an information-theoretic, statistical, or engineering-focused sense are likely to muddy the waters for those fields. The paper's setting and solution are already worth publishing; in my opinion, the unsubstantiated data privacy claim is hurting more than it's helping.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Foward-only Online Analytic Learning (F-OAL), an exemplar-free approach designed for Online Class Incremental Learning. The method addresses Catastrophic Forgetting by utilizing a pre-trained frozen encoder and a recursive least square updated linear classifier, which significantly reduces memory usage and computational time. The authors conducted extensive experiments to demonstrate the effectiveness of F-OAL on multiple benchmark datasets, showing its superior performance over existing exemplar-free methods and several replay-based methods.\n\nSTRENGTHS:\nThe F-OAL framework introduces a forward-only learning mechanism that avoids back-propagation, effectively reducing computational overhead and memory footprint.\n\nBy not relying on exemplar storage, F-OAL maintains data privacy, a crucial requirement in many real-world applications where data sensitivity is a concern.\n\nWEAKNESSES:\nWhile the paper is strong in many aspects, it lacks a detailed discussion on potential limitations of the proposed method, such as its dependence on the quality of the pre-trained encoder and the challenges that might arise in different data scenarios.\n\nAlthough the paper compares several baseline methods, including more recent and varied techniques could provide a more comprehensive evaluation of F-OAL’s relative performance. Some recent exemplar-free works could be easily generalized to OCIL setting and should be considered include:\n\n[1] Divide and not forget: Ensemble of selectively trained experts in continual learning, ICLR 2024\n\n[2] R-dfcil: Relation-guided representation learning for data-free class incremental learning, ECCV 2022\n\n[3] Self-sustaining representation expansion for non-exemplar class-incremental learning, CVPR 2022\n\n[4] DiffClass: Diffusion-Based Class Incremental Learning, ECCV 2024\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses. I think providing a more comprehensive comparison with recent state-of-the-art works, a complexity analysis and an overhead comparison would help justify the contribution of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-21"}