F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning

Authors: huiping  zhuang
yuchen  liu
run  he
kai  tong
ziqian  zeng
cen  chen
yi  wang
lap-pui  chau
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:58:31

Abstract: Online Class Incremental Learning (OCIL) aims to train models incrementally, where data arrive in mini-batches, and previous data are not accessible. A major challenge in OCIL is Catastrophic Forgetting, i.e., the loss of previously learned knowledge. Among existing baselines, replay-based methods show competitive results but requires extra memory for storing exemplars, while exemplar-free (i.e., data need not be stored for replay in production) methods are resource friendly but often lack accuracy. In this paper, we propose an exemplar-free approach—Forward-only Online Analytic Learning (F-OAL). Unlike traditional methods, F-OAL does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. Cooperating with a pre-trained frozen encoder with Feature Fusion, F-OAL only needs to update a linear classifier by recursive least square. This approach simultaneously achieves high accuracy and low resource consumption. Extensive experiments on bench mark datasets demonstrate F-OAL’s robust performance in OCIL scenarios. Code is available at: https://github.com/liuyuchen-cz/F-OAL

Main paper: 
 introduction: 
 Class Incremental Learning (CIL) updates the model incrementally in a task-by-task manner with new classes in the new task. Traditional CIL most plans for static offline datasets which historical data are accessible. However, with the rapid increase of social media and mobile devices, massive amount of image data have been produced online in a streaming fashion, and render training models on static data less applicable. To address this, Online Class Incremental Learning (OCIL) is developed taking an online constraint in addition to the existing CIL setting. OCIL is a more challenging CIL setting in which data come in mini-batches, and the model is trained only in one epoch (i.e., learning from one pass data stream) [14] . The model is required to achieve high accuracy, fast training time, and low resource consumption [25] .However, CIL techniques (including OCIL) suffer from Catastrophic Forgetting (CF) [27] , also known as the erosion of previous knowledge when new data are introduced. The problem becomes more severe in online scenarios since the model can only see data once. Two major factors contribute to CF: (1) Using the loss function to update the whole network leads to uncompleted feature capturing and diminished global representation [11] . (2) Using back-propagation to adjust linear classifier results in recency bias, which is a significantly imbalanced weight distribution, showing preference only on current learning data [15] .To address CF in an online setting, replay-based methods [9, 21] are the mainstream solution by preserving old exemplars and revisiting them in new tasks. This strategy has strong performance but is resource consuming, while exemplar-free methods [17, 20] have lower resource consumption but show less competitive results.Recently, Analytic Continual Learning (ACL) [49] methods emerged as an exemplar-free branch, delivering encouraging outcomes. ACL methods pinpoint the iterative back-propagation as the main factor behind catastrophic forgetting and seek to address it through linear recursive strategies. Remarkably, for the first time, these methods achieve outcomes comparable to those utilizing replaybased techniques.There are two limitations in existing ACL methods: (1) Multiple iterations of base training are needed when the model is applied. Subsequently, the acquired knowledge is encoded into a regularized feature autocorrelation matrix by analytic re-alignment. The incremental learning phase then unfolds, utilizing the recursive least squares method for updates. This pattern is repeated when the dataset is switched, significantly elevating the temporal cost in an online scenario. (2) Classic ACL methods demand data aggregation from a single task, facilitating analytic learning in one fell swoop. This process increases GPU memory footprint and is unsuitable for online contexts where data for each task is presented as mini-batches.To address those limitations, we propose Forward-only Online Analytic Learning (F-OAL) that learns online batch-wise data streams. The F-OAL consists of a frozen pre-trained encoder and an Analytic Classifier (AC). The frozen encoder is capable of countering the uncompleted feature representation caused by using the loss function to update and replace the time-consuming base training. With Feature Fusion and Smooth Projection, the encoder provide informative representation for analytic learning. The AC is updated by recursive least square rather than back-propagation to solve recency bias and decrease calculation. Therefore, F-OAL is an exemplar-free countermeasure to CF and reduces resource consumption since the encoder is frozen and only the AC is updated.Our main contributions can be concluded as follows:• We present the F-OAL, an exemplar-free technique that achieves high accuracy and low resource consumption together for the OCIL.• F-OAL redefines the OCIL problem into a recursive least square manner and is updated in a mini-batch manner.• F-OAL introduces a framework of frozen pre-trained encoder with Feature Fusion to generate representative features and Smooth Projection for recursively updating AC to counter CF.• We conduct massive experiments on benchmark datasets with other OCIL baselines. The results demonstrate that F-OAL achieves competitive results with fasting training speed and low GPU footprint. 
related works: 
 Online Class Incremental Learning focuses on extracting knowledge from one pass data stream with new classes in the new task. Time and memory consumption requirements are particularly critical in OCIL, given the fast and large nature of online data streams [13] .Replay-based methods [33, 25, 21, 9, 31, 12, 2, 11, 15, 41, 37, 36, 1, 38, 19, 6, 39, 35, 9] are mainstream solutions for OCIL problems by preserving historical exemplars and using them in new tasks. The accuracy is better than exemplar-free methods', but the training time and memory consumption are higher.Analytic Learning (AL), also referred to as pseudo-inverse learning [10] , emerges as a solution to the pitfalls of back-propagation by recursive least square. Analytic Learning's computational intensity is demanding since the entire dataset is processed. The obstacle is solved by the block-wise recursive In the encoder , features from each block of the ViT are extracted, summed, and averaged to form a composite feature. This feature is then randomly projected into a higher-dimensional space and normalized using the sigmoid function, serving as the activation for updating the classifier. All parameters in the encoder remain frozen. In the analytic classifier section, we introduce R to retain historical information and update the linear classifier using recursive least squares. This process is forward-only with no gradients.R k-1,n • • • • • • R 1,n R 0 R k,n Identity Matrix Initialization Task #1 Task #k-1 Task #k Task k Online Data Stream (b) F-OAL learning agenda of task k • • • • • • Information Storing • • • • • • • • • • • • Mini-batches Mini-batchesMoore-Penrose algorithm [47] , which achieves equivalent precision with joint-learning (i.e., training with all data). AL has already been used in online reinforcement learning [22] , which shows the potential of AL in OCIL.Exemplar-free methods can be categorized into regularization-based, prototype-based, and recently emerged ACL methods. Regularization-based methods [20, 17, 42, 8, 44] apply constraints on the loss function or change the gradients to preserve knowledge. These solutions reduce resource consumption but commonly can not outperform replay-based methods. Prototype-based methods [43, 45, 29, 40, 32, 7, 28] mitigate CF by augmenting the prototypes of features to classify old classes or generating psudo-features of old classes from new representations. ACL methods represent a new branch of the CIL community and show great potential in the OCIL scenario. Analytic Class Incremental Learning (ACIL) [49] is the first approach that applies AL to the CIL problem. The ACIL achieves a competitive performance in the offline CIL scenario. Gaussian Kernel Embedded Analytic Learning (GKEAL) [48] , following the ACIL, focuses on solving CF in the few-shot scenario by adopting the kernel method. DS-AL [46] overcomes the under-fitting problem of AL.3 Methodology 
proposed framework: 
 Our F-OAL framework consists of two modules:Encoder. Encoder has two parts: backbone with Feature Fusion and Smooth Projection. The performance of analytic learning is highly dependent on the quality of the extracted features. Therefore, we employ Vision Transformer (ViT) [5] as the backbone instead of CNN, because ViT generates more comprehensive feature representation [30] . To further enhance the representativeness of the features, we utilize Feature Fusion. Specifically, we extract the outputs from each block of the ViT, sum them, and take their average to form the image feature.Subsequently, we expand this feature into a higher-dimensional space using random projection and apply the sigmoid function for smoothing (i.e., Smooth Projection). This results in the final activation used to update the classifier. Therefore, the encoder ϕ(•) is defined as:EQUATIONwhereB 1 (•) • • • B n (•) are blocks in ViT, LP(•) is linear projection and σ(•) is sigmoid activation function.Backbone is pre-trained and frozen and weight matrix of projection is sampled from normal distribution and is frozen. Freezing the encoder avoids selective learning caused by loss function, which prioritizes features that are easiest to learn rather than the most representative [11] and significantly reduces the number of trainable parameters.Analytic Classifier. Unlike back-propagation, we employ the least squares method to obtain the analytic solution for the linear mapping from the activation to the one-hot label. We then recursively update the weight matrix of the linear mapping. This approach is forward-only and does not require the use of gradients, resulting in low memory usage and fast computation speed.Name Description Dimensionϕ(X) Activation of all images. V × D Y One-hot label of all images. V × M Ŵ Joint-learning result of classifier's weight matrix. M × D X (a) k,iActivation matrix of the i-th batch of the k-th task.S × D Y train k,i One-hot label matrix of the i-th batch of the k-th task. S × C s X (a) k,1:i Activation matrix from the start to the i-th batch of the k-th task.V s × D Y train k,1:i One-hot label matrix from the start.to the i-th batch of the k-th task.V s × C s Ŵ (k,i) Classifier of the i-th batch of the k-th task.C s × D R k,iRegularized feature autocorrelation matrix to the n-th batch of the k-th task. D × D Table 1 : Description of notations and their dimensions. Here, V is the number of all images, D is the encoder output dimension, M is the number of all classes, S is the batch size, C s is the number of classes seen so far, and V s is the number of images seen so far.The overview is shown in Figure 1 . To derive our solution, the notations needed are listed in Table 1 . Unlike other methods that treat model training as a back-propagation process, our method formulates the problem using linear regression, which allows for direct computation of the optimal parameters in a closed-form solution:EQUATIONwhere ϕ(•) is the pre-trained and forzen encoder and W is the learnable parameter of linear classifier.The learning problem can be rewritten into an optimization form:EQUATIONwhere ∥ • ∥ F represents the Frobenius norm and γ is the regularization term. The optimal solution is defined as:EQUATIONGiven dataset {X train k , Y train k } m 1 be the training set of task k (k = 1, 2, ... , m). Every task is divided into n mini-batches. We denote {X train k,i , Y train k,i }, as the i mini-batch (i=1, 2, ..., n) of training set of task k. At task 1 batch k, the model aims to optimize argminEQUATIONwhereEQUATIONThe optimal solution to parameter W (1,i) is given as:EQUATIONRegardingX (a)1,1:i and Y train 1,1:i as stacks of row activation vectors and one-hot label vectors respectively, we observe that (X(a) 1,1:i ) T (X (a) 1,1:i ) and (X (a) 1,1:i ) T Y train1,1:i are both sums of outer products w.r.t feature vectors. Notice that the solution contains no data correlation. Thus we can further derive Equation 7intoEQUATIONLetEQUATIONbe the regularized feature autocorrelation matrix at batch i-1 of task 1, where both historical and current information is encoded in.Therefore, we can calculate both R and W in a recursive manner by the following theorems when the serial numbers of tasks and batches are given:Theorem 3.1 For the batch 1 of task k, Let Ŵ(0) be the null matrix. Let Ŵ(k-1,n) ′ = Ŵ(k-1,n) 0 and Ŵ(k,1)can be calculated via:EQUATIONwhereEQUATIONTheorem 3.2 For the batch i ( i > 1 ) of task k, Ŵ(k,i) can be calculated via:EQUATIONwhereEQUATIONProof. See Appendix A.Thus, we achieve absolute memorization in an exemplar-free way with all data used only once. The learning agenda of F-OAL is summarised in Algorithm 1. 1) and R k,i-1 ; end if end for end forInput: Mini-batches {X train k,i , Y train k,i } m,n 1,1 . Initialization: Identity matrix R 0 ; Null matrix W (0) . for k = 1 to m do for i = 1 to n do if i = 1 then # Theorem 3.1: Load Ŵ(k-1,n) and R k-1,n ; Expand Ŵ(k-1,n) to Ŵ(k-1,n) ′ ; Obtain activation X (a) k,1 based on X train k,1 ; Update R k,1 by X (a) k,1 and R k-1,n ; Update W (k,1) by X (a) k,1 ; Y train k,1 , W (k-1,n) ′ and R k ; else # Theorem 3.2: Load W (k,i-1) and R k,i-1 ; Obtain activation X (a) k,i based on X train k,i ; Update R k,i by X (a) k,i and R k,i-1 ; Update W (k,i) by X (a) k,i ; Y train k,i , W (k,i-In terms of space complexity, our trainable parameters are only R and W. The R matrix is of size D × D, where D is the output dimension of the encoder. In our paper, the encoder output dimension is 1000. Therefore, according to Equation 9, the size of the R matrix is 1, 000 × 1, 000. The W matrix has dimensions of C × D, where C is the number of classes in the target dataset. For example, with CIFAR-100, its size is 100 × 1, 000. The total number of trainable parameters is relatively small and does not require gradients. This results in our method using less than 2GB of GPU memory.In terms of computational complexity, we denote the batch size as S, encoder's output size as D, and class number as C. Therefore, the dimensions of X, Y, R and W are S × D, S × C, D × D, and C × D, respectively. Thus, the calculation is shown below:The computational complexity for updating R is dominated by the matrix multiplications, thus:EQUATIONThe computational complexity for updating W is dominated by the matrix multiplications and the matrix inversion:EQUATION)In the OCIL setting, the batch size is relatively smaller. Therefore, the overall computational complexity is primarily controlled by D.In terms of space overhead, compared to the conventional backbone + classifier structure, F-OAL introduces an additional linear projection to control the output dimension D of the encoder, and a matrix R, where only R is trainable. According to Equation 9, the dimension of R remains a fixed size of D × D. Other methods require more extra space. For instance, LwF [20] employs knowledge distillation, necessitating the storage of additional models, while replay-based methods require extra storage to retain historical samples. In contrast, the overhead introduced by F-OAL, consisting of an additional matrix and a linear layer, is smaller. In terms of time overhead, our method primarily consists of a forward pass and matrix multiplication, which is determined by the output dimension of the encoder. By changing the output dimension of the encoder, we can balance the accuracy and time. According to [16] , the backward pass in back-propagation (forward pass + backward pass) accounts for 70% of the time. Therefore, our method's time overhead is also relatively small. 
experiment: 
 To show the effectiveness of F-OAL, we conduct extensive experiments to compare our approach with baseline methods. We build our code and reproduce relevant results based on [24, 34] . All baselines are with pre-trained ViT as the backbone.We focus on coarse-grained data scenarios, such as CIFAR-100, Tiny-ImageNet, and Core50, as well as fine-grained data scenarios, including DTD, FGVCAircraft, and Country211. All of these are open-source benchmark datasets. However, there are other data scenarios, such as long-tail distributions, where unbalanced data distribution will make it harder to achieve good performance by only training the classifier. We will study this case in future work.• CIFAR-100 [18] is constructed into 10 tasks with disjoint classes, and each task has 10 classes. Each task has 5,000 images for training and 1,000 for testing.• CORe50 [23] is a benchmark designed for class incremental learning with 9 tasks and 50 classes: 10 classes in the first task and 5 classes in the subsequent 8 tasks. Each class has 2,398 training images and 900 testing images on average.• FGVCAircraft [26] The test set contains 10,000 images. The dataset is evenly divided into 10 tasks.• Country211 contains 211 classes of country images, with 150 train and test images per class. The dataset is evenly divided into 10 tasks with 210 classes chosen.We define a i,j as the accuracy evaluated on the test set of task j after training the network from task 1 through to i, and the average accuracy is defined asEQUATIONWhen i = m (i.e., the total number of tasks), A m represents the average accuracy by the end of training.Forgetting rate at task i is defined as Equation 17. f i,j represents how much the model has forgot about task j after being trained on task i. Specifically, max l∈{1,...,k-1}(a l,j ) denotes the best test accuracy the model has ever achieved on task j before learning task k, and a k,j is the test accuracy on task j after learning task k.EQUATIONwhereEQUATION)ViT-B [5] , pre-trained on ImageNet-1K [4] , serves as the backbone network for all methods. The data stream is kept identical across all experiments to ensure a fair comparison. The learning rate and batch size are set to 0.001 and 10, respectively. The optimizer is SGD. We assign 5,000 memory buffer sizes for replay-based methods. In F-OAL, the expansion size is 1,000 (i.e., the output activation size to update AC is 1,000). The regularization term is set to be 1. All experiments are conducted on a single RTX 4070ti GPU 12GB, and an average of 3 runs is reported.We tabulate the average accuracy (A avg ), last task accuracy (A last ), and the forgetting rate (F ) from the compared methods in Table 2 .For fine-grained datasets such as FGCVAircraft, DTD, and Country211, F-OAL achieves the best performance, with average accuracies of 66.2%, 82.8%, and 24.4%, respectively. The second-best results are 55.6%, 76.0%, and 17.8%, respectively. Similarly, F-OAL also outperforms in last task accuracy. This demonstrates the excellent transferability of our method in the OCIL setting, effectively leveraging the feature extraction capabilities of the pre-trained encoder.On coarse-grained datasets such as CIFAR-100, CORe50, and Tiny-ImageNet, F-OAL still demonstrates excellent performance, achieving the highest accuracy among all exemplar-free methods, except on Tiny-ImageNet where it is 0.8% behind EASE in average accuracy. Typically, a lower forgetting rate is better, but forgetting is based on accuracy. Therefore, when comparing forgetting rates, it is important to consider models with similar accuracy levels. When comparing F-OAL to the well-performing DVC, F-OAL exhibits a lower forgetting rate. On CIFAR-100, F-OAL's forgetting rate is 5.5%, while DVC's is 8.2%. This indicates that F-OAL not only maintains a high level of accuracy but also effectively manages forgetting. Based on [15] , when recency bias happens, L2 norm of current task is significantly larger. In F-OAL, the L2 norm of current task is in a average level.GPU. Peak GPU memory footprint on CIFAR-100 is shown in Figure 2 . As we state in Section 3.5, without using gradient and extra space for exemplars, F-OAL requires a low memory footprint while having good performance (i.e., higher accuracy than LAE and EASE on most datasets).Training Time. Table 3 illustrates training time including feature extraction. F-OAL is fast with competitive accuracy (i.e., higher accuracy than LAE). Only the classifier and regularized feature autocorrelation matrix are updated, leading to fewer of trainable parameters and fast training speed.As demonstrated in Figure 3 , the linear classifier of AC obtained through F-OAL training does not exhibit recency bias. Notably, the weights corresponding to the classes in the most recent tasks do not significantly surpass those of the earlier classes. The frozen encoder and the recursive least square updating manner ensure equal treatment for all samples.We design the ablation study in As Table 5 (See appendix B) shows, we prove analytic classifier is key to high accuracy. Herein, we define the Fully Connected Classifier (FCC) as having the identical structure to the AC, but it is updated through back-propagation rather than utilizing the R and recursive least square. Without AC, the accuracy drops from 91.1% to 32.4% on CIFAR-100.Regularization Term. Table 6 (See appendix B) shows the effects of varying γ. For large volume dataset, F-OAL gives a robust performance in wide range of γ value (e.g., 10 2 -10 -3 ), while small datasets need larger value (e.g., 10 2 -1). The comprehensive results show that γ=1 is the suitable choice.Projection Size. Figure 4 (See appendix B) demonstrates the influence of different random projection sizes. The results suggest that the setting of a 1,000-dimensional projection is appropriate.The key advantage of our approach is its ability to achieve exemplar-free OCIL with fast training and low memory footprint, offering an environmentally friendly and efficient solution for this research track. However, the main limitation of our method lies in its reliance on a powerful pre-trained encoder. As a result, it is crucial for us to leverage open-source pre-trained backbone networks from the deep learning community, rather than training our own, which will otherwise lead to higher GPU usage and increased resource consumption.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The results can be reproduced in released code.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this paper, we propose Forward-only Online Analytic Learning (F-OAL), an exemplar-free method for addressing several limitations of the Online Class Incremental Learning scenario. We use Analytic Learning to acquire the optimal solution of the classifier directly instead of training for dozens of epochs by back-propagation. Leveraging the frozen pre-trained encoder with Feature Fusion and Smooth Projection and the Analytic Classifier updated by recursive least square, our approach achieves the identical solution to joint-learning on the whole dataset without preserving any historical exemplars, achieving high accuracy and reducing the resource consumption. Our experiments show the competitive performance of F-OAL.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We release the code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
acknowledgment: 
 This research was supported by the National Natural Science Foundation of China (62306117), the Guangzhou Basic and Applied Basic Research Foundation (2024A04J3681, 2023A04J1687), the South China University of Technology-TCL Technology Innovation Fund, the Fundamental Research Funds for the Central Universities (2023ZYGXZR023, 2024ZYGXZR074), the Guangdong Basic and Applied Basic Research Foundation (2024A1515010220), and the CAAI-MindSpore Open Fund developed on Openl Community.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have detailed information in Experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification:We do not include statistical significance.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have this in Experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Proof of Theorem 3.1 and 3.2For Theorem 3.1, we start with proving the case in batch 1 of task k.According to Equation 8, of task k -1, we can expand the stacked activation and label matrixes:Ŵ(k-1,n) = k-1 m=1 n i=1 X (a)T m,i X (a) m,i + γI -1    X (a)T 1,1 Y train 1,1. . .EQUATIONHence, at batch 1 of task k, we haveŴ(k,1) = k-1 m=1 n i=1 X (a)T m,i X (a) m,i + γI + X (a)T k,1 X (a) k,1 -1       X (a)T 1,1 Y train 1,1. . .EQUATION)We have defined regularized feature autocorrelation matrixEQUATIONTo facilitate subsequent calculations, here we also define a cross-correlation matrixEQUATIONThus, we can rewrite Equation 20asEQUATIONTherefore, at batch 1 of task k we haveEQUATIONFrom Equation 21we can recursively calculate R k,EQUATIONAccording to the Woodbury matrix identity, we haveEQUATIONLetA = R -1 k-1,n ,U = X (a)T k,1 , C = I , V = X (a)k,1 in Equation 26, we haveEQUATIONHence, R k can be recursively updated using its last task counterpart R k-1,n and current data (i.e., X k,1 ). This proves the recursive calculation in Equation 11.Next, we derive the recursive formulation ofŴ (k,1). To this end, we also recuse the cross-correlation matrix Q k,1 at the batch 1 of task k:EQUATIONwhereEQUATIONwhere d(a) is the dimension of activation and d yk is dimension of total classes of task k. Note that the concatenation in Equation 29 is due to fact that Y k,1 of task k contains more data classes (hence more columns) thanEQUATIONAsEQUATIONHence, Ŵ (k,1) can be rewritten asEQUATIONBy substituting EquationEQUATIONAccording to Equation 30, Equation 33 can be rewritten asR k,1 Q ′ k-1,n = Ŵ (k-1,n) ′ -R k,1 X (a)T k,1 X (a)k,1EQUATION)By inserting Equation 33 into Equation 32, we haveEQUATIONwhich proves the case in the batch 1 of task k.For Theorem 3.2, we consider the case at rest batches of task k. In the rest of batches, the number of classes maintain unchanged compared with batch 1, which means no column expansion is required. According to Equation 35, we substituteŴ (k-1,n) ′ by Ŵ (k,i-1). Similar substitution is applied to R k-1,n with R k,i-1 according to Equation 27 since the shape of regularized feature autocorrelation matrix is remained through whole learning agenda. Then we haveEQUATIONEQUATIONwhich complete the proof. 11, higher projection sizes escalate training time while increase little performance. On small datasets, higher projection sizes may even lead to reduction.Figure 4 demonstrates the influence of different random projection sizes. These experiments reveal that higherdimensional projections may not always improve accuracy, suggesting that projection size must fit the dataset's volume. If the volume of the dataset is small, a high-dimensional projection results in over-fitting to the training set. The maximum accuracy on DTD is at 1,000 projection size, while the accuracy slightly increases after 1,000.When the projection size is 2,000, the results of FGVCAircrft and CORe50 are a little higher than the results on 1,000, but time consumption is twice higher due to the cubic time complexity in the matrix inverse. Thus, we limit the projection size measurement to 2,000 for CORe50 and Tiny-ImageNet due to the significant increase in time consumption. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: The reviewers were generally appreciative about the responses that authors gave to their concerns, and appreciated both the empirical results and computational efficiency of the proposed method. The authors are strongly encouraged to incorporate additional experiments and explanations they included in their rebuttal. These are, in particular:


* The extensive additional experiments against competitors included in the pdf of the rebuttal.
* The ablation w.r.t. capturing information from different levels of abstraction. 
* The discussion on both computational and space complexity, as well as the comparison of overheads.
* A table with notations.
* A clear statement that evaluations include the feature extraction time.
* A discussion of limitations.
* A discussion on long-tailed distributions.
* A thorough correction of all typos indicated by reviewers, and a careful proofreading.

Reviews: review: paper_summary: paper_summary: The paper introduces Foward-only Online Analytic Learning (F-OAL), an exemplar-free approach designed for Online Class Incremental Learning. The method addresses Catastrophic Forgetting by utilizing a pre-trained frozen encoder and a recursive least square updated linear classifier, which significantly reduces memory usage and computational time. The authors conducted extensive experiments to demonstrate the effectiveness of F-OAL on multiple benchmark datasets, showing its superior performance over existing exemplar-free methods and several replay-based methods.
strength_weakness: strength_weakness: While the paper is strong in many aspects, it lacks a detailed discussion on potential limitations of the proposed method, such as its dependence on the quality of the pre-trained encoder and the challenges that might arise in different data scenarios.

Although the paper compares several baseline methods, including more recent and varied techniques could provide a more comprehensive evaluation of F-OAL’s relative performance. Some recent exemplar-free works could be easily generalized to OCIL setting and should be considered include:

[1] Divide and not forget: Ensemble of selectively trained experts in continual learning, ICLR 2024

[2] R-dfcil: Relation-guided representation learning for data-free class incremental learning, ECCV 2022

[3] Self-sustaining representation expansion for non-exemplar class-incremental learning, CVPR 2022

[4] DiffClass: Diffusion-Based Class Incremental Learning, ECCV 2024
questions: questions: Please refer to the weaknesses.
limitations: limitations: Please refer to the weaknesses. I think providing a more comprehensive comparison with recent state-of-the-art works, a complexity analysis and an overhead comparison would help justify the contribution of this work.
paper_summary: paper_summary: The authors address the problem of online class incremental learning (OCIL), where new tasks arrive periodically in a data stream and the trainer seeks to learn these new tasks without catastrophic forgetting of past performance. The paper presents two modes of OCIL; replay-based methods and exemplar-free methods. Replay-based methods offer strong performance but require storing some amount of replay data from the stream to include in incremental training. Exemplar-free methods lift this limitation, but have not thus far achieved comparable performance to replay-based methods. The authors method (forward-only online analytic learning, or F-OAL) uses projections of blocks from a frozen, pre-trained ViT encoder as input to a trainable linear classifier, and then perform recursive least-squares updates on the linear classifier to avoid the catastrophic forgetting that backpropagation would cause in the same setting. The authors extensively compare their method by accuracy, training time, and memory usage to other recently-SOTA methods for OCIL, both replay-based and exemplar-free. The comparisons show that F-OAL significantly improves over existing exemplar-free methods and achieves comparable performance with replay-based methods, while being significantly more efficient in terms of both training time and memory usage. Lastly, the authors ablate their solution and find that keeping the ViT encoder frozen and using their analytically-learned classifier (instead of learning it through backpropogation) are both key ingredients in F-OAL's accuracy.
strength_weakness: strength_weakness: This paper (and past papers in this vein) suggest that exemplar-free methods are "good for data privacy", but there is very little justification for this claim. I understand the basic premise as this: if replay-based methods force you to store some subset of the data stream, which is worse for users' data privacy than exemplar-free methods that don't require such storage. While it's worth noting as a design consideration/feature, I do not agree with this framing as relevant for "data privacy". There is no legitimate security model of privacy in the literature that would recognize this as "higher privacy". Once the model provider has seen/processed the data by running it through the model, any legitimate security model would view this as data that's been made public. Methods that enhance security (e.g. SMPC, HE, TEEs) and methods that provably reduce statistical privacy leakage (i.e. differential privacy) are orthogonal; these methods can be used interchangeably with both replay-based and exemplar-free methods! In a realistic setting, I can see how exemplar-free methods might help assuage compliance concerns or company-specific rules, but I know of no data privacy regulation or cryptographic threat model where this would be a relevant factor. I would suggest that the authors attempt to correct this misconception in the literature by simply stating the feature in terms of its utility: data need not be stored for replay in production. The reader can implicitly understand that this can have several benefits depending on the circumstances of the deployment.

Otherwise, the only criticism I'd have for the paper is its ablation study. While it's a useful sanity check to see their result of ablating AC and FCC, the results with Frozen are obvious and unnecessary. The ViT-B model was developed for ImageNet-sized datasets, of course it will overfit CIFAR-100! I think there are more useful ablations that could be performed (more on that below).
questions: questions: It seems clear to me that the authors method of fusing the ViT blocks $B_i(.)$ with random linear projection is valuable, but also not entirely necessary. The goal of this approach appears two-fold; (1) capture information from different levels of abstraction in the representation that their analytic classifier uses, and (2) be able to control the dimensionality of that representation, which will surely need to be tune-able at training time for different datasets (e.g. to avoid overfitting). The ablation in Appendix B reassures the reader that (2) is necessary, but none of the ablations suggest that (1) is necessary. A simple ablation that could've helped would be to compare their block-averaging + smoothed projection approach with a simpler method that applies the smoothed projection to the last ViT block. I'd be curious to know why the authors chose this particular approach. The paper itself states that this feature fusion was implemented to "further enhance the representativeness of the features", but is there any work or experiment they can point to that suggests this?

In any case, this seems to be the only weak point of the paper, and a clarification or improvement would be nice.
limitations: limitations: The only concern I would have is that the paper (and its cited works) abuse the notion of "data privacy", which I have previously argued against in this review. Works that claim to "improve data privacy" without treatment of the staggering amount of literature that has gone into defining and proving what is and is not "private" in an information-theoretic, statistical, or engineering-focused sense are likely to muddy the waters for those fields. The paper's setting and solution are already worth publishing; in my opinion, the unsubstantiated data privacy claim is hurting more than it's helping.
paper_summary: paper_summary: This paper presents an analytic class incremental learning method that does not need backpropogation. The main idea is to use a pre-trained mode to extract features followed by random projection to higher dimensional space, and then use recursive least squares to update the linear regression weights. By doing so, the closed-form solution solves for all seen data thus guarantees no forgetting. The experiments in several class incremental image classification tasks show superior results than many continual learning baselines.
strength_weakness: strength_weakness: - Starting from section 3.2, it's better to give more precise definition for all notations, such as their dimensionality.
- There are some confusing notations that can be improved. E.g., at line 121 you use $k$ to denote task, but at line 123 you use $k$ to denote batch, then in equation 9 you use change to $n$ to denote batch.
- There are some writing issues like typos in the main, e.g. line 17, line 117, equation 4 (should be $\phi(X)^\top Y$?)
questions: questions: - Does the time recorded in Table 2 for F-OAL include the ViT feature extraction time?
limitations: limitations: The limitations of the method are not discussed.
paper_summary: paper_summary: The author proposed a method called F-OAL for online class incremental learning, which does not rely on back-propagation and is forward-only, significantly reducing memory usage and computational time. In summarize, the contributions are as follows:
1) The paper presents theF-OAL, which is an exemplar-free method.
2) F-OAL can be updated in an mini-batch manner;
3) The methods are evaluated in several benchmarks;
strength_weakness: strength_weakness: 1) Some of the descriptions are unclear, such as Formula 4. The author may want to give a more vivid explanation;
2) The innovation is limited, See Limitations No 5;
3) The author may want to discuss more methods in evulation part;
questions: questions: See Limitations. 
The author may want to addresses the questions prpopsed in Limitations.
limitations: limitations: 1. The formula No.4 seems to be wrong. Please check the formula carefully and give the proof process to ensure that the equation is valid.
2. Why does the formula No.4 show the optimal solution? For each mini-batch of data, the parameter W is the solution that makes the formula No.5 always equal to 0, in theory. But considering all batches of data, parameter W is too idealized and may even overfit the data of the current batch.
3. It is difficult for the parameter W calculated using a mini-batch of data to have an effect on other batches of data, especially when the distribution of data for different tasks is significantly different. Even if all the parameters W of the mini-batch are combined, I don’t think it can exceed the back-propagation based method in terms of effect, because this combination is linear.
4. Is the optimization process recursive on all batches of data? Why is the recursive method more efficient? Less computation? GPU parallel computing? Please explain the reason.
5. The innovation of this paper is insufficient. It seems that the main contribution is to calculate the parameter W by using the forward process and the least squares method. In fact, this method faces many disadvantages, such as overfitting.
6. Many models that appeared in the comparative experiments did not have annotated references, such as DVC in Table 1.

Comments: title: Thank you for the response
Thank you for the response
Response to authors
Thank you for the response
comment: Thank you for taking the time to read our respose. Could you kindly adjust our score if our response addresses your cocerns, before the discussion closes?  Thanks!
Thank you to the authors for their detailed response. I have also checked the feedback from other reviewers and will adjust my score accordingly.
Thank you for taking the time to read our response! We are glad that the response addressed your concerns.
Thank you for acknowledging and updating based on my remarks, these improvements are satisfactory in my view. However, I misunderstood the scoring procedure here. My score was contingent on those remarks being addressed, so it's unlikely to increase. I'm interested in the ongoing conversation with Reviewer UyXG as well, I hope that will be resolved before the closed discussion.
Thank you for taking the time to read our response! This means a lot to us! Please let us know if there is anything more needed from us!
I appreciate the authors' rebuttal. I still have minor concerns on the paper's presentation, as well as the limitation that it relies on a strong pretrained encoder. I will consider adjusting my score during the closed discussion. Thanks!