GACL: Exemplar-Free Generalized Analytic Continual Learning

Authors: huiping  zhuang
yizhu  chen
di  fang
run  he
kai  tong
hongxin  wei
ziqian  zeng
† cen chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 09:55:39

Abstract: Class incremental learning (CIL) trains a network on sequential tasks with separated categories in each task but suffers from catastrophic forgetting, where models quickly lose previously learned knowledge when acquiring new tasks. The generalized CIL (GCIL) aims to address the CIL problem in a more real-world scenario, where incoming data have mixed data categories and unknown sample size distribution. Existing attempts for the GCIL either have poor performance or invade data privacy by saving exemplars. In this paper, we propose a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). The GACL adopts analytic learning (a gradient-free training technique) and delivers an analytical  (i.e., closed-form) solution to the GCIL scenario. This solution is derived via decomposing the incoming data into exposed and unexposed classes, thereby attaining a weight-invariant property, a rare yet valuable property supporting an equivalence between incremental learning and its joint training. Such an equivalence is crucial in GCIL settings as data distributions among different tasks no longer pose challenges to adopting our GACL. Theoretically, this equivalence property is validated through matrix analysis tools. Empirically, we conduct extensive experiments where, compared with existing GCIL methods, our GACL exhibits a consistently leading performance across various datasets and GCIL settings. Source code is available at https://github.com/CHEN-YIZHU/GACL.

Main paper: 
 introduction: 
 Class incremental learning (CIL) [1] , an important form of continual learning, aims to effectively tune an off-the-shelf network on incoming new datasets, with data excluding various categories from its previous states. The CIL has gained significant traction due to its ability to refine learned models for new and unfamiliar data classes, eliminating the need to start the training process from scratch. This elimination of retraining saves valuable computational resources, which is especially important in the era of pre-trained models that have absorbed a massive amount of data. One significant challenge in CIL is catastrophic forgetting [2, 3] , which causes trained models to lose existing knowledge when gaining new information quickly. This can be attributed to the fundamental property of gradient-based iterative algorithms that impose a task-recency bias, i.e., predictions favor recently updated categories [4] . To the authors' knowledge, no solutions exist for these gradient-trained CIL models to fully tackle catastrophic forgetting.On the other hand, traditional CIL assumes that the number of samples in each task is fixed and that new tasks are entirely disjoint from previous ones. This paradigm does not align with real-world scenarios, where training data may include both new and previously encountered categories, and the number of data points often exhibits arbitrariness in each task. This extended CIL setting is referred to as generalized CIL (GCIL) [5, 6] . Such an uneven task-wise distribution of training samples and data categories further complicates the forgetting issue. For instance, GCIL may lead to the neglect of minority samples within a batch, thereby undermining representation during the training process.To mitigate catastrophic forgetting, a simple but effective approach is to replay historical samples. Replay-based CIL [1, 4] mitigates forgetting by storing a small number of samples from historical categories for the model to review while learning new information. However, this replay mechanism poses risks to data privacy. Thus, the exemplar-free CIL (EFCIL) without saving old exemplars gains prominence due to the increasing concern for privacy. However, many EFCIL methods perform poorly due to the task-recency bias caused by the nature of gradient-based algorithms [4] . Recently, this dilemma has been alleviated by the analytic continual learning (ACL) [7, 8] , an emerging EFCIL branch that first achieves comparable or even more competitive performance over the replay-based CIL. This improvement occurs because, for the first time, ACL achieves a near "complete nonforgetting" by allowing an equivalence between the incremental learning and its joint training (i.e., the weight-invariant property).The ACL provides a powerful toolbox for traditional EFCIL scenarios where data categories among training tasks are mutually exclusive. However, an apparent gap exists between the existing ACL techniques and the more desired and real-world GCIL scenario. Exploring the possibility of incorporating the weight-invariant property into the GCIL framework is both a significant and natural motivation, as it has the potential to enhance overall performance. To achieve this, we propose a generalized analytic continual learning (GACL), a new and compensated ACL member, offering a weight-invariant property solution to the GCIL. The key contributions are summarized as follows.• We present the GACL, an exemplar-free technique that achieves the equivalence between the GCIL (with split incoming data) and its joint training (with data centralized in a single task). • We theoretically establish the GACL's weight-invariant property. It is achieved and proved by separating the incoming data into exposed and unexposed components and aligning them structurally with matrix decomposition techniques. • We isolate the distinctive component of the GACL, namely the exposed class label gain (ECLG), from the existing ACL. This module explains the feasibility of achieving GCIL's analytic learning, offering a high interpretability in the GCIL realm. • Experiments on various benchmark datasets are presented, showing that the GACL outperforms the existing EFCIL by a large margin. It also exceeds most state-of-the-art replay-based methods. 
related works: 
 This section reviews existing methods for CIL and its more real-world counterpart, i.e., GCIL.Existing CIL methods can be roughly divided into three categories: replay-based methods, regularization-based methods, and prototype-based methods.The replay-based CIL methods such as iCaRL [1] , LUCIR [4] , PODNet [9] , AANets [10] , FOSTER [11] , and OHO [12] , retain past training samples as exemplars and utilize them during the learning of new ones. However, storing original training samples presents a significant challenge, particularly in scenarios with strict data privacy requirements.The regularization-based CIL aims to design a loss function that prevents the change of activations or important weights. Methods such as the Less-forgetting learning [13] and the LwF [14] introduce knowledge distillation [15] into their loss function to prevent the forgetting caused by activation drift. EWC [16] , EWC++, RWalk [17] , and Rotate your Networks [18] , introduce regularization that slows down learning on the weights important for old tasks by calculating the Fisher information matrix.The prototype-based CIL maintains distinct prototypes for each category, which prevents overlapping representations of new and old categories. For example, the PASS [19] distinguishes prior categories by augmenting feature prototypes. The SSRE technique [20] enhances the dissimilarity between old and new categories via selecting prototypes to incorporate with new samples into a distillation process. The FeTrIL [21] uses new representations to generate pseudo-features of old categories.The ACL is a recently developed EFCIL branch inspired by the analytic learning [22, 23, 24] where the training of neural networks yields a closed-form solution using least squares. The ACIL [7] first converts a continual learning problem to a batch recursive least-squares problem, eliminating the need to store samples by preserving the correlation matrix, and the RanPAC [25] applies this trick to pre-trained models. The GKEAL [8] focuses on the few-shot CIL scenarios by leveraging a Gaussian kernel projection. The DS-AL [26] introduces an additional linear classifier to learn the residue of the ACIL to enhance the plasticity, while the REAL [27] introduces the representation enhancing distillation to improve the backbone's generalization capabilities. The AFL [28] extends the ACL to federated learning, transitioning from temporal increment to spatial increment, and similar techniques are applied to the reinforcement learning [29] . The ACL is an emerging competitive CIL branch with a closed-form solution that leads to a valuable weight-invariant property, securing the equivalence between CIL and its joint learning. However, existing ACL methods are designed for the CIL scenario in which the categories of samples in each task must be entirely distinct. This restricts their applicability in real-world scenarios.The GCIL simulates real-world incremental learning, as distributions of data category and size could be unknown in one task. The GCIL arouses problems such as intra-and inter-task forgettings and the class imbalance problem [30] . The key GCIL properties can be summarized as follows: (i) the number of classes across different tasks is not fixed; (ii) classes shown in prior tasks could reappear in later tasks; (iii) training samples are imbalanced across different classes in each task [6] (See Appendix B).There are several GCIL settings. In the BlurryM setting [5] , a% of the classes are disjoint between tasks, while the remaining classes appear in every task. The i-Blurry-N-M [31] setting has blurry task boundaries and requires the model to perform anytime inference. However, the i-Blurry scenario has a fixed number of classes in each task with the same proportion of new and old classes. The Si-Blurry [30] is the most complex and realistic GCIL setting satisfying all three GCIL properties since it has an ever-changing number of classes and is capable of effectively simulating newly emerging or disappearing data, highlighting the problem of uneven distribution in real-world scenarios.To address the issue of the GCIL, gradient-based sample selection methods such as the GSS-IQP and the GSS-Greedy are proposed by [5] . The RM [32] proposes a memory management strategy based on per-sample classification uncertainty and data augmentation, while the management in the CLIB [31] eliminates samples based on a per-sample importance score. The DualPrompt [33] , as an EFCIL method, introduces the prompt-based learning to the CIL problem, and the MVP [30] proposes an instance-wise logit masking and contrastive visual prompt tuning loss.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In Section 4.5, a discussion of the limitations and the future work of the GACL is conducted.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the proposed method: 
 In this section, we deliver details of the proposed GACL. We first define the learning problem. Then, we derive the GACL by employing matrix decomposition techniques. A corresponding theoretical analysis follows to indicate the interpretability of our work. An overview is depicted in Figure 1 .We denote the complete set of available data as D. When D is partitioned into a sequence of GCIL tasks, we assume that D train unexposed takes in the contribution of unexposed class data (see (11) ). The other is contributed by the ECLG moduleŴ (k)ECLG (e.g., see ( 12)), which reflects the gain of exposed class data on the seen categories. The recursive formulation flows aided by the autocorrelation memory matrix R throughout the GCIL.is the test dataset in task k. The goal of GCIL in task k is to train networks using D train k and evaluate their performance on the test dataset D test 1:k . Here, D 1:k denotes the joint dataset spanning tasks 1 to k.In each GCIL task, classes may not appear exclusively. Hence, in any GCIL task k, we refer to classes that have appeared in previous tasks 1 to k -1 as the exposed classes of task k, while classes making their initial appearance are the unexposed classes of task k as shown in Figure 1 (a). This distinction helps to characterize the evolving nature of class occurrences throughout different GCIL tasks.In a task-wise GCIL scenario, we can involve all class labels in a set S. In task k, the set of the exposed class labels is denoted as S exposed, k ⊆ S, while the set of unexposed class labels is marked by S unexposed, k ⊆ S, where S exposed, k ∩ S unexposed, k = ∅. Note that S exposed, k and S unexposed, k may evolve from task k -1 to task k, that isS exposed, k = S unexposed, k-1 ∪ S exposed, k-1 = S unexposed, k-1 ∪ S unexposed, k-2 . . . ∪ S unexposed, 1 .From the scope of exposed-unexposed classes, the d y k can be represented as d y k = |S exposed, k | + |S unexposed, k | = d y k-1 + |S unexposed, k |,EQUATIONwhereȲ train k ∈ R N k ×dy k-1 is the exposed class label matrix and Ỹ train k ∈ R N k ×(dy k -dy k-1 )is the unexposed class label matrix. They correspond to segments displaying the appearance of exposed classes and unexposed classes.The power of pre-trained models allows the GACL to adopt a frozen backbone from structures such as ViT [34] to extract the features of images shown in Figure 1 (b) . LetEQUATIONbe the features extracted by the backbone, where Θ backbone indicates the backbone weight. Then, we use a buffer layer to project features, i.e.,EQUATIONwhere f buffer indicates the operation of the buffer layer. Several options for the buffer layer exist, including a randomly initialized linear mapping in the ACIL [7] or a kernel embedding projection in the GKEAL [8] . The selection of the buffer layer is not our focus. For convenience, we follow the ACIL, taking the random linear projection followed by a non-linear activation function as the buffer layer, i.e. f buffer (X (E) ) = ReLU(X (E) W B ), where the elements of the buffer layer weight W B are randomly sampled from a normal distribution.Here, we derive the GACL by partitioning training samples into unexposed and exposed categories, as shown in Figure 1 (c). Let X total 1:k and Y total 1:k be the accumulated feature and label matrices in task k, which can be extended from the accumulated matrices X total 1:k-1 and Y total 1:k-1 in task k -1 as follows.X total 1:k = X total 1:k-1 X (B) k , Y total 1:k = Y total 1:k-1 0 Ȳ train k Ỹ train k .Subsequently, one could formulate the learning problem in task k by a fully connected network (FCN) as the classifier argminEQUATIONwhere ∥•∥ F is Frobenius-norm, γ ≥ 0 is the regularization term and WFCN indicates the FCN layer weight. The optimal solution to (4) isEQUATION)The goal of the GACL is then to obtainŴ (k) FCN recursively from Ŵ (k-1)FCN without directly involving historical samples (e.g., X total 1:k-1 and Y total 1:k-1 ). That is to solveEQUATION)by recursively updating the previous-task weight Ŵ (k) FCN . To achieve this, we define an autocorrelation memory matrix as follows.EQUATIONAccordingly, we summarize the recursive formulation of the proposed GACL in Theorem 3.1.Theorem 3.1. Let Ŵ (k)FCN be the optimal estimation of (6) with all the training data from task 1 to task k. Then Ŵ (k) FCN is equivalent to its recursive formEQUATIONwhereEQUATIONProof. See Appendix A.As indicated in Theorem 3.1, the weight Ŵ (k) FCN in task k recursively obtained using the previous-task weightŴ (k-1)FCN is identical to its joint-learning counterpart formulated in (6) . That is, the GACL maintains the same weight-invariant property in the GCIL scenario as other ACL methods. EQUATIONEQUATIONW (k) FCN ← W (k) unexposed + W (k) ECLG end forThe pseudo-code of the GACL is listed in Algorithm 1.Exemplar-free. The recursive formulation is aided by R k as indicated in (9) . Note that this autocorrelation memory matrix records the inverse of inner products among the historical embedding matrices as shown in (7) . Hence, the embeddings (e.g., X (B) k ) are not reversible. Saving R k instead of used samples is a safe alternative to preserve past knowledge. That is, our GACL is an exemplar-free technique without the need to keep any historical samples.To more properly explain our GACL, as indicated in Figure 1 (c), the recursive solution in ( 8) can be rewritten as the sum of the unexposed-class contributed weight Ŵ (k) unexposed and the ECLG weightEQUATIONwhereEQUATIONEQUATIONUnexposed-class Contributed Weight. The unexposed-class contributed weight Ŵ (k) unexposed is recursively updated by the data of the unexposed class only. Note that the unexposed class label Ỹ train k is applied on the concatenated weight along with new data X (B)⊤ k , which is reasonable as historical information should not intervene with the weight update of unseen classes. On the other hand, new data X (B)⊤ k could also affect historical knowledge. This is marked by the gain of-R k X (B)⊤ k X (B) k Ŵ (k-1) FCN to the original weight Ŵ (k-1)as indicated in (11) . ). It is important to note that when Ȳ train k is empty (i.e., no classes reappear in task k), this component does not contribute to the update of Ŵ (k) FCN . This module is also isolated to distinguish GACL's difference from the existing ACL methods in a mathematical analysis manner (indicated as follows).Difference from Existing ACL Methods. Overall, the GACL can be treated as a nontrivial generalization of ACIL [7] , GKEAL [8] , and various other ACL methods. For instance, in conventional CIL where no classes reappear in new tasks (i.e., ∀k,Ȳ train k ∈ R * ×0 ), the classifier of the GACL Ŵ (k) FCN = Ŵ (k)unexposed , which is equivalent to the recursive classifier of the ACIL. That is, the ACIL is a special case of our proposed GACL. The major difference lies in the ECLG module, corresponding to the exposed-class gain. This pattern makes sense as there must be compensation on top of ACIL updates (specifically designed for traditional CIL) when exposed data (out of setting) participate. 4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The proof of the Theorem 3.1 is listed in Appendix A. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 In the section, we conduct experiments on various benchmark datasets and compare the GACL with both EFCIL and replay-based state-of-the-art methods, including LwF [14] , L2P [36] , DualPrompt [33] , ER [35] , EWC++ [16] , SLDA [37] , RM [32] , MVP [30] , and MVP-R (MVP with exemplars). 3Datasets. We conduct experiments on three datasets: CIFAR-100 [38] , ImageNet-R [39] , and Tiny-ImageNet [40] . We evaluate each method under the Si-Blurry setting [30] (the most complex GCIL setting) with 5 independent seeds. For the Si-Blurry setting, we set the disjoint class ratio r D to 50% and the blurry sample ratio r B to 10%. More details about Si-Blurry are listed in Appendix C.Implementation Details. We utilize the DeiT-S/16 [41] as our backbone. Following [42, 43] , we pre-train the backbone on 611 ImageNet classes after excluding 389 classes that overlap with CIFAR and Tiny-ImageNet to prevent data leakage. To ensure a fair comparison, all methods utilize a frozen backbone. All methods under comparison are implemented as specified in [30] . The memory sizes of compared relay-based methods are set to 500 and 2000.There are two hyperparameters in the GACL, the regularization term γ and the size of the buffer layer.Here, we adopt γ = 100, which is determined by the grid search of {0, 10, 100, 500, 1000, 10000} on CIFAR-100 (by a 90%-10% train-val split). As the regularization term γ is not sensitive in a proper range [7] , we adopt this value for all datasets for convenience. We relocate its analysis to Appendix E. The size for the buffer layer W B is set to 5000 for both the GACL and ACIL for convenience.Evaluation Protocol. Three metrics are adopted to evaluate GCIL tasks. The real-time performance is evaluated by the area under the curve of accuracy A AUC [31] , i.e., A AUC = k i=1 f (i • △n) • △n, where △n is the number of samples observed between evaluation and f (•) is the curve in the accuracyto-{number of training samples} plot, measuring anytime inference performance during training. A higher A AUC corresponds to a method consistently maintaining high accuracy throughout the training. The overall performance is evaluated by the average incremental accuracy (or average accuracy)A Avg = 1 K+1 K k=1 A k ,where the task-wise accuracy A k indicates the average test accuracy in task k by testing the network on D test 1:k . A higher A Avg score is preferred when evaluating algorithms. The last evaluation metric is the last-task accuracy A Last evaluating the network's last-task performance after completing all tasks.As shown in Figure 2 , we comprehensively compare the GACL with both EFCIL and replay-based methods. Compare with EFCIL Methods. EFCIL methods address privacy concerns and mitigate catastrophic forgetting without exemplars. Among EFCIL methods, our GACL consistently exhibits superior performance across all three datasets, as illustrated in the lower panel of Table 1 .For instance, on CIFAR-100, our method surpasses the second-best method SLDA, by 4.99%, 6.15%, and 8.52% for A AUC , A Avg , and A Last , respectively. On Tiny-ImageNet, the GACL achieves impressive results with A AUC , A Avg , and A Last reaching 63.14%, 69.32%, and 62.68%, respectively, surpassing the previous best EFCIL by 13.97%, 21.39%, and 9.55%. Similar patterns are evident in the results of ImageNet-R, further confirming that the GACL is an exceptional tool for GCIL.Owing to the weight-invariant property, the GACL exhibits more accurate and stable evolutions as k increases as observed in Figure 2 (a) . All compared EFCIL methods exhibit sharp declines in accuracy, while the GACL delivers nearly non-declining curves. In particular, on CIFAR-100, the GACL shows an unnatural improvement of task-wise accuracy throughout the learning tasks, with the GACL initially lagging behind other EFCIL methods. This is because the Si-Blurry samples more than 70% of the CIFAR-100 categories in the first two tasks (see Appendix F), constructing a scenario where gradient-based algorithms could largely avoid the forgetting issue. Moreover, our method produces more stable predictions across diverse scenarios, as indicated by much smaller standard errors (colored shades in Figure 2 (a) ). In summary, the experimental results demonstrate that our proposed GACL is exceedingly accurate and robust, exhibiting exceptional generalization ability.Compare with Replay-based Methods. Replay-based methods are considerably competitive as they leverage historical samples. The memory size is a key adjustment, as increasing it typically leads to performance improvements by allowing more historical knowledge to be reviewed. For instance, the MVP-R achieves 4.42%, 3.97%, and 8.95% gains for A AUC , A Avg , and A Last (see Table 1 ) on CIFAR-100 when increasing the memory size from 500 to 2000.As an exemplar-free technique, our GACL avoids re-using the historical samples. However, as indicated in Table 1 , the GACL still outperforms most existing replay-based results. For instance, the GACL achieves the best A Last results among all settings. The GACL's A AUC and A Avg results are also mostly superior, except that our performance is slightly weaker than that of MVP-R with a memory size of 2000 on CIFAR-100 and ImageNet-R. Although increasing the number of exemplars can further improve the results of replay-based methods, this approach could lead to higher training and memory costs and, more importantly, more severe privacy invasion.As indicated in Figure 2 (b), replay-based methods experience accuracy declines similar to those observed in the EFCIL case. This decline is due to an inherent limitation of gradient-based iterative algorithms, which tend to favor recently trained categories and thus lead to catastrophic forgetting. The GACL is iterative-free and then not constrained by this forgetting issue, thereby achieving nearly no performance reduction as K increases.Why the GACL Gives Leading Performance. The above comparisons show that the proposed GACL is a powerful GCIL technique. Its competitive performance can be explained as follows. (i) Weight-invariant property. As shown in Theorem 3.1, the weight obtained recursively is equal to its joint-learning counterpart, indicating that the GACL is a "completely non-forgetting" technique (under the condition of a frozen backbone). (ii) Analytical solution. Existing GCIL techniques are gradient-based iterative algorithms prone to catastrophic forgetting by nature. The GACL is a new member of the ACL and inherits its non-iterative gradient-free essence with an analytical solution, thereby avoiding the task-recency bias to address forgetting.The ECLG module is a core component that allows the GACL to obtain the weight-invariant property in the GCIL scenario. Here, we conduct an ablation study to justify the ECLG's contributions under various blurry sample ratios r B with r D = 50%. Larger r B indicates more complex data distributions in the Si-Blurry setting. As shown in Table 2 , the GACL without ECLG exhibits poor performance with a visible gap for A AUC , A Avg , and A Last . For instance, on CIFAR-100 with r B = 10%, the ECLG contributes a 23.01% A Last gain to the GACL. As claimed in Theorem 3.1, the classifier without the ECLG module fails to absorb knowledge from joint classes in each task (i.e., classes that reappear), leading to substantial information loss under the GCIL setting. The GACL, equipped with the ECLG module, demonstrates competence in handling overlapping classes in realistic scenarios.Here, we conduct a robust analysis by varying the disjoint class ratio r D and the blurry sample ratio r B . The comparison happens among the GACL, the second-best EFCIL method SLDA, and the top-performing replay-based method MVP-R with a memory size of 500.We evaluate our method under various r D , including extreme cases where each task shares classes (r D = 0%) and traditional CIL scenarios (r D = 100%). Table 3 illustrates that our GACL consistently outperforms the compared methods (e.g., leads the SLDA by 2%-10%) and produces near-identical A Last values with varying r D . This shows the accurate and robust traits of the GACL.We also evaluate our method using various r B values, as shown in Table 4 . Similar patterns observed here align with those in Table 3 , further demonstrating the robustness of the proposed GACL, which delivers exceptional performance across different GCIL settings. Overall, the GACL exhibits various good characteristics as an exemplar-free GCIL technique. The major limitation here is the need for a well-trained backbone because the GACL does not update backbone weights. This could motivate the exploration of adjustable backbones to continuously improve their feature extraction abilities, thereby further enhancing GACL's performance.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Our paper comprehensively outlines both the experimental implementation and algorithmic details, ensuring transparency in our method. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we introduce the exemplar-free generalized analytic class incremental learning (GACL) approach to address the GCIL problem. Building upon analytic learning, the GACL delivers closedform solutions to GCIL through the decomposition of GCIL data into exposed and unexposed classes.The GACL achieves the weight-invariant property that provides identical solutions for GCIL to its joint learning counterpart. We theoretically validate this property and provide high interpretability through the matrix analysis tool. Various experiments are conducted under the Si-Blurry setting, demonstrating that our proposed GACL achieves remarkable performance with high robustness compared to state-of-the-art EFCIL and replay-based methods.A Proof of Theorem 3.1Proof. in task k -1, we haveEQUATIONHence, in task k, we haveEQUATION)We have defined the autocorrelation memory matrix R k-1 in the paper viaEQUATIONTo facilitate subsequent calculations, here we also define a cross-correlation matrixEQUATION)Thus we can rewrite (13) asEQUATIONTherefore, in task k we haveEQUATIONFrom ( 15), we can recursively calculate R k from R k-1 , i.e.,EQUATION)According to the Woodbury matrix identity, we haveEQUATIONHence, R k can be recursively updated using its last-task counterpart R k-1 and data from the current task (i.e., X (B) k ). This proves the recursive calculation of the autocorrelation memory matrix.Next, we derive the recursive formulation of Ŵ (k) FCN . To this end, we also recurse the cross-correlation matrix Q k in task k, i.e.,EQUATIONwhereEQUATION)Note that the concatenation in ( 22) is due to the assumption that Y train 1:k in task k contains more data classes (hence more columns) than Y train 1:k-1 . It is possible that there are no new classes appear in task k, then Ỹ train k should be 0. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The models and benchmark datasets mentioned in the paper are all openly accessible with no personally identifiable information or offensive content. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have verified that this paper cites all the datasets and models we used. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We have made our source code publicly available at https://github.com/ CHEN-YIZHU/GACL. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: EQUATIONWe haveEQUATIONHence, Ŵ (k) FCN can be rewritten asEQUATION)By substituting (20EQUATIONTo simplify this equation, letK k = (I + X (B) k R k-1 X (B)⊤ k ) -1 . Since I = K k K -1 k = K k (I + X (B) k R k-1 X (B)⊤ k ),we haveEQUATIONSubstituting ( 27) into (26), R k Q ′ k-1 can be written asEQUATIONSubstituting ( 28) into (25) implies thatEQUATION)which completes the proof.The GCIL scenario [6] is a recent CIL focus. Given task-wise learning tasks, we can involve all class labels in a set S with the number of classes N . The sample sizes, such as the numbers of input images of different classes appearing in task k, are modeled as a random vector c k ∈ R N . Each entry c k,i is a random variable denoting the sample size of class i in task k. In the generalized form, c k is sampled from a task-dependent distribution. The GCIL scenario can be summarized as the following three key properties. Property B.1. The number of classes in a task is not fixed. Suppose m k is the number of classes in task k, we have:EQUATIONwhere M k is a task-dependent distribution. Property B.2. Classes appearing in different tasks could overlap. For two tasks k and k ′ , k ̸ = k ′ , we have:EQUATIONwhere ⊙ denotes element-wise multiplication of two vectors and P (•) is the probability. Property B.3. Sample sizes of different classes at the same task could be different. That is, for task k, we haveEQUATIONIn short, the number of classes and samples could vary throughout the continual learning. The Si-Blurry setting [30] satisfies all the three properties of GCIL mentioned in Appendix B and can be treated as its good realization. As shown in Figure 3 , for a K-task learning, the Si-Blurry first randomly partitions all classes into two groups: disjoint classes that cannot overlap between tasks and blurry classes that might reappear. The ratio of partition is controlled by the disjoint class ratio r D , which is defined as the ratio of the number of disjoint classes to the number of all classes. Then disjoint classes and blurry classes are randomly assigned to disjoint tasks (T D ) and blurry tasks (T B ) respectively. Next, each blurry task further conducts the blurry sample division by randomly extracting part of samples to assign to other blurry tasks based on blurry sample ratio r B , which is defined as the ratio of the extracted sample within samples in all blurry tasks. Finally, each Si-Blurry task T B+D with a stochastic blurry task boundary consists of a disjoint and blurry task. We adopt Si-Blurry with different combinations of r D and r B for reliable empirical validations.GPU Usage. We conduct experiments in PyTorch on one Nvidia Geforce RTX 4090 GPU with a batch size of 64 for training and 128 for inference. Figure 4 shows that the GACL uses minimal GPU memory. Our GACL significantly reduces GPU memory usage since it requires no back-propagation, thereby detaching gradients from tensors during calculations. This characteristic allows our approach to be applied with a larger batch size without memory leaks. Training Time. Table 5 further illustrates the GACL's training time compared to others on one Nvidia Geforce RTX 4090 GPU, highlighting its efficiency. The GACL is faster than any other baselines except SLDA on three datasets because only the classifier and autocorrelation memory matrix R are updated, leading to small numbers of trainable parameters compared to those baselines in a back-propagation manner. The regularization term γ plays a crucial role and demonstrates robust behavior throughout our experiments. We assess the impact of the regularization term γ in Table 6 and visualize the real-time accuracy of the GACL as it learns from training samples in Figure 5 . Table 6 reveals the GACL's consistent performance across a broad range of γ values, spanning from 10 to 10000. This highlights the versatility and robustness of our proposed GACL. However, as indicated in Figure 5 , γ of 10000 leads to slightly poorer performance because the ACL is prone to underfitting due to simple linear regression [44] . Notably, both Table 6 and Figure 5 demonstrate that the absence of regularization results in a significant decline in performance. This underscores the crucial importance of incorporating γ in the model. As indicated in (7) , if we eliminate regularization by setting γ to 0, the initial autocorrelation memory matrix R 0 becomes zero. Subsequently, the computation of the autocorrelation memory matrix in task 1, denoted as R 1 , is expressed as:R 1 = (X total⊤ 1 X total 1 ) -1 = (X (B)⊤ 1 X (B) 1 ) -1 .However, it's crucial to emphasize that X (B)⊤ 1 X (B) 1 might result in a singular matrix, rendering it non-invertible. This potential singularity introduces an error in calculating R 1 , leading to a decrease in accuracy.As depicted in Figure 2 (a), the task-wise accuracy of the GACL on CIFAR-100 demonstrates an increase. Notably, in the initial two tasks, the accuracy is lower compared to other EFCIL methods. However, on the other datasets, the GACL remains relatively stable. Upon a more detailed examination of the dataset split, we infer that the observed variations in trends are attributed to the specific dataset settings.For a dataset with N classes, the class number ratio r c after training on i-th samples is defined as r c = d i /N , where d i is the number of classes that have been seen observed at that point. As Figure 6 indicates, by examining the real-time accuracy and the class number ratio r c across the three sets of figures, a notable observation is made: when the sample size is small, the class number ratio r c on CIFAR-100 always surpasses that of the other two datasets on 5 seeds. This suggests that tasks on CIFAR-100 are notably more complex and intricate, resembling a few-shot learning scenario.Consequently, the GACL exhibits lower task-wise accuracy compared to other gradient-based EFCIL methods, particularly in the initial stages. However, as more training samples are acquired, its accuracy progressively improves. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.The results for MVP and MVP-R are based on their official implementation, committed on October 26, 2024 (commit ID: ad8d1426a497545ba634521c00008c34ceece799). 


Decision: The paper addresses the problem of a generalized form of lifelong learning, where learning is pursued in a continuous manner, with the possibility of revisiting samples from previously learned concepts. The authors demonstrate that this problem can be efficiently addressed using the concept of analytical learning. Initially, the paper received mixed feedback: 1x3 (reject), 1x4 (borderline reject), 1x6 (weak accept), and 1x8 (strong accept). After the rebuttal and the author-reviewer discussion period, the scores improved to 1x5 (borderline accept), 1x6 (weak accept), 1x7 (accept), and 1x8 (strong accept), with all reviewers recommending acceptance.

The reviewers acknowledged the novelty of the idea within the context of the problem, and the AC agrees with this assessment, recommending the paper for acceptance. Congratulations.

The AC suggests including all the results discussed during the rebuttal and the author-reviewer discussions. Additionally, the formulation of the classification problem as a regression, as done in the paper, can be acknowledged as a limitation of the current work.

Congratulations again.

Reviews: review: paper_summary: paper_summary: This paper deals with the generalized CIL (GCIL) problem where incoming data have mixed data categories and unknown sample size distribution. The author proposes generalized analytic continual learning (GACL) which adopts a pre-trained and fixed backbone and uses least squares to get a closed-form solution. Experiments verified that the GACL achieves better performance compared to other baselines.
strength_weakness: strength_weakness: * Main concern. Most of the content on page 4 and page 5 (e.g., Theorem 3.1) in Section 3 is overly similar to existing ACL works [7, 8]. The main difference claimed by the authors, i.e., the ECLG module that corresponds to the exposed-class gain, is a trivial extension of the original ACL [7].
* Experiments. This paper adopts the pre-trained backbone developed in [40, 41]. However, many baselines (Table 2 in [40]) are missing in experiments. The GACL method is similar to RanPAC [25]. Experiments are needed to compare the performance of GACL with RanPAC.
* Comparison of the memory cost of weights in the buffer layer and other replay-based method.
questions: questions: * How and why the dimensionality of the random projection affects the performance.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a new exempler-free generalized continual learning (GCIL), named generalized analytic continual learning (GACL) technique. It does not depend on gradient-based tranining, which avoids the task-recency bias leading to the forgetting issue. It also delivers an closed-form solution to the GCIL scenario that provides identical solutions to its joint training. Entensive experiments demonstrate that the proposed GACL achieves consistently leading performance.
strength_weakness: strength_weakness: - Missing baselines. As described in the Section 2, many ACL methods have been proposed. Comparing GACL with those methods (especially RanPAC) derives more solid results.
- Comparison to the joint-traning is not included. Although it is argued that GACL can bridge the gap between continual learning and joint-training, such result is not provided in the experiments.
questions: questions: - I believe that the proposed gradient-free training scheme is beneficial also in the perspective of training speed. Does GACL run more fast than other (ACL) methods?
limitations: limitations: - As described, GACL cannot update backbone weights, which can be a problem. However, as new backbones emerge frequently, showing that GACL can work well not only with the used DeiT-S/16 but also others makes the arguments more convincing, instead.
paper_summary: paper_summary: Class Incremental Learning (CIL) faces the problem of catastrophic forgetting when training a network, i.e., the model loses previous knowledge when learning a new task. Generalized CIL (GCIL) aims to address more realistic scenarios, but existing methods are either ineffective or violate data privacy. This paper propose a new exemplar-free GCIL technique called Generalized Analytic Continual Learning (GACL), which provides a closed-form solution through analytic learning. GACL achieves equivalence between incremental learning and co-training by disaggregating the input data to keep the weights constant. This approach is theoretically validated and performs well across multiple datasets and settings.
strength_weakness: strength_weakness: * The experimental setup is not clear. Given that the most recent method MVP [1] compared in this paper is the GCIL method for online CIL, were the experiments in this paper also conducted in an online scenario?
* The technical improvements on the previous work of ACL are somewhat incremental.

[1] Jun-Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning. ICCV 2023.
questions: questions: 1. What was the motivation for introducing ACL into GCIL? The claims in this paper seem to be an extended experiment of ACL in the GCIL scenario. It is difficult to generate further inspiration to the reader for solving the GCIL problem.
2. The proposed method utilize the DeiT-S/16 as backbone but previous works typically use ViT-B/16. Regarding the comparison methodology, what backbone network was used to obtain the reported results?
3. What are the technical improvements and contributions of this paper over previous ACL approaches such as ACIL [2] and DS-AL [3]?

[2] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, RENCHUNZI XIE, Kar-Ann Toh, and Zhiping Lin. ACIL: Analytic class-incremental learning with absolute memorization and privacy protection. NeurIPS 2022
[3] Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, and Zhiping Lin. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning. AAAI 2024.
limitations: limitations: It is hard to foresee any potential negative societal impact of this theoretical work.
paper_summary: paper_summary: This paper proposes a new exemplar-free GCIL technique named generalized analytic continual learning (GACL). It adopts analytic learning (a gradient-free training technique), and delivers an analytical (i.e., closed-form) solution to the GCIL scenario, which is derived via decomposing the incoming data into exposed and unexposed classes. The GACL attains a weight-invariant property, supporting an equivalence between the incremental learning and its joint training. This method is both therectically and empircally validated.
strength_weakness: strength_weakness: 1. Could you explain why the avg accuracy on CIFAR-100 is much lower than the last accuracy?
2. You need to explain more regarding "weight-invariance" in Theorem 3.1. It is not very easy to follow for those who are not in this area of research.
3. In this paper, the authors assume that the pre-trained backbone is generalizable so that the model can be frozen throughout the learning process. However, there are also some cases where the pre-trained model cannot generalize to downstream tasks. It would also be interesting to see the scenario where the pre-trained backbone yields a large domain gap.
questions: questions: 1. Continual learning has many variations (e.g., domain/task/class -incremental learning). Since the experiments only focus on generalized class-incremental learning, it would be better to adjust the title accordingly.
2. Since generalized CIL is also a specific case of class-incremental learning, it would be better to discuss why the proposed method is specially designed for GCIL. For example, when the data stream only contains new classes, will GCIL still work?
3. As the authors discussed in the main paper, the proposed method is a super case of ACIL and other analytical methods. As many readers would like to know, the authors are suggested to compare this method to other analytical methods to show how this method stands against other analytical works.
limitations: limitations: Yes

Comments: title: Thank you for the response
Thank you for the response
Comparison update
More updates
Thank you for the response
Official Comment by Reviewer
comment: Thank you for taking the time to read our response! We are glad that our response addressed your concerns.
I appreciated the author's responses and have raised my score to 5.
Thank you for taking the time to read our response! We are glad that our response addressed your concerns!
Dear Reviewer,

We would like to provde our updates during the discussion period, in which we have still been working on the response regarding the comparion with the RanPAC.

We have managed to produce the results in comparison with RanPAC as follows.

Overall, the RanPAC is a very strong-performing counterpart with average and final results comparable to ours. However, it is **NOT designed for GCIL/online tasks**, and hence **gives limited performance in ACC_AUC** (area under the curve of accuracy), which measures the online performance. For instance, on CIFAR-100, the GACL obtains 66.79% while RanPAC only has 51.62%.

|                        | Buffer | CIFAR100 |         |         | ImageNet-R |         |         |  Tiny-ImageNet  |         |         |
|:----------------------:|:------:|:--------:|:-------:|:-------:|:-------------:|:-------:|:-------:|:-------------:|:--------:|:-------:|
|                        |        | ACC_AUC | ACC_AVG | ACC_LAST |ACC_AUC | ACC_AVG | ACC_LAST |ACC_AUC |ACC_AVG | ACC_LAST |
| RanPAC (NeurIPS,2023)     |  0  |  51.62 |     63.45 |  77.83  |  42.39  |  61.68  |   57.77  |  62.80  |  82.80  | 78.54  |
| GACL               | 0   | 66.79   | 63.94    | 77.34 |57.02      |62.26      |57.68      |77.65      |82.95      |77.80     |

Let us know if more clarification is required!
Dear Reviewer,

We would like to provde our updates during the discussion period, in which we have still been working on the response.

1) We have managed to produce the results in comparison with RanPAC as follows.

Overall, the RanPAC is a very strong-performing counterpart with average and final results comparable to ours. However, it is **NOT designed for GCIL/online tasks**, and hence **gives limited performance in ACC_AUC** (area under the curve of accuracy), which measures the online performance. For instance, on CIFAR-100, the GACL obtains 66.79% while RanPAC only has 51.62%.

|                        | Buffer | CIFAR100 |         |         | ImageNet-R |         |         |  Tiny-ImageNet  |         |         |
|:----------------------:|:------:|:--------:|:-------:|:-------:|:-------------:|:-------:|:-------:|:-------------:|:--------:|:-------:|
|                        |        | ACC_AUC | ACC_AVG | ACC_LAST |ACC_AUC | ACC_AVG | ACC_LAST |ACC_AUC |ACC_AVG | ACC_LAST |
| RanPAC (NeurIPS,2023)     |  0  |  51.62 |     63.45 |  77.83  |  42.39  |  61.68  |   57.77  |  62.80  |  82.80  | 78.54  |
| GACL               | 0   | 66.79   | 63.94    | 77.34 |57.02      |62.26      |57.68      |77.65      |82.95      |77.80     |

2) Regrarding your concern of novelty

we have clarified this in "**Response to W_1**" in our rebuttal. Reviewer qU32 also pointed out this issue, and **has aknowledged our clarification**. Could you have a look and let us know if more is required to clarify this?
I thank the authors for providing the rebuttal. After reading the rebuttal and other reviewers' comments, all my previous concerns have been adequately addressed. I will keep my positive rating.
Thank you for taking the time to read our response and increasing your score! We are glad to hear that the response addressed your concern.
Thanks to the authors for the response. This rebuttal addresses my concerns well. It fully explains the motivation for introducing ACL to address GCIL and the improvements this paper makes to existing ACL methods in response to the GCIL problem. Taking into account the comments of other reviewers and the authors' rebuttal, I decide to increase my rating.