Continual Generalized Category Discovery: Learning and Forgetting from a Bayesian Perspective
Authors: Hao Dai
Jagmohan Chauhan
Venue: ICML 2025 poster (2025)
Keywords: Continual Learning
Generalized Category Discovery
Variational Bayes
Primary area: applications->computer_vision
TL;DR: VB-CGCD, a variational Bayes-based framework for Continual Generalized Category Discovery, achieved a +15.21% improvement outperforming state-of-the-art methods.

Abstract: Continual Generalized Category Discovery (C-GCD) faces a critical challenge: incrementally learning new classes from unlabeled data streams while preserving knowledge of old classes. 
    Existing methods struggle with catastrophic forgetting, especially when unlabeled data mixes known and novel categories. 
    We address this by analyzing C-GCD’s forgetting dynamics through a Bayesian lens, revealing that covariance misalignment between old and new classes drives performance degradation. 
    Building on this insight, we propose Variational Bayes C-GCD (VB-CGCD), a novel framework that integrates variational inference with covariance-aware nearest-class-mean classification. 
    VB-CGCD adaptively aligns class distributions while suppressing pseudo-label noise via stochastic variational updates. 
    Experiments show VB-CGCD surpasses prior art by +15.21% with the overall accuracy in the final session on standard benchmarks. 
    We also introduce a new challenging benchmark with only 10% labeled data and extended online phases—VB-CGCD achieves a 67.86% final accuracy, significantly higher than state-of-the-art (38.55%), demonstrating its robust applicability across diverse scenarios. Code is available at: https://github.com/daihao42/VB-CGCD

Continual Generalized Category Discovery:
Learning and Forgetting from a Bayesian Perspective
Hao Dai 12
Jagmohan Chauhan 12
Abstract
ting- phenomenon where previously learned information is overwritten (Wang et al., 2024; Rebuffi et al., 2016; Kirk- patrick et al., 2016). This capability is essential in dynamic, real-world applications such as lifelong personal assistants, adaptive robotics, and autonomous systems, where tasks and categories evolve over time. Continual Category Discovery (CCD) (Zhou et al., 2024b) builds on this foundation, tack- ling the question of how to discover and learn new categories from streaming unlabeled data while retaining the ability to classify previously learned categories. However, real-world data streams often present a more complex challenge: they contain a mix of known and unknown classes, rather than being neatly partitioned into entirely new or entirely old categories (Geng et al., 2021).
Continual Generalized Category Discovery (C- GCD) faces critical challenge: incrementally learning new classes from unlabeled data streams while preserving knowledge of old classes. Exist- ing methods struggle with catastrophic forgetting, especially when unlabeled data mixes known and novel categories. We address this by analyzing C-GCD's s forgetting dynamics through Bayesian lens, revealing that covariance misalignment be- tween old and new classes drives performance degradation. Building on this insight, we propose Variational Bayes C-GCD (VB-CGCD), a novel framework that integrates variational inference with covariance- fication. VB-CGCD adaptively aligns class dis- tributions while suppressing pseudo-label noise via stochastic variational updates. Experiments show VB-CGCD surpasses prior art by +5.21%% with the overall accuracy in the final session on standard benchmarks. We also introduce a new challenging benchmark with only 10% labeled data and extended online Dh-CGS- achieves a 67.86% final accuracy, significantly higher than mate-of-theearr (38.55%), demonstrat- ing its robust applicability across diverse scenar- ios. Code is available at: https: // //gitub.. com/ da i hao42//B-CCGC
saarestcclass-mene classi-
C-GCD (Zhang et al., 2022; Wen et al., 2023; Wu et al., 2023) aims to push CI. closer to real-world applicability by addressing the challenge of mixed-class data streams. This paradigm raises two critical questions: (1) How can a model differentiate between known and unknown classes in an interwoven data stream without misclassifying known samples? (2) How can a model effectively balance learning classes while preserving old knowledge within noises? These challenges underscore the delicate trade-off between stability, the ability to preserve previously learned knowl- edge, and plasticity, the capacity to adapt to new categories. Existing methods address this tension through contrasting strategies: some prioritize plasticity (Zhang et al., 2022; Rypesc et al., 2024), focusing learning new classes while applying knowledge distillation or constraints to retain old class representations; while others prioritize stability (Douil- lard et al., 2022; Hu et al., 2023; Cendra et al., 2024), freez- ing feature extractors trained on earlier tasks and incremen- tally expanding classifiers.
1. Introduction
Continual learning (CL.) seeks address one of the most pressing challenges in machine learning: enabling mod- els to incrementally acquire knowledge from continuous stream of tasks without succumbing to catastrophic forget-
To maintain stability, recent approaches have increasingly leveraged pre-trained models (PTMs) (Zhou al., 2024a), which provide robust feature representations and strong generalization capabilities derived from extensive self- supervised training on diverse datasets. By fine-tuning PTMs, the issue feature drift can be mitigated to some extent. Many studies have focused on improving new-elass learning. such as optimizing inter-class distances (Wu et al., 2023) or extracting category-speciff feature representa-
'Department of Computer Science, UCL Centre for Artificial Intelligence, University College London, London, UK University of Southampton, Southampton, UK. Correspondence to: Hao Dai <daihaovigg @aaigggg@@mmmmmmmmmmmm www.g..ao........... com.>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning and Forgetting from Bayesian Perspective
tions (Han al., 2022). While these advancements have sig- nificantly improved stability, current PTM-based approaches often mitigate forgetting by introducing regularization to restrict the learning process, which limits their flexibility and adaptability when modeling new and emerging classes.
tions, facilitating the separation of known and novel cate- gories. Additionally, semi-suuerviieed learning (Rizve et al., 2021) has been integrated to group data into meaningful categories, further improving the discovery of new classes.
Continual Generalized Category Discovery. GCD typi- cally assumes simultaneous access to labeled data for known classes and unlabeled data for new classes. Continual Gen- eralized Category Discovery (C-GCD) builds on this by integrating the challenges of continual learning and GCD. Specifically, C-GCD requires models to incrementally learn from task streams while discovering new classes and retain- ing knowledge of previously learned ones. This combines the complexities of NCD and GCD and draws on techniques such as feature representation, contrastive learning, and metric learning.
Motivated by this limitation, we revisited the relationship between learning new classes and forgetting to explore a more dynamic solution. To this end, we propose a Bayesian framework that leverages Gaussian distribution for modeling dynamically evolving categories. We examine the factors influencing classification performance, with particular focus on the critical role of covariance in balancing the trade-off between stability and plasticity. Our contributions can be summarized as follows:
We formalize the role of covariance dynamics in C- GCD's prggetting-leaming tradeoff, revealing that dis- tribution misalignment (measured via Bhattacharyya distance) irreversibly biases decision boundaries.
A unique challenge of C-GCD lies in balancing the learning of new categories with the retention of prior knowledge, especially in exemplar-free settings where raw data from past tasks is unavailable. To address this, many methods rely on auxiliary information, such as prototypes (Snell et al., 2017; Belouadah & Popescu, 2019), attention maps (Saha et al., 2021), or leverage holistic representations of class distributions. CPP (Li et al., 2024) uses class means pro- totypes (prompts) and integrates a Transformer as the clas- sifier, while class means are often susceptible to the curse of dimensionality. Among these, Gaussian distribution has emerged as a powerful tool for modeling class distributions. Some approaches (Zhu et al., 2021) use multivariate nor- mal distribution (MVN) to generate replay data to mitigate forgetting, such as Happy (Ma et al., 2024) (which directly estimates distributions) and OCM (Ye & Bors, 2022) (which uses a VAE to approximate Gaussian distributions). On the other hand, PrompiCCD (Cendra et al., 2024), utilize Gaus- sian Mixture Models (GMM) as a prompting mechanism to enhance classification performance. FeCAM (Goswami et al., 2023) directly estimates Gaussian prototypes using statistical methods and performs classification via nearest- neighbor approach based on these prototypes.
We propose VB-CGCD, a Variational Bayes frame- work with Mahalanobis distance for C-GCD. By incor- porating covariance constraints, our approach enables robust learning of new categories while preserving pre- viously acquired knowledge efficiently.
We validate VB-CGCD on multiple datasets, showing an average 15.2 21% improvement in final accuracy over SOTA. We also introduce a label-limited benchmark (10% categories labeled, online sessions), where VB- CGCD's final accuracy reached 67. 67.86% outperform- ing the SOTA accuracy of 38.55%
2. Related Work
Novel Category Discovery. New Class Discovery (NCD) aims to identify new, unlabeled classes by leveraging metrics or evaluation criteria learned from known data. For instance, the rank statistics method introduced by AutoNovel (Han et al., 2022) labels new data by sorting features and em- ploying a BCELoss-based learning framework. Similarly, meta-learning approaches like MetaGCD (Wu et al., 2023) have garnered attention for their ability to generalize across tasks by learning class-agnostic similarity functions. These methods rely heavily on effective feature representations and robust distance functions to handle noise and ensure accurate differentiation between classes.
However, these methods often focus exclusively on either reducing forgetting or improving learning, lacking a unified analysis of the interplay between the two. VCL (Nguyen et al., 2018), GVCL (Loo et al., 2021), VAR-GP (Kapoor et al., 2021), and S-FSVI (Rudner et al., 2022) are methods that approach continual learning as a sequence of tasks, uti- lizing variational inference to regularize parameter updates via the KL divergence. These techniques bridge Bayesian inference with continual learning, providing insights into the trade-off between plasticity and stability. Although these approaches employ variational inference to regular- ize parameter updates for continual learning, they generally overlook the specific interplay between old and new classes in alassification-focusse CGCD scenarios.
Generalized Category Discovery. Generalized Category Discovery (GCD) builds upon NCD by addressing more complex scenarios where unlabeled data contains a mix of new and previously known categories. A central challenge in GCD is distinguishing between known and unknown data within unlabeled datasets. To tackle this, many ap- proaches leverage supervised and unsupervised contrastive learning (Chen et al., 2020) to enhance feature representa-
2

---

Learning and Forgetting from Bayesian Perspective
To address this gap, we propose directly modeling the distri- bution of each class and analyzing the relationship between forgetting and learning from a Bayesian perspective. This approach provides insights into optimizing overall classifica- tion accuracy. paving the way for efficient continual learning capable of adapting to dynamic, real-world scenarios.
3.2. Forgetting and Learning in C-GCD
Forgetting in C-GCD arises from various factors, the pri- mary challenges are associated with label bias, label error, and trade-off between learning and forgetting.
3.2.1. CHALLENGE 1: LABEL BIAS
3. Methodology
Conventionally, classification tasks rely on optimizing the maximum likelihood function to model P(y][x)). However, in the continual learning setting, labels are introduced se- quentially across different online sessions, Ieading to the issue of label bias. This bias arises because the model tends to overfit to the newly observed categories after training on a new dataset, often misclassifying samples into these newly learned classes, that is, the knowledge related to old classes is forgotten. This forgetting manifests in two key aspects: (1) Feature Space Shift: Continuous updates to the backbone network cause shifts in the feature distribu- tion, resulting in the failure of the old classification mem- ory. (2) Class Distribution Imbalance: Imbalanced category distributions across sessions lead the classifier to assign dis- proportionately higher likelihood probabilities to the new classes.
3.1. Problem Formulation
C-GCD involves two parts of datasets: labeled dataset Dj - {x{x v()} containing Cj categories, and an unlabeled dataset D. = {xx)) ), encompassing Cu categories. The un- labeled data is introduced incrementally over T sessions 1, .,.... ,.). requiring the model to learn continu- ously in an online fashion. Crucially, the unlabeled data con- tains both novel categories C, and instances from previously seen categories, such that C. C + C.. This assumption closely mirrors real-world scenarios, where streams of data often mix known and novel categories.
[ {Se
To simplify the analysis, we assume that each session in- troduces a fixed and identical number of new categories C This assumption does not limit the generality of our method, as it can be seamlessly extended handle varying numbers of new categories. When the number of new categories is unknown, existing approaches (Ronen et al., 2022; Zhao et al., 2023; Cendra et al., 2024) can estimate it effectively. We implemented an extension for unknown new categories counts based on the Silhouette Score; please refer to the Appendix D.1 for experimental details.
Self-Supervise Learning based Feature Extractor: To avoid feature shift, many studies (Cha al., 2021; Han al., 2022; Zhang al., 2022; Ma et al., 2024) have introduced self-supervised learning approaches. These methods learn aabel-independent features from the data through techniques such as data augmentation, thereby mitigating bias. For example, by optimizing a contrastive loss:
1 Lcon = | log
exp((h(x,)) (M(x)) /T)
The primary goal of C-GCD is to maximize classification accuracy across all sessions while maintaining or improving performance on previously learned categories. Formally, let the classification model be denoted as F with parameter O. Given an input xX" the predicted output is ÿ = F(x";: 9). The classification loss L measures the discrepancy between the predicted label and the true label. An intuitive solu- tion is to constrain the loss associated with learning new classes so that it does not degrade the loss on previously learned classes, as exemplified by Gradient Episodic Mem- ory (GEM) (Lopez- Paz & Ranzato, 2017). At session S,, the optimization objective of GEM is to minimize the clas- sification loss over the dataset D', as follows:
(2)
D Lifs
(x)(()()))) (h(x,)) /T)
here, h(.) denotes a label-agnostic feature extraction func- tion (R. - RRA).. With the rapid development of Vision Transformers (ViT'), numerous self-supervised pre-trained models, such as DINO (Caron et al., 2021), have been in- tegrated into continual learning frameworks, effectively re- ducing feature-level label bias.
Previous methods (Zhang et al., 2022: Ma et al., 2024), while leveraging pre-trained models, continued to use self- supervised loss during continual learning and incorporated cross-entropy loss with pseudo-labels as supervision sig- nals. This approach introduces the following challenges: (1) Feature drift remains unresolved; (2) Errors introduced by pseudo-labels exacerbate the drift; (3) The training process becomes more burdensome, with redundant forward and backward passes consuming significant training time (Shi et al., 2024).
O¹ = argmin Σ 2(F(x(;)))
BEH
X!ED!
s.L.
Σ X
C(F(x; 0'))
Σ L(F(x; 11-1))
X
1 -=(,,... T}
Offline Fine- Tine-Tunigg To mitigate feature drift, we per- form fine-tuning exclusively during the offline stages and
3

---

Learning and Forgetting from Bayesian Perspective
keep the backbone network parameters frozen throughout all sessions, including offline and online classification learn- ing. This decoupling of fine-tuning and continual learning eliminates the burden of extensive backbone training and preserves the stability of the feature distribution.
quadratic discriminant (Goswami et al., 2023). While mitigating label bias, this method provides inherent ro- bustness-outliers yield low likelihoods, while 2 down- weights noisy features.
3.2.2. CHALLENGE 2: LABEL ERRORS
is anticipated that incorporating correct labels to fine-tune the backbone network can enhance feature discriminability. leading to improved classification performance. Fortunately, the labeled data available from the offline phase be
In the C-GCD scenario, where data is unlabeled. novelty learning relies on pseudo-labels generated by algorithms such as clustering. However, this process inevitably intro- duces substantial label errors (misclassifications)) The error originates from two primary sources: (1) interference from data belonging to known classes, which disrupts the gen- eration of accurate pseudo-labels for new classes, and (2) inherent errors in the clustering algorithm.
leveraged for this purpose. Combined with the contrastive and cross-entropy loss, the fine-tuning loss is expressed as:
f = (1 Aie + ALcon
(3)
Generative Classifier: To mitigate label bias caused by class imbalance, we adopt generative classifier that di- rectly models class-conditional feature distributions, which involves modeling P(x:lus) instead of learning (P(s: (x:)- A representative example of this paradigm is prototypical learning (Snell et al., 2017), which learns a prototype for each class by estimating the class mean C E R Build- ing on a predefined distance function d == RM x R + [0, +oo), this approach employs the nearest class mean (NCM) method to estimate the likelihood:
Distinguishing between New and Old Categories. Effec- tively separating unlabeled samples into new and known classes is crucial to minimizing interference. While likelihood-based methods often struggle with this distine- tion, distance- based approaches (Zhang et al., 2022) classify samples by applying a distance threshold. However, these methods rely heavily on arbitrary cutoffs, which can signifi- cantly hinder generalizability.
Re-Labeling. To address this, we propose self-correcting re-labeling mechanism that operates in two stages: (I) Pseudo-Label Initialization: Unlabeled data is clustered to assign provisional labels, forming an initial estimate of the new-class distribution NV44, Σb). (2) Classifier Merg- ing and Refinement: The latent variables of old classes and the pseudo-labeled distribution are merged into a unified MVN-NCM classifier, which then refines label assignments for the unlabeled data.
exp(-d(h(xx).ox)
PP(y = k),,) = Exeep( d(((x))),,,
(4)
However, this method suffers the curse of dimensionality. As data distribution becomes increasingly sparse in high- dimensional spaces, Euclidean distances tend to concentrate, rendering classification failure.
MVN-based NCM. To alleviate the issue of classification failure in high dimensions, kernel functions can be em- ployed to collapse the high-dimensional hypersphere. There- fore, extend NCM by modeling each class as a multi- variate normal distribution (MVN), denoted as N(ux, Ex), estimated via:
The yseudo-labeld assigned by the clustering algorithm in- evitably contain a substantial amount of noise, causing the fitted distribution N N( Σxe to deviate significantly from the old classes' distributions. Consequently, data originally belonging to old classes remains naturally closer to the old distribution, making it more likely to be reassigned to its correct label (old). At the same time, since the fitted distribu- tion is closer to the true new-class distribution N N(rg E}), new-class samples tend to align more closely with the fitted distribution than with old classes' distributions, preserving their pscudo labels.
Σ h(x;)
1 |
x.ED
(5)
Σg | X.ED,
Σ (M(x;) - (h(x,) - px)
This captures inter-feature correlations critical in high- dimensional spaces. The distance metric then generalizes to the Mahalanobis similarity:
Noise-Robuse Distribution Fitting. While re-labeling helps reduce known class noise, clustering errors inevitably prop- agate into the estimated distributions when using statis- tical approach, as shown in Equation (5). As a result, a noise-robust, learnable estimation method is necessary to effectively fit P(x). However, in most cases, the marginal distribution P(() is a non-analytic integral and does not have a flosed-form solution.
d(M(x), = exp(- 1 ( (M) TT. (b(x))
(Mx)
14)))
(6)
which adapts to the data's covariance structure, address- ing the curse of dimensionality. Under Gaussian class- conditional assumptions, MVN-NCM classifier is Bayes-
optimal, with the Mahalanobis term in d(-) acting as the
Variational Bayes MVN. To approximate the marginal
4

---

Learning and Forgetting from Bayesian Perspective
re-labeling
online
X; row inpor
Algorithon 1: VB-OUDD
labled dalasel
unlabeiec datasel
y; target
Function (KI R.3 while not coustrye
Si peeaaoo lahel
(.
9; re-labeling h(): featurres
Alpul Slu] + (A... VL,I)K))
early stopping
calagary uhattusken
backbone
backbone
ir x and (e(()) 2 onggdel)))) then lreak 7/ oftlino sesalon
Fine-tuning
N(K)
early stop
* for (0K.,30)] + do
variational
Glustering
variational inference
classifier To
2/ rine-tuning
421 + v(2 A.C.. Al (K.))
kana penado peauso custutirr
11 A.E 5x2((/2)))
early stop
deil
// online nesstion &
N(u, 2,}
Nijk, >
inference
re-labeling
// sw-labwling
15. 6i t- clustecin((iix)))
Nip
Niga,
R. 2 SVI(M(s))
knOwN categery
Ni D
NoK. 2
A. SV)(M)a)))8.
Figure 1. The overall architccture of VB-CGCD. Offline Stage: Labeled data is used fine-tune the backhone network. After fine-tuning, features are extracted, and variational inference is performed to obtain and store the class distribution. Online Stage: Unlabeled data is first processed by the feature network extract features, which then clustered assign pseudo-lahels. These pseudo-labels are combined with first round of variational inference to obtain an approximate distribution, then combined with the stored class distribution from the previous session re-labeling.
distribution of x, we propose the Variational Bayes-based Multivariate Normal distribution (VB-MVN) to estimate the posterior distribution. We treat the means and covariances in posterior distribution N( Σ&) latent variable and approximate the posterior pa (2|xx) with a variational dis- tribution ge (zlx). This yields the Evidence Lower Bound (ELBO) (Kingma& & Welling, 2014):
However, as learning progresses and the accuracy of the new class continues to improve, the accuracy of the old classes begins to experience a slight decline (P2). Beyond certain point, the continued improvement of new class accuracy comes at cost: the overall accuracy progressively decreases, marking the onset of catastrophic forgetting (P3), Restricting the decline in old class accuracy to accommo- date the learning of new classes may seem like an intuitive solution. However, as depicted by the black dashed line in Figure 2(a), such approaches often lead suboptimal performance.
CELBO- = Pn (x|=)] Dx (xe((xx))) (=))
(7)
where (=) initialized to NV II) acts as a conjugate prior,
ensuring closed-form updates.
Covariance Impacts Forgetting. As variational inference it- erates on, the posterior variance Σ, monotonically increases (Figure 2(b)), diverging from I and eventually exceeding the variance of older classes. This divergence arises because ELBO naturally penalizes over-regularization, allowing Eg to expand until it sufficiently explains the new data's co- variance. From the perspective of the NCM classifier, when assuming the covariance matrix is diagonal, the distance function can be expressed as:
We optimize ELBO via stochastic variational inference (SVI) (Hoffman et al., 2012), which provides two critical advantages: (I) Outlier Suppression: Mislabeled samples exhibit low likelihood, contributing minimally to the recon- struction term. Simultaneously, their deviation from the dis- tribution of incurs a strong Kullback-Leibler (KI.) penalty, effectively downweighting their influence. (2) Scalability: SVI's stochastic gradients enable efficient optimization large-scale data, avoiding the cost of full-batch inference.
M (x] - -12))²
d(h(xs),
(8)
(o
3.2.3. CHALLENGE 3: TFA BETWEEN LEARNING AND FORGETTING.
The distributional difference between new and old classes can be quantified using the Bhattacharyya distance:
Suppose the old model captures the distributions [N(,, Σ.) i l....,k 1} of k 1 known classes. When learning new category cu. the prior set to (V 1), zero-mean isotropic Gaussian, to avoid premature bias toward existing classes while ensuring numerical stability during early training phases. By design, the test dataset initially exhibits stronger similarity (Equation 6) to the dis- tributions of known classes than to the new class, preserving the old model's accuracy (Figure 2(a), P1).
Σ1 + E2
d P2) = (u2
- (42--11)
(T)
(#2
111
2
+ ス/ In det En
1
(9)
Vdet (En det(
The three stages depicted in Figure 2(a) can be explained via covariance follows:
5

---

Learning and Forgetting from Bayesian Perspective
sight: optimal classification performance requires balancing the covariance scales of old and new classes. According to Equation (9), propose a regularization term to align covariance determinants by penalizing deviations from the old classes' average covariance:
80
60
40
P1
Testset old
20
k-1 Lett = (log det(2x) - llg k Σ det(2,)) (10)
P2
all
200
400
600
However, naively optimizing alongside ELBO fails in prac- tice due to: (1) ELBO's inherent bias: The variational objec- tive prioritizes data likelihood over regularization, leaving it under-optimized. (2) Data asymmetry: In the online phase, the number of samples and categories differs significantly from the offiin phase, resulting in complex hyperparameter tuning. Instead, we propose an early stopping rule that halts training when new-class covariances align with old ones. Specifically, monitor the ratio:
Learning Steps
(a) Accuracy
Testset
o
30
new
all
10
- 1) det R = log Σ det(E:)
(11)
200
400
600
Learning Steps
terminating training when R. < (empirically, e - 0.01),
effectively minimizing Leet. When Σ, approach to E we have: (1) the Bhattacharyya distance is minimized, yield- ing the most compact and separable decision boundary; (2) feature weighting balances stability (old classes) and plas- ticity (new class), avoiding bias toward either. As illustrated in Figure 2(b), the intersection of the covariances marks the point where the overall accuracy reaches its peak.
(b) Determinants of covs
Figure 2. (a) The evolution of accuracy for old classes, new classes, and overall during the learning process online session. (b) The variation in the average determinants of the covariance.
P1 (Stability): The new-class accuracy improves while old-class accuracy remains intact, as the initial prior mini- mally perturbs existing distributions.
P2 (Critical Transition): As Σx expands, the new class's Mahalanobis distance (Equation (8)) becomes increas- ingly weighted by features' variance, which induces over- lap between old and new classes, causing slight decline in old-class accuracy.
3.3. Variational Bayes C-GCD
As outlined in Table 1, we propose method called VB- CGCD (Variational Bayes C-GCD), addressing label bias, pseudo-label errors, and the Marning-forgettion tradeoff through these core components: (1) Self-supervised learning based offline fine-tuning. (2) Self-corrective re- re-labeling. (3) Variational Bayesian distribution estimation and nearest class mean classifier. (4) Covariance-driver early stopping.
P3 (Forgetting): When Σg exceeds the variance of old classes, d grows asymmetrically, skewing the decision boundary toward the new class. Old-class accuracy de- grades irreversibly, despite new-class accuracy stabilizing.
The root cause of forgetting lies in the covariance mismatch. According to Equation (8), the distance metric d(-) applies soft feature selection by weighting dimensions inversely by of. As covariance grows, the method increasingly ignores high-variance (noisy) features the new class, effectively narrowing its focus to stable, discriminative features. While this improves new-class accuracy, also: (1) distorts the Mahalanobis space for old classes, whose fixed Σ1.k- no longer align with the new feature weighting scheme; (2) diases samples toward the new class, when Σn >> Σ1.--1, the second term in Equation (9) dominates, inflating dp and fragmenting the decision boundary.
Table Summary the components VB-CGCD.
Label Bias
Label Errors
Trade-Off
Offline Fine-Tuning
Re- Re-Labeling VB-MVN + NCM
X
/
/
/
Early-Stopping
X
X
Figure 1 iillustrates VB-CGCD's 's two-phase design: Offline Phase: (1) Fine-tune the backbone network on initial labeled data; (2) Fit labeled class-conditional distributions via SVI.
Online Phase: (I) Assign pseudo-labels to unlabeled data through clustering, then refine them via re-labeling. (2) Fit distributions via SVI, constrained by covariance early stopping.
Early Stopping. To mitigate catastrophic forgetting while maximizing new-class plasticity, we formalize a critical in-
6

---

Learning and Forgetting from Bayesian Perspective
Table 2. Performance comparison on Cifar100 (C100),
ImageNet100 (IN100), TinylmageNet (Tiny), and CUR200 (CUB).
Si
S
S2
Sa
S₁
S
Datasets
Methods
All
All
Old
New
All
Old
New
Old
New
All
Old
New
All
Old
New
GM
90.36
76.58
79.80
60.50
71.10
74.52
50.60
63.51
68.
31.00
59.74
62.51
37.60
54.11
54.74
48.40
MetaGCD
90.82
76.12
83.60
38.70
69.40
72.82
48.90
61,95
65.76
35.30
58.22
61.21
34.30
55.78
58.47
31.60
C100
PromptCCD
85,20
63.71
65.03
57.13
61.80
62.86
55.46
60.14
61.81
53.54
52.65
52.89
50.72
55.53
55.64
54.54
Happy 90.10
83.62
85.08
76.30
75.83
80.85
45.70
70.03
73.50
45.70
64.70
65.06
61.80
60:77
62.02
49.50
VB-CGCD
91,68
88,40
89.46
83.10
86.0
87.06
79,60
83,61
84.75
75.60
82.72
82.85
81,70
81.23
81.84
75.70
GM
85,86
76.42
82,40
46,50
68.87
73.82
39.20
58.68
63.43
25.40
52.86
57.21
18.10
46.90
50.62
13.40
MetaGCD
84,20
60.88
64.90
40.80
57.20
61.03
34.20
54.36
57.19
34.60
50.83
53.59
28.80
48.14
50.16
30.00
Tiny
PrompiCCD
79.23
64, 10
64,72
61.02
61.43
62.07
57.58
56.39
57.08
51.53
$5.38
55.52
54.27
50,74
50. 30.82
50.0
Happy
85.76
77.80
82.68
53,40
72.26
76.02
49.70
65.29
70.06
31.90
57.79
61.40
28.90
54.93
56.08
44.60
VB-CGCD
88.32
85.10
86.44
78.40
82.40
84.15
71.90
79.35
81.38
65.10
76.72
78.43
63.0
74.52
75.88
62.20
GM
96.20
89.53
95.04
62.00
82.34
86.93
54,80
77.97
79.17
72.80
74.65
58.00
71.08
71.76
65.00
MetaGCD
95.96
75.27
78.20
60.60
73.79
75.93
54,90
69.35
72:20
49.40
67.22
70.10 10
44.20
66.68
69.31
43.00
IN100
PrompiCCD
91.18
82.11
82.36
80.84
80.03
80.36
78.02
77.78
78.03
76.05
67.37
67.54
65.99
63.97
64.08
62.95
Happy
96.20
91.20
95.36
70.40
87.83
90.83
69.80
85,22
86.40
77.00
81.93
83.00
73.40
78.58
79.11
73.80
VB-CGCD
94.32
93.13
93.16
93.0
90.45
91.43
84.60
90.10
90.0
90.8
838.97
89.30
86.40
87,32
88.08
80.40
GM
90.26
76.17
80.23
56.51
67.91
73.38
34.58
61.12
66.53
23.00
55.90
57.49
43.38
51.96
54.40
30.10
MetaGCD
89,20
67.08
70,21
51.92
60.77
62.39
50.86
57.53
59.33
37.78
51.90
52.22
49,40
49.60
49.96
46.38
CUB
PromptCCD
82.10
63.21
63,91
59.71
51.0
52.17
43,98
51.80
52.57
41.43
46.21
46.86
41.61
50.95
51.16
49,06
Happy
86.0
77.83
82.44
55.37
68.19
76.24
21.51
63.44
67.47
34.63
58.39
61.90
31.14
54.37
56.70
33.67
VB-CGCD
85.72
78.46
82,92
55.45
74.60
76.72
62.01
71.84
73.38
60.83
68.60
70.91
49.64
66.44
67.76
54.38
4. Experiments
newly introduced classes in the current session ('New"), the accuracy of all previously encountered classes ('Old'), and the accuracy all classes currently ('All'). (2) Forgetting rate My. the difference in the accuracy of labeled classes between the model in the offline phase and the model after completing all online sessions.
We evaluated VB-CGCD on a variety of datasets. Addi- tionally, we conducted comparisons with several baseline approaches to assess its performance.
(3) Novelty learning rate Ma, the average accuracy of new classes in each session.
4.1. Experimental Setup
Datasets. In line with mainstream C-GCD works, we uti- lized four CIlssttt-CFARRRI
More details of the data configuration, metrics, implementa- tion, and hyperparameters, can be found in the Appendix.
(Krizhevsky, 2009), Tiny- ImageNet (Le& & Yang. 2015), ImageNet100 (Russakovsky et al., 2015), and CUB200 (Wah et al., 2011) (011)eeach repre- senting distinct characteristics. To ensure fair comparisons, we followed the experimental setup described in (Ma et al., 2024):
4.2. Results
A Accuracy The overall results are shown in Table 2. When the number of online sessions is set to 5, VB-CGCD con- sistently outperforms the baselines in overall accuracy, as well as old-class and new-class accuracy. Throughout the continual learning process, VB-CGCD demonstrates strong resistance to forgetting while significantly enhancing per- formance on novelty learning. As a result, by the end of all continual learning sessions, the final overall accuracies ('All' of S3)) achieved by VB-CGCD exceed those of the baselines across four datasets, with an average improvement of 15.21%.
Training data: The labeled data consists of 50% of the classes, with 80% of the training samples per class. The unlabeled data comprises the remaining 50% of the classes, evenly distributed across all online sessions, with 80% of the training samples per class. Additionally, 20% of the data from all previously encountered classes is carried forward into each session.
Testing data: In each session, the test set includes samples from all classes encountered up to that session.
Forgetting and Learning. To further investigate the factors influencing performance, we analyzed the forgetting rate Mj and novelty learning rate Ma. For comparison, we constructed a performance bound by dividing the dataset into two parts and performing supervised learning on cach part separately. This setup eliminates the effects of label error (since all samples are fully labeled) and label bias (which is restricted to only two stages). From this analysis,
Baselines. We compared VB-CGCD with several state-of- the-art approaches. including MetaGCD (Wu et al., 2023), GM (Zhang et al., 2022), PrompiCCD (Cendra et al., 2024). Happy (Ma et al., 2024).
Evaluation Metrics We employed the following metrics to evaluate the performance of VB-CGCD: (1) the accuracy of
7

---

Learning and Forgetting from Bayesian Perspective
high accuracy. For datasets with sufficient samples per class, such C100, Tiny, and IN100, the accuracy reached approximately 95%. However, on the CUB dataset, which contains limited number of samples, the mixing of old
Table Forgetting rate and novelty learning rate ( indicates that the higher the metric. the better, and vice versa).
C100
Tiny
IN100
CUB
Methods
M
Ma
M 4
Ma1
M 4
Ma 1
M, V
Ma
and new classes posed greater challenges, resulting in a lower accuracy of around 65%. This limitation provides a
PrompiCCD
21.66
54,28
23,46
54,88
18.76
72.77
19.10
47.15
Happy
12.08
55.80
9.16
41,70
10.13
72.88
3.22
35.26
vB-cocc
6.94
79.14
5.72
68.12
5.68
87.04
6.72
56.46
partial explanation for the relatively lower overall accuracy
UpperBound
77.04
3.16
89.04
5.13
79.65
observed on CUB.
Ablation Study. To assess the contribution of each mod- ule, we conducted ablation experiments focusing four key components: SVI, fine-tuning, early stopping, and re- labeling. The experimental configurations, outlined in Ta- ble 4, include four distinct setups. Notably, in the absence of SVI, the mean, and covariance are estimated directly us- ing Equation (5) through point estimation a non-learning- based approach. Consequently, early stopping cannot be applied in this configuration.
we infer the following: (1) Surpassing the upper bound of Ma will likely result in excessive forgetting. (2) Falling be- low the lower bound of M may lead to inadequate novelty learning.
As shown in Table 3, VB-CGCD achieves performance closer to the bounds for both the forgetting rate and novelty learning rate. On IN100, VB-CGCD nearly reaches the bounds, likely due to the large dataset size (1000 samples per class), which facilitates learning a more accurate distri- bution. contrast, on CUB, where each class contains only 30 samples, the limited data exacerbates the effects of label noise, resulting in lower novelty learning performance. Ad- ditionally, it is noteworthy that Happy falls below the lower bound for forgetting, highlighting its insufficient novelty learning capabilities.
Table 4. Sctup of the ahlation study.
SVI Fine-Tuning
Early-Stop
Re-Labeling
w/o SVI w/o fine-tuning w/o carly-stopping w/o re-labeling w/ all
X
x
V
X
/
The ablation results, presented in Table 5, reveal that SVI has the most significant impact accuracy. Even during the supervised So session, performance without SVI is sub- stantially worse than with it. This disparity arises because the number of samples per class is smaller than the number of features, resulting in a non-full-rank covariance matrix. This issue prevents the computation of the distance function, making classification ineffective. To address this, dimen- sionality reduction was applied to the datasets during the experiments, ensuring the results remained meaningful.
CIFAR-100
Tiny ImageNet
Number of Samples
15000
10000
5000
0
ImageNet100
CUB-200
Number of Samplts
15000
Type
In summary, the removal of any module results in perfor- mance degradation, with each module contributing uniquely to the overall system. SVI and fine-tuning enhance the cor- relation between features, aiding both the mitigation of for- getting and the learning of new classes, while early stopping and re-labeling focus achieving a balance between new- class learning and forgetting. Early stopping prevents the covariance values from exceeding those of known classes, thereby avoiding overfitting to new classes, which would otherwise reduce old-class accuracy and ultimately lower overall accuracy. Without re-labeling, however, significant amounts of noisy data are introduced, reducing the average covariance during training and delaying the early stopping mechanism. Consequently, in experiments without these two methods, the accuracy for new classes is higher, but overall accuracy is diminished.
Novel
10000
All
5000
2
5
2
5
Scssion
Scssioo
Figure 3. Comparison of the identified number of samples belong- ing to new classes (denoted as 'Novel') with the total number of samples (denoted as 'All". The red horizontal dashed line repre- sents the ground truth of 'Novel'.
Novelty Detection. Additionally, we tracked the number of samples identified as belonging to new classes in each session, as shown in Figure 3. The re-labeling process effectively distinguished new classes from old ones with
Among the modules, the contribution of re-labeling to over- all ('All') accuracy is relatively minor, yielding approxi-
8

---

Learning and Forgetting from Bayesian Perspective
achieves 6.8% higher accuracy than Happy in the super- vised session (Sp), resulting in net improvement of 2% in My. For CUB, the trend remains consistent with earlier results: although Happy strictly prevents forgetting, this comes at the expense of new-class accuracy, ultimately lead- ing to lower final accuracy. We also compared VB-CGCD to a baseline with 50% labeled classes ('b50"). Despite reduced labeled categories, VB-CGCD showed only an av- erage 9.56% accuracy drop, far lower than the 22. 87% drop in Happy,, demonstrating its strong generalizability.
Table 5. Ablation study results across multiple datasets.
Datasets
Methods
A
All(S.)
My 4 Ma t
w/o SVI
88.92
68.90
17.10
72.24
w/o fine-tuning
88.54
73.48
8.18
68.0
C100
w/o early-stopping
91.68
77.71
15.8
81.62
w/o re-labeling
91.68
80.20
9.58
80.74
91.68
81.23
6,94
79.14
w/o SVI
84.78
61.14
14.22
56,52
w/o fine-tuning
83,94
68.69
5.74
60.62
Tiny
w/o early-stopping
88.32
71.23
14.24
70.60
w/o re-labeling
88.32
73.67
8.48
70.38
w/ all
88.32
74.52
5.72
68.12
w/o SVI
92.56
79.90
8:45
80.32
5. Conclusion and Discussion
w/o fine-tuning
93.04
84.0
7.36
83.88
IN
w/o early-stopping
94.32
84,96
11.92
89.04
This paper presents VB-CGCD, a Bayesian framework for C-GCD that models feature distributions via variational inference and mitigates forgetting through covariance align- ment. The study investigates the underlying causes of for- getting within this framework, identifies the critical role of covariance, and proposes an early stopping rule that halts training at optimal stability balance. Experimental across diverse benchmarks validate that VB-CGCD signifi- cantly outperforms existing baselines, demonstrating robust performance and excellent scalability.
w/o re-labeling
94.32
86.54
7.16
88.28
w/ all
94.32
87,32
5.68
87.04
w/o SVI
55.34
27.06
40.08
49.30
w/o fine-tuning
80.16
51.10
15.12
40.99
CUB
w/o carly- stoppig
85.72
56.43
29.22
60.77
w/o re-labeling
85.72
63.32
13.44
59.65
w/ all
85.72
66.44
6.72
56.46
Table 6. Performance of each session with 10-sessions setting.
Datasets
Methods
My 4
Ma
All(Su)
All(S)
h50-All(S))
C100
Happy
36,90
48.40
98.10
36.16
60.77
While VB-CGCD demonstrates significant potential, several challenges remain, presenting avenues for future research. (1) Its reliance on initial clustering quality can impact perfor- mance, particularly in high-dimensional spaces. Integrating deep clustering techniques or adaptive mixture models could enhance robustness and reduce sensitivity initialization. (2) Small-sample covariance estimation, seen in datasets like CUB, remains limitation. Investigating hierarchi- cal Bayesian priors or meta-learning strategies to improve covariance estimation under limited data conditions could further enhance generalization.
VB TB-CGCD
18.30
78.38
98,90
74.72
81.32
Happy
66.46
33.39
88,80
22.11
54.93
TINY
VB BBCCCC
7.40
67.68
92.70
66.30
74.52
Happy
10.0
62.15
91.40
59.12
75.58
IN100
VB-CGCD
14.8
85.76
98,20
82.88
87.32
Happy VB-CGCD
21.07
42.87
94.17
36.
54.37
CUB
28.54
49.54
94.02
47.46
66.44
mately 1% improvement. This is because the SVI method effectively mitigates the influence of outliers, ensuring that moderate levels of label noise do not significantly impair the learning process. However, in scenarios with smaller datasets and higher noise ratios-such as CUB-tt ad- vantages of re-labeling become more evident, delivering an improvement of around 3%.
A Acknowledgements
This work was supported by the Engineering and Physi- cal Sciences Research Council (EPSRC) grant, MultiTask- ing and Continual Learning for Audio Sensing Tasks on Resource-Constrained Platforms [EP/X01200X/1].
Fewer Labeled Categories Scenario. Previous studies often assume that the majority of old classes labeled. However, given the ability of VB-CGCD to mitigate label noise and bias, it can be extended to handle more extreme scenarios with a larger proportion of unlabeled classes. To evaluate this capability, we designed an extreme case where 90% of the classes are unlabeled, and distributed across 9 sessions, with only 10% of the classes having partially labeled data (80% of their samples).
Impact Statement
By leveraging variational inference and covariance con- straints, this method addresses catastrophic forgetting and label noise in online learning scenarios. Its robustness, noise resistance, and broad applicability make it promising solu- tion for more general class-incremental learning tasks.
The results, shown in Table 6, highlight the advantages of our method over the current state-of-the-art, Happy, in both novelty learning and final accuracy. Notably, on IN100, while Happy demonstrates 4.8% less forgetting compared to VB-CGCD, this is offset by the fact that VB-CGCD
9

---

Learning and Forgetting from Bayesian Perspective
References
Processing Systems, volume 36, pp. 6382-6595. Curran Associates, Inc., 2023.
Belouadah, E. and Popescu, A. 112m: Class incremental learning with dual memory. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 583- 592, 2019. doi: 10. 10.110CCV22000000000000
Han, K., Rebuffi, S.-A., Ehrhardt, S., Vedaldi, A., and Zis- serman, A. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44( 1411))67777- 6781, 2022. doi: 10. 11110//TPAML2222.309999999
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs. http: //github. com/ jax- ml/ jax, 2018.
Hoffman, M. D., Blei, D. M., Wang, C., and Pais- ley, J. W. Stochastic variational inference. J. Mach. Learn. Res., 14:1303-1347, 2012. URL httpt::/pp:/pppt /api. semanticscholar.coom hwt.:hho........roooooo.oo.aaaa.
Caron, M., Touvron, H.. Misra, L. Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in superiiee vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Hu, Z., Li, Y., Lyu, J., Gao, D., and Vasconcelos, N. Dense network expansion for class incremental learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11858-11867, 2023, doi: 10. 1110//CVPR52229.2233...
Cendra, F. J., Zhao, B., and Han, K. Promptccd: Learn- ing gaussian mixture prompt pool for continual category discovery. In European Conference on Computer Vision 2024.
Kapoor, S., Karaletsos, T., and Bui, T. D. Variational auto-regressive gaussian processes for continual learn- ing. Meila, M. and Zhang, T. (eds.), Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5290-5300. PMLR, 18-24 Jul 2021. URL /tttap::://prreeeee http://ooeeeddeeee ml presss/ 1139/kapoor211 html.
Cha, H., Lee, J., and Shin, J. Co21: Contrastive continual learning. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9496-9505, 2021. doi: 10.1 00.1099/CCV44992222221 202000009
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org. 2020.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd Interna- tional Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Douillard, A., Ramé, A., Couairon, G., and Cord, M. Dytox: Transformers for continual learning with dynamic token expansion. In 2022 JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9275-9285, 2022. doi: 10. 10.10099CPP555288.220000 w0t..0000.....000.0..
Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ra- malho, T., Crabska-Barwinskk A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catas- trophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2016. URL httpp::/aaat semant. cscholar. hegg/oorpuID::47000
Feng, T., Wang, M., and Yuan, H. Overcoming catas- trophic forgetting in incremental object detection via elas- tic response distillation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9417-9426, 2022. doi: 10.1 101100//22522268.2220 00921.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, 2009.
Geng, C., Huang, S.-J., and Chen, S. Recent advances in open set recognition: A survey. IEEE Transactions on Patlern Analysis and Machine Intelligence, 43(10): 3614-3631, 2021. doi: 10.1109/TAAM..222..2299
Le, Y. and Yang. X. S. Tiny imagenet visual recog- nition challenge. 2015. URL https://api: semanticscholl cttpaaroor/ccooo.D:::::6)1
Goswami, D., Liu, Y., Twardowski, B. o., and van de Wei- jer, J. Fecam: Exploiting the heterogeneity of class dis- tributions in exemplar-fre continual learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, (eds.), Advances in Neural Information
Li, W., Fan, Z.,, Huo, J., and Gao, Y. Modeling inter-class and intra-class constraints in novel class discovery. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3449-3454 2023. doi: 10.1009/CVPR55222.2023.. 201.322777..223003333
10

---

Learning and Forgetting from Bayesian Perspective
Li, Z.,. Zhao, L., Zhang, Z., Zhang, H., Liu, D., Liu, T., and Metaxas, D. N. Steering prototypes with prompt- tuning for rehearsal-free continual learning. In 2024 IEEE/CVF Winter Conference on Applications of Com- puter Vision (WACV), pp. 2511-2521, 2024. doi: 10. W19/WACV57701.224444
Ronen, M., Finder, S. E., and Freifeld, O. Deepdpm: Deep clustering with an unknown number of clus- ters. 2022 IEEE/CVF Conference
Computer Vi- sion and Pattern Recognition (CVPR), pp. 9851-9860, 2022. URL https: http://ppp.............e. htg//oorpaiD:227762222
semant ieanniiccee cscholar.
Liu, W., Qiu, Z., Feng, Y., Xiu, Y., Xue, Y., Yu, L., Feng, H., Liu, Z., Heo, J., Peng. S., Wen, Y., Black, M. J., Parameter-efficient orthog- onal finetuning via butterfly factorization, In The Twelfth International Conference on Leurning Representations, 2024. URL https: httpp//ppptppepptpt/tpppppoeepet htwp:/eevveeeeee/oreeeeee met/fform?? id=7NzgkEdGyy
Rudner, T. G. Bickford Smith, F., Feng, Q., Teh, Y. W., and Gal, Y. Continual learning via sequential function- space variational inference. In Chaudhuri, K., Jegelka, S., Song, L.,, Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18871-18881 PMLR, 17-23 Jul 2022. URL http//pooceeennge////e mlr.
Weller, A., and Schölkopf,
Loo, N., Swaroop, S., and Turner, R. E. Generalized variational continual learning. In International Confer- ence on Learning Representations, 2021. URL https: //oppnreviex..o////////////e/emp net forum? IM AfFhna9.
press vvl 62 rudner22a .tml..
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):11-252, 2015. doi: 10.10077s11233-015-000
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6470-6479, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Rypesc, G., Marczak, D., Cygert, S., Trzcinski, T., and Twardowski, B. Category adaptation meets projected distillation in generalized continual category discovery. In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), Computer Vi- sion ECCV 2024 18th European Conference, Mi- lan, Italy, September 29-October 4, 2024, Proceedings, Part XI, volume 15069 of Lecture Notes in Computer Science, pp. 320-337. Springer, 2024. doi: 10.1007/ 978-3-031-73247--1 19. URL https: //::://doo //diiooo.ggg//// 10. 1001000//00 1-1--77----111-7-------11-111111
Ma, S., Zhu, F., Zhong, Z., Liu, W., Zhang. X.-Y., and Liu, C.-L. Happy: A debiased learning framework for contin- ual generalized category discovery. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL hitps: / http//ppeeeeeeeeeeeeeeeee forum?id=huuCzi:0.
Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. Variational continual learning. In International Confer- ence on Learning Representations, 2018 URL https: //openreview net. / forum? wiwimmmiddddooe 1d=BkQqq0gRb.
Saha, G., Garg, I., and Roy, K. Gradient projection mem- ory for continual learning. In International Confer- ence on Learning Representations, 2021. URL https: / / //ppeeeveeeww.. www.reeiiee...... net/ www.mmiiddooooooor...rrrrriiroie 3=0 3 DRCNC2.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, 7.,, Lin, Z.,, Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. 2017.
Shi, J.-X., Wei, T., Zhou, Z., Shao, J.-J., Han, X.-Y., Y--F. Long-tail learning with foundation model: heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
Y., and Li,
Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representa- tion learning. 2017 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 5533-5542, 2016. URL https: / http:p/://app..smatiiii...oo arg/CorpusID. :00=pusSD:206596660 www.ntiisccoooo...oo
206596260.
Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S.. and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:s/proceedings..n hewprrrooceeeiin................ cc ft/p cc//appee_fflee//pppe///////// @bB467777661788122222222aaa
Rizve, M. N., Duarte, K., Rawat, Y. S., and Shah, M. In defense of pseudo-labeling: An uncertainty-awant pseudo-label selection framework for semi-supervised learning. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL ht tps::: //openreviens www.eeewwwwwweeeeeeeeee formm di DEEE--DNNSbiio
pdf.

---

Learning and Forgetting from Bayesian Perspective
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The Caltech-UCSD birds-200-2011 dataset. Tech- nical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
survey. IEEE Transactions on Pallern Analysis and Machine Intelligence, 46(12)998519955 2024b. doi: 10.1 10.1100/TPAM122044.443339
Zhu, F., Zhang, X.-Y., Wang, C., Yin, E,, and Liu, C.-L. Pro- totype augmentation and self-supervision for incremental learning. In 2021 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5867-5876. 2021. doi: 10.
Wang, L., Zhang, X., Su, H., and Zhu, J. A compre- hensive survey of continual learning: Theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(8):5362-553) 5383, 2024. doi: 10.1 (O90TPAMM.2222433673332
Wen, X., Zhao, B., and Qi, X. Parametric classification for generalized category discovery: A baseline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16590-16600, 2023.
Wu, Y., Chi, Z., Wang, Y., and Feng, S. Metaged: Learning to continually learn in generalized category discovery. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV), pp. 1655-1665, October 2023.
Ye, F. and Bors, A. G. Continual variational autoen- coder learning via online cooperative memorization. In Computer Vision ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part XXIII, pp. 531-549, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN 978-3-031-20049-6 doi: URL https://dol. org/10. 0
Zhang, X., Jiang, J., Feng, Y., Wu, Z.-F., Zhao, X., Wan, H., Tang, M., Jin, R., and Gao, Y. Grow and merge: A unified framework for continuous categories discovery. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. A.d.....dvvncce in Neural Informa- tion Processing Systems, volume 35, pp. 27455-27466 Curran Associates, Inc., 2022.
Zhao, B., Wen, X., and Han, K. Learning semi-supervised gaussian mixture models for generalized category dis- covery. In 2023 IEFF/CVF International Conference Computer Vision (ICCV), pp. 16577-16587, 2023, doi: 0L1109ICCV51107..222..111
2023.
Zhou, D.-.., Sun, H.-L., Ning, J., Ye, H.-.,, and Zhan, D.C.. Continual learning with pre- pre-trained models: A survey. In Larson, K. (ed.), Proceedings of the Thirty- Third International Joint Conference on Artificial Intel- ligence, IJCAI-24, pp. 1363-8371 International Joint Conferences on Artificial Intelligence Organization, 8 2024a. doi: 10.24963/ijcii 2024/924. 2249963//cai.2222299444,4
URL https: //doi. woo.//rggg.. htt/:/...449600 2493242933333 20244/24 Sur- vey Track.
Zhou, D.-.,, Wang, Q.- Qi, Z.HH,, Ye, H.J.., Zhan, D.-C., and Liu, Z. Class-incremental learning: A
12

---

Learning and Forgetting from Bayesian Perspective
A. Pseudocode
Algorithm 1: VB-CGCD
Input :steaming datasets D (D t= I,...,Ty Th VITB-16 h(-) Output: : posterior distribution (N(14, Σx) i= -ii==,.... 1,....(c)
Function SVI (X;, pi, Σ):
14: E, I
// initialize with identity matrix
while not converge do
Ps. Σ + (14, Ej) + a Veiibo x Pa: E,) // early stopping
5
if Σ * |] and det(())) 2 avg (eg(det(2)) (de((t) [)) then
break
end if end while
return (4, 23
17 + -
12 / offline session So, perform supervised learning +/
// fine-tuning
for [(x,,,))) D do
h(.)) h(.) a+ V[(1 X)Loo (x) ALoon (x:)
end for for
{(x,, y:) t Dj do MR, E, SVI (h(x;), 1T,
Aus + 14: E(yy]] t Σg end for
/* online session Sn, perform unsupervised learning */
Ci - mmar(j;))
23 Co C:
for S. Sr do
for (x,) D', do
Ú: + (lsteriing((xx):;.o., Ct)
end for // re-labeling
1 +1''' + for
(x,,0i]+ + (Di 1 Y) do
14 t- SVI (x(xs))) 17,
Fl0 + ur. Σ 0:] t E
end for
for
{x,} + D', do D: classi f (T( (U
end for
for
{x,, 8i} + (Di, Y) do
Ns. Σg t SVI (VVI(h(x)))))) Σ)
Aly t- p4: [(ir] + Σg
39
end for
C, f- C, + C₁
end for
43 return J7, 5 C-GCD differs from generalized category discovery (GCD) and novel category discovery (NCD) in the following two aspects:
13

---

Learning and Forgetting from Bayesian Perspective
Unlabeled data divided into multiple distinct online stages, where each stage S; cannot access data from previous stgges (Sn; St..... - This means that the model only use data from the current stage for learning. Unlabeled data and test sets for each online stage contain data from all previously encountered categories. This poses data.
challenge for the model, as it needs to distinguish between new and known categories while learning from the unlabeled
The objective of C-GCD to improve the classification accuracy for all categories. This means that the model should strive identify and learn classify new categories in each stage without compromising the accuracy of previously learned categories (Feng et al., 2022: et al., 2023; Nguyen al., 2018). The overall pseudocode for the proposed algorithm shown in Algorithm
B. Learning from Unlabeled Data with Variational Inference
Distribution
000
00
10
00
-15
2
6
8
11 Feature Dimension
9
10
12
13
14
15
16
17
18
19
20
Figure 4. Comparison of the feature distributions for 8 dimensions across two different classes. A longer distribution indicates a higher variance of the dimension.
As illustrated in the Figure 4, the variances of data features across different dimensions exhibit significant differences, reflecting their varying contributions to classification. Modeling these features with MVN effectively captures such characteristic information.
Table Accuracy comparison various classification methods, including class mean (CM), point estimation (PE), and variational inference (VI).
Datasets
CM
PE
VI
CIFAR-100
75.17
77.16
82.68
ImageNet- 100
86.64
88.0
90.18
ImageeteeNet
71.47
70.66
76.95
CUB-200
64.60
66.87
73.02
Variational inference (VI) optimizes the ELBO, effectively capturing feature distributions while mitigating the impact of outliers. We compared class modeling approaches based on statistical mean (CM), statistical distribution (PE), and variational inference. As shown Table and Figure 5, VI achieves superior classification accuracy, highlighting its effectiveness over traditional statistical methods.
C. Experiment Details
C.1. Dataset Details
We performed experiments on four datasets: CIFAR-100, ImageNet-100, TinylmageNer, and CUB-200. The specific configurations for each dataset are outlined in Table 8.
For comparison purposes, we conducted supervised learning on all datasets to determine the thcoretical upper bound (Table of model accuracy at the final session. Comparing this upper bound allows us assess the performance and error
14

---

Learning and Forgetting from Bayesian Perspective
Mrons
PR Coarianos
-0.1
0.2
VI
0,0000
170175
9.000
1J025
-30000
1.0300
-20000
Figure.5 5. A comparison of the mean and covariance derived from PE and VI. It shows that the covariance in PE is an order of magnitude larger than that in VI. This indicates that many features in PE are scaled down much smaller range.
Table 8. The data configuration for the scenario with 50% categories labeled and 5 continual sessions.
Labeled
Unlabeled
Datasets
categories
samples per category
novel categories
samples per novel samples per known
CIFAR-100
50
400
10
400
25
ImageNet- 100
50
1000
10
1000
60
Tiny ImageNet
20
400
CUB-200
100
25
20
25
margin of C-GCD methods.
Table Accuracy upperbound of supervised learning.
Datasets
Pre- Trained
Tune-Tuued
CIFAR-100
82.68
85.72
ImageNet-100
90.18
90.24
Tiny ImageNet
76.95
81.14
CUB-200
73.02
80.49
We further analyzed the upper bound of forgetting in supervised learning manner. demonstrated in Table 10, in supervised learning scenario, the accuracy of model trained the complete dataset ('AII(S,))) noticeably lower the old class test set compared phenomenon, which can be termed as "inevitable forgetting"."))
model trained exclusively on the old class data ('S,"). This decline is an inherent
C.2. Metries Details
Accuracy: The primary metric used assess the classification performance of the models. The accuracy metrics categorized as follows:
-'All: Accuracy on the test set that includes all categories encountered up to the current session S,, denoted as ACCan 1
[| Dis Pl 1 (b/ v. F(x;; O²)); Vi E (1,...,C&++.?
"Old": Accuracy on the test set comprising only the previously encountered classes, excluding the new classes introduced
in the current session S,, denoted as ACCatd | ElP - 1 y, F(x;: O¹)): )): E .t....CC-1k.
":New"; Accuracy the test set of the newly introduced categories specific to the current session, denoted ACCnew ([aast 1 (y: = J A F(x;; O²)); y (rt...,,,+mm...
P
Forgetting Rate: After completing all online learning phases, the forgetting rate defined as the difference between the final model's accuracy on the initially labeled dataset (Di) and the accuracy obtained during the initial offline supervised learning phase. This measures the degree of forgetting on the labeled dataset during the update process from Sp to Sr. denoted as:
15

---

Learning and Forgetting from Bayesian Perspective
Table 10. Comparison of the accuracy the So set between models trained exclusively data and models trained the entire dataset.
Datasets
Sg
A))(Sg)
CIFAR-
88,90
82.68
ImageNet-100
93.16
89.68
TinyImageNet
84.26
79.21
CUB-200
80.43
73.54
My |Di
Σ Pill Ei
[
- F(x;;00) - Lyy: = J - F(x;; 07))l: E (k.
Novelty Learning: The average learning performance of the new classes from Si to Sr, is evaluated by calculating the mean accuracy of the new classes across all sessions, denoted as: Ma = Ei=1 ACC new.
C.3. Hyperparameters and Hardware.
The proposed method was implemented using Pytorch (Paszke et al., 2017) and JAX (Bradbury et al., 2018). For the feature extractor, we used DINO (Caron et al., 2021) pre-trained ViT-B/16 model with weights from TorcHHub, Fine-tuning was conducted via the BOFT algorithm (Liu et al., 2024), updating only the MLP and output dense layers of the ViT. The model was optimized using AdamW optimizer with a learning rate 1e for 10 epochs.
To balance overall performance and runtime efficiency, we applied PCA reduce the dimensionality of the output features to 384 dimensions. We employed the standard k-means algorithm for clustering, maintaining consistency with other baselines. For classifier training, fixed learning rate of le was applied across all stages, with 1000 update steps conducted. All experiments were executed on RTX 8000 GPU with 48GB of memory. All random seeds are set to 0.
C.4. Impact of Hy perparameeeers
We conducted hyperparameter R sensitivity analysis C100 to evaluate the impact of early stopping the performance of our method. The results are presented in Table 11 13.
Table 11. Comparisons in overall accuracy as R. varies.
R
So
S
S2
S3
S4
3
85.55
82.20
79.38
77.74
75.61
2
86.85
83.30
80.20
78.41
76.13
88.40
85.90
83.41
82.57
81.11
88.40
86.0
83.61
82.72
81.23
-1
87.03
83.75
81.31
80.18
77.91
-2
87.01
83.71
81.23
79.94
77.71
Table 12. Comparisons in the accuracy of old classes varies.
R
S
S
S
S3
Sa
91.62
85.31
81.98
79.17
77.44
91.32
86.63
83.15
80.01
78.11
1
90.18
87.20
84.75
82.81
81.92
0
89.46
87.06
84.75
82.85
81.84
-1
87.34
84.20
81.91
79.57
77.91
-2
87.30
84.10
81.82
79.31
77.56
The experimental results indicate that the parameter R serves to balance the accuracy between new and old classes, with the overall accuracy being the best when R 0.
16

---

Learning and Forgetting from Bayesian Perspective
Table 13. Comparisons in the accuracy of new classes as R. varies.
R
So
S
S₂
Sg
S₁
55.20
63,50
61.20
66.30
59.10
64.50
63.30
59.50
65.60
58.30
79.50
78.10
74.80
80.70
73.80
0
83.10
79.60
75.60
81.70
75.70
-1
85.50
81.10
77.10
85.10
77.90
-2
85.60
81.40
77.10
85.0
79.0
C.5. Ablation Study
The following Table 14 presents the performance breakdown for each session in the ablation experiment.
Table 14. Comparison of the ablation study results across multiple datasets.
So
St
S2
Sa
&
S:
Methods
All
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
w/o SVI*
88.92
78.65
79.22
75.80
75.44
76.31
70.20
72.30
72,80
68. 08.80
71.33
70.62
77.0
68.90
68. 84
69.40
w/o fine-tuning
88.54
82.51
85.52
67.5
79.25
80.81
69.90
76.71
78.35
65.20
75.13
76.01
68.10
73.48
73.94
69.30
C100
win earlystop
91.66
87.02
87. 30
85.60
83.71
84.10
81.40
81.24
81.83
77.10
79.94
79.31
85.0
77.71
77.57
79.0
w/o re-labeling
91.68
30
89.32
83,20
85,63
86.40
81.0
83.16
83.89
78.10
82.18
82.04
83,30
80,20
80.43
78.10
89.46
83.
86.0
87.06
84.75
75.66
82.72
82.85
81.70
81.23
81.84
75.70
w/o SVI
84.78
75.03
77.26
63.90
70.47
72.31
59.40
66.86
68.72
$3.80
63.74
65.30
51.30
61.14
61.91
54.20
w/o fine-tuning
83.94
80,78
81.86
75, 40
77.62
79.85
64.30
74.22
76.61
57.50
71.15
73.51
52.30
68.69
70.36
53.60
Tiny
w/o carlystop
88.32
83,36
83.96
80.40
79.91
80.55
76.10
76.82
78. 15
67.50
73.76
74.95
64.30
71.23
71.95
64.70
who re-labeling
88.32
85.03
86. 22
79. 10
82.04
83,13
75.50
78.89
801.7
67.80
76.11
77.59
64,30
73.67
74.61
65.20
w/ all
88.32
85.10
86,44
78.40
$2.40
84.15
71.90
79.35
81,38
65.10
76.72
78.43
63.0
74.52
75.88
62.20
w/o SVI
92.56
89.23
88.88
91.0
86. 14
87.63
77.20
84.57
85.05
81.20
82.31
83.20
75.20
79.90
80.22
77.0
w/o fine-tuning
93.04
92.10
91.36
95.80
90.80
90.40
93.20
88.44
89.77
79.40
86.53
87.17
81.40
84.0
85.60
69.60
IN100
wo earlystop
94.32
91.18
91.08 08.08
95.60
88.31
88.76
85.60
88.27
87.48
93.80
86.80
86.65
88.0
84,96
85,26
82.20
w/o ro-labeling
94.32
93.10
92.96
93.80
90.23
91.23
84.20
89.90
89.46
93.0
88.38
87.4
86.54
86.93
83.0
94.32
93.13
93.16
93.0
90.45
91.43
84.66
90.10
90.9
90.8
88.99
89,30
86.40
87.32
88.08
80.40
w/o SVI'
55.34
28.16
23.01
54.75
24.77
22.53
38.16
27.27
22,30
62.76
26.76
24.84
42.42
27.06
24.73
48.42
w/o fine-tuning
80.16
68.77
74.18
40.84
61.43
65.88
34.92
57.81
58.82
55.59
53,50
56.12
32.40
51.10
52.18
41.22
CUB
w/o earlystop
85.72
68.91
70.46
60.91
63.46
63.54
63.03
61.31
60.57
66.60
58.11
58.61
54.04
56.43
56.12
59.29
w/o re-labeling
85.72
77.95
81.94
57.39
72.97
74.72
62.52
69.82
70,32
66.
65, .1
67. -644
53.17
63.32
63.80
58.95
85.72
78.46
82.92
55.45
74.60
76.72
62.01
71.84
73.38
60.83
68.60
70.91
49.64
66.44
67.76
54.38
We also conducted an ablation study on the benefit of introducing Mahalanobis distance, and the results obtained by replacing the Mahalanobis with the Euclidean distance are shown in Table 15.
Table 15. Ablation study results without Mahalanobis.
Datasets
Methods
All(So)t
All(Sa)t
My
Ma f
C100
wo Muhalanobis
90.52
76.06
10.10
74.96
Tiny
we Mahalanobis
87.64
70.33
9.70
67.50
IN100
w/o Mahalanobis
93.60
85.06
8.04
86.56
CUB
w/o Mahalanobis
83.78
55.79
0.82
28.15
Compared to VB-CGCD with Mahalanobis distance, the overall final accuracy decreased by approximately 5.56 on average. This is because Euclidean distance, being a special case Mahalanobis, is susceptible to collapse in high-dimensional spaces. By incorporating covariance information, Mahalanobis distance effectively mitigates this issue, thereby achieving a more optimal classification boundary.
17

---

Learning and Forgetting from Bayesian Perspective
C.6. Parameter Size
In the proposed method, the primary storage demand arises from the covariance matrix, whose size scales quadratically with the number of feature dimensions. However, leveraging the symmetric property of covariance matrices, only the lower triangular part needs to be stored, effectively halving the storage requirements.
To examine the sensitivity of the proposed method to the structure of the covariance matrix, we also designed a simplified model that uses a diagonal covariance matrix. As shown in Table 17, while there is a noticeable performance gap, the diagonal matrix model still outperforms other baselines. Furthermore, dimensionality reduction techniques, such as PCA, can be applied reduce the number of feature dimensions, thereby dramatically decreasing the size of the covariance matrix while maintaining competitive performance.
Table 16. Comparison of performance with varying parameter sizes.
S
S:
Sq
S.
Sy
Datasets
Methods
All
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
diagonal
90.46
83.78
83.46
85.40
79.59
79.95
77.40
76.70
77.74
69.40
75.88
75.64
77.80
74.11
75.49
61.70
C100
dim=100
90.86
86.48
87.08
83.50
83.02
84.10
77.80
80.09
81.29
71.70
78.82
78.45
81.80
76.41
76.59
74.80
dim=200
91.34
86.86
87.28
84,80
83.80
84.35
80.50
81.12
81.92
75.50
79.94
79,41
84.20
77.73
77.75
77.50
dim=384
91.68
88.40
89.46
83.10
86.0
87.06
79.60
83.61
84.75
75.60
82.72
82.85
81.70
81,23
81.84
75.70
diagonal
86.66
82,20
82,90
78.70
79.56
81.05
70.60
76.19
79.03
56.30
73.03
75.97
49.50
70.04
72.74
45.70
Tiny
dim=100
87.40
82.90
83.90
77.90
79.20
80.27
72.80
75.96
77.56
64.80
72.66
74.11
61.0
69.94
70.97
60.70
dim=200
88.06
83.75
84.52
79,90
80.48
81.18
76.30
77:42
78.85
67.40
74.32
75.68
63.40
71.87
72.75
63,90
dim=384
88.32
85.10
86.44
78 .0
82.40
84.15
71.90
79.35
81,38
65.10
76.72
78.43
63.0
74.52
75.88
62.20
diagonal
93.20
84,97
82.36
98.0
79.63
78.90
84.0
80.25
78.80
79.44
84.0
78.02
78.91
70.0
IN100
dim=100
93.72
91.47
90.88
94.40
88.20
89.0
83.40
87.57
87.06
91.20
86.04
86.13
85,40
84.26
84.42
82.80
dim=200
93.92
92.0
91.40
95.0
88 82
89.46
85,20
88.37
87.82
92.20
86.93
86.87
87.40
85.16
85, .44
82.60
dim=384
94.32
93.13
93.16
93.0
90.45
91.43
84.60
90.10
90.0
90.8
88.97
89.30
86.40
87.32
88.08
80.40
Table Comparison of performance with varying parameter sizes.
Datasets
Methods
ACC(S,)
AIKS,)
M 4
Ma f
diagonal
90.46
74.11
15.26
74.34
C100
dim=100
90.86
76.41
13.62
77.92
dim=384
91.68
81.23
6,94
79.14
diagonal
86.66
70.04
5.86
60.16
Tiny
dim=100
87.40
69.94
12.28
67.44
dim=384
88.32
74.52
5.72
68.12
diagonal
93.20
78.02
21.48
85.28
IN100
dim=100
93.72
84.26
10.56
87.44
dim=384
94.32
87.32
5.68
87.04
C.7. Experiments in Diverse Scenarios
To explore the practical applicability of the proposed method, we designed two sets of experiments. The first is an extended experiment based on half-classes labeled, where the number of online sessions expanded from (denoted as 'b50t5') to 10 (referred to as 'b50t10'). The experimental setup is as follows:
Table 18, The data configuration for the scenario with 50% categories labeled and 10 continual sessions.
Labeled
Unlabeled
Datasets
categories
samples per category
novel categories
samples per novel
samples per known
CIFAR-100
50
400
400
25
ImageNet- 100
50
1000
1000
60
TinylmagoNet
100
400
10
400
25
Compared to the online sessions experiment, this setup reduces the difficulty of the clustering algorithm, thereby
18

---

Learning and Forgetting from Bayesian Perspective
diminishing some errors caused by clustering inherent. However, it simultaneously introduces noise data from known classes (nearly double). As shown in the results table, the performance decline compared to b50t5 only about 1-3%, demonstrating the robustness and noise resistance of the proposed algorithm.
Table 19, Performance of each session with the b50t10 setting.
Datasets
So
S₁
S₂
S3
S4
Ss
Se
Sg
S9
S10
C100
91.68
90.58
88.51
86,56
85.30
83.90
83.22
82.05
80.40
80.24
79.28
Tiny
88.32
85.80
84.16
82.16
80.91
78.88
78.27
76.85
75.30
74.35
73.45
IN100
94.32
93.96
91.36
88,46
87.45
87. 04.04
86.40
85,50
85.20
85.07
84,48
Table 20.
Performance of each session with the b1019 setting.
Data
Methods
So
S₁
S₃
S3
S.
So
S.
S,
Ss
S,
b50-AII(S,)
C100
VB-CGCD
98.90
93.55
86.60
82.95
81.22
79.31
78.02
76.50
76.31
74.72
81.32
TINY
VB-CGCD
92,70
81.05
79,56
76.05
75.0
73.75
71.52
70.26
68.20
66.30
74.52
IN100
VB-CGCD
98.20
96.20
91.26
91.35
90.08
89.76
87.88
85.80
83.62
82.88
87.32
The second experiment evaluates the proposed method in extremely limited labeled categories scenario. In this setup, only 10% of the classes are labeled, while the remaining 90% are unlabeled and evenly distributed across 9 subsequent online sessions, denoted as 'b10r9'. As shown in Table 20, the proposed method demonstrates strong performance, maintaining high accuracy even with significantly reduced labeled data. These results underscore the robustness and adaptability of the method in scenarios with scarce labeled data.
D. Extensions
D.1. Handling Unknown Number of Novel Classes
In our prior experiments, the number of unknown categories per session was fixed. For scenarios where the count of new classes is unknown, note that numerous off-the-shelf estimation methods (Ronen et al., 2022; Cendra et al., 2024) exist, all of which can be seamlessly integrated into VB-CGCD. For demonstration purposes, we employed the classic silhouette score to implement a method for estimating the number of unknown classes, which we integrated into VB-CGCD. The results are shown in Table 21.
Table 21. VB-CGCD's performance when the number of new categories is unknown.
Sp
Si
S₁
S₃
S₁
Si
Datasets
All
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
C100
91.68
82.48
87,20
58.90
79.08
82.31
59,70
76.25
78.90
57.70
75.24
76.07
68.60
73.13
74.94
56.80
Tiny
88.32
85.55
86.82
79.20
82.41
84.01
72.80
79.87
81.41
69.10
77.20
78.82
64.20
74.77
76.34
60.60
IN100
94.32
91.93
93.12
86.0
89.45
90.43
83.60
88.60
88.28
90.80
86.11
87.62
74.0
84.56
85.08
79.80
CUB
85.72
79.98
83.68
60.91
65.98
67.
57.41
63.91
65.94
49.47
57.54
58.24
51.76
54.40
57.54
25.61
Due to errors in estimating the number of unknown classes, the clustering error is exacerbated, ultimately leading to an average overall accuracy reduction of approximately 5.66. Nonetheless, it demonstrates the scalability of our method to handle scenarios with an unknown number of types, and our performance still outperforms the SOTA method, Happy, which also uses the silhouette score, as shown in Table 22.
D.2. VB-CGCD as a Replay Mechanism
The proposed method, when combined with the NCM classifier, can serve as a standalone C-GCD algorithm. Additionally, VB-CGCD can be utilized for distribution modeling in other C-GCD approaches that rely on Gaussian distributions as a replay or prompt mechanism, enabling seamless integration with these methods.
19

---

Learning and Forgetting from Bayesian Perspective
Table 22. The average accuracy across sessions C100 with silhouette score.
Methods
All
Old
New
Happy
68.80
72,40
45.74
VB-CGCD
77.23
79.88
60.34
To demonstrate its versatility, we designed an experiment where the posterior distributions obtained by VB-CGCD were used as a replay mechanism to train a neural network classifier. In this experiment, the neural network does not learn directly from the training data but instead from samples drawn from the Gaussian distributions generated by VB-CGCD. During sampling, an equal number of samples drawn uniformly for each class, mitigating label bias and avoiding catastrophic forgetting, as the neural network is retrained with a complete and balanced set of data in each session. Experimental results (see Table 23) show that, despite roughly 3% decrease in accuracy, the achieved accuracy is almost identical to that of VB-CGCD, the MLP classification head derived from this process still surpasses existing baselines. This demonstrates the potential of the method promising extension for broader applications.
Table 23. The accuracy of training an MLP when using VB C-CD as the replay mechanism (dim=100).
So
Si
S₃
Ss
S₁
Sa
Datasets
Methods
All
All
Old
New
Old
New
All
Old
New
All
Old
New
All
Old
New
VB CGCD
90.86
86.48
87.08
83.50
83.20
84.10
77.80
80.08
81.28
71.70
78.82
78.45
81.80
76.41
76.58
74.80
C100
VB-CGCD+MLLP
90.68
85.36
77.20
74.50
77.98
79.68
66.10
76.33
76.08
78.30
73.36
73.40
73.30
VB CGCD
87.40
82,90
83,90
77.90
80.26
72,80
75.96
77.56
64.80
72.66
74.11
61.0
69.94
70.97
60.70
Tiny
VB CGCD+LLPP
87,21
80.78
81.42
77.60
77.14
78.0
72.0
72.46
73.76
63.40
68.83
70.28
57.20
65.95
66.79
58.40
VB CGCD
93.72
91.47
90.88
94.40
88.20
89.0
83,40
87.58
87.06
91.20
86.04
86.13
85 $5.40
84,26
84.42
82.80
IN100
VB CBCCGGD+MLP
93.11
90.67
90.24
92.80
87.20
88.53
79.20
86.53
85.86
91.20
84.02
83.95
84,60
82.0
82,22
80.0
D.3. Upgrade to DINOv2
DINO recently introduced a new pre-trained version, DINOv2, demonstrating significant performance improvements. We upgraded the backbone network in VB-CGCD method DINOv2-VIT- VYT-B/14, and the results are shown in Table 24. While all datasets showed slight improvement, the accuracy ImageNet100 remained consistent. This may be attributed the fact that this dataset was already well-leveraged during the pre-training phase of DINO and DINOv2, leaving little room for further gains.
Table 24. Performance with DINOv2
S,
S;
S2
S;
S,
S
Datasets
Methods
All
All
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Old
New
C100
VB-CGCD+DINOw2
94.88
91,41
92.0
88.50
88.78
89.51
84.40
86.42
87.02
82.20
84.46
85.38
77.10
84.03
83.28
90.70
Tiny
VB-CGC
92.10
88.48
88.74
87.20
85.22
86.38
78.30
82,28
83.57
73.30
79.64
81.03
68.50
76.87
78.
65.60
IN100
VB-CGCD+DINON2
96.44
93.16
96.08
78.60
92.45
93.0
89.20
90.42
92.31
77.20
88.60
90.25
75.40
87.28
88.53
76.0
VB-CGCD+DINON2
92.45
84.92
82.21
65.07
76.26
77.11
70.28
72.85
20


Reviews:
[{"review_id": "THadj2n44o", "reviewer": "Reviewer_vXPB", "summary": "This manuscript investigates the problem of Continual Generalized Category Discovery, addressing the challenges of mixed new and old categories and high uncertainty in unlabeled data under a continual learning setting. The authors propose a new variational Bayesian framework that utilizes offline fine-tuning and self-correcting re-labeling to mitigate label bias and label noise. By employing covariance-aware early stopping, the framework balances new-category plasticity and old-category stability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">1.\n\nWe conducted an ablation study on the benefit of introducing Mahalanobis distance, and the results obtained by replacing the Mahalanobis with the Euclidean distance are as follows:   \nDatasets|Methods|All\\(S0\\)|All\\(S5\\)|Mf|Md\n--|---------|---|---|----|---\nC100|w/oMahalanobis|90\\.52|76\\.06|10\\.1|74\\.96\nTiny|w/oMahalanobis|87\\.64|70\\.33|9\\.7|67\\.5\nIN100|w/oMahalanobis|93\\.6|85\\.06|8\\.04|86\\.56\nCUB|w/oMahalanobis|83\\.78|55\\.79|0\\.82|28\\.15\n\nCompared to VB-CGCD with Mahalanobis distance, the overall final accuracy decreased by approximately 5.56 on average. This is because Euclidean distance, being a special case of Mahalanobis, is susceptible to collapse in high-dimensional spaces. By incorporating covariance information, Mahalanobis distance effectively mitigates this issue, thereby achieving a more optimal classification boundary. We appreciate your valuable feedback and will incorporate this aspect into the revised version of our paper.   \n\n> 2.\n\nSince our proposed approach primarily serves as an incremental learning classifier and is decoupled from the clustering module, it allows integration with any clustering method (including those that estimate the number of categories).   \nAs mentioned in the paper, there are many off-the-shelf methods [1][2] available for estimating the number of categories.   \nNonetheless, we employed the classic silhouette score to implement a method for estimating the number of unknown classes, which we integrated into VB-CGCD.   \nThe results are as follows:   \n|Datasets|S0|S1|||S2|||S3|||S4|||S5|||\n|--|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|\n||All|All|Old|New|All|Old|New|All|Old|New|All|Old|New|All|Old|New|\n|C100|91\\.68|82\\.48|87\\.2|58\\.9|79\\.08|82\\.31|59\\.7|76\\.25|78\\.90|57\\.7|75\\.24|76\\.07|68\\.6|73\\.13|74\\.94|56\\.8|\n|Tiny|88\\.32|85\\.55|86\\.82|79\\.2|82\\.41|84\\.01|72\\.8|79\\.87|81\\.41|69\\.1|77\\.20|78\\.82|64\\.2|74\\.77|76\\.34|60\\.6|\n|IN100|94\\.32|91\\.93|93\\.12|86\\.0|89\\.45|90\\.43|83\\.6|88\\.6|88\\.28|90\\.8|86\\.11|87\\.62|74\\.0|84\\.56|85\\.08|79\\.8|\n|CUB|85\\.72|79\\.98|83\\.68|60\\.91|65\\.98|67\\.42|57\\.41|63\\.91|65\\.94|49\\.47|57\\.54|58\\.24|51\\.76|54\\.40|57\\.54|25\\.61|\n\nDue to errors in estimating the number of unknown classes, the clustering error is exacerbated, ultimately leading to an average overall accuracy reduction of approximately 5.66. Nonethless, it demonstrates the scalability of our method to handle scenarios with an unknown number of types, and our performance still outperforms the SOTA method, HAPPY, which also uses the silhouette score. \n\n|Methods|All|Old|New|\n|- |--- |--- |--- |\n|HAPPY|68.80|72.40|45.74|\n|VB-CGCD|77.23|79.88|60.34|\n\n [1] DeepDPM: Deep Clustering with an Unknown Number of Clusters. Meitar Ronen, Shahaf E. Finder, Oren Freifeld. CVPR 2022.  \n[2] PromptCCD: Learning Gaussian Mixture Prompt Pool for Continual Category Discovery. Fernando Julio Cendra, Bingchen Zhao, Kai Han. ECCV 2024. \n\n> 3. \n\nOur approach distinguishes between known and unknown classes by approximately estimating the unknown distribution, thereby effectively reducing the interference of known classes on the learning of new ones—a strategy that is similar to certain open-set methods. However, there are two key differences between GCD and CGCD: 1) GCD does not need to address the forgetting problem inherent in continual learning; 2) In the unsupervised learning phase of CGCD, supervised data is inaccessible, making it unsuitable for semi-supervised approaches. For example, during training, DPN relies on using L_{known} as a part of the loss function, which renders it inapplicable in CGCD scenarios. Moreover, methodologically, DPN uses the mean as its prototype; however, in high-dimensional spaces, this approach is prone to feature collapse. In contrast, utilizing a Gaussian distribution is more robust. We will further emphasize this distinction in the related work section. \n\nAdditionally, we evaluated VB-CGCD on the same datasets which were evaluated in DPN ‘s paper, and the results are shown below. \n\n|        |   banking | |   |   stackoverflow  | |  |   clinc     | | |\n|-------|--------|-------------|---------------------|----------------------|---------------------|---------------------------|---------------------|---------------------|--------------------|\n|        |   All       |   Known           |   Novel     |   All       |   Known     |   Novel     |   All       |   Known     |   Novel    |\n|   DPN        |   72\\.96    |   80\\.93          |   48\\.60    |   84\\.23    |   85\\.29    |   81\\.07    |   89\\.06    |   92\\.97    |   77\\.54   |\n|   VB\\-CGCD   |   75\\.55    |   82\\.88          |   53\\.15    |   84\\.2     |   83\\.73    |   85\\.6     |   90\\.97    |   95\\.16    |   78\\.19   |\n\nThe experimental results indicate that VB-CGCD achieves performance comparable to DPN. However, it is important to note that VB-CGCD does not need access to any labeled data during the unsupervised learning phase, which is a distinct difference from DPN.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**:\n\n1.The paper is well written, with strong readability and logical clarity.\n\n2.The newly designed framework demonstrates impressive performance, achieving state-of-the-art results.\n\n\n\n**Weaknesses:**\n\n1.The experimental design is not sufficiently comprehensive. The ablation study only covers the removal of entire modules but does not analyze specific design components within those modules. For instance, the paper mentions using Mahalanobis similarity to account for covariance structures and mitigate the curse of dimensionality. However, the benefit of introducing Mahalanobis similarity into the proposed framework has not been clearly demonstrated.\n\n2.Although the paper states that existing methods can effectively estimate the number of new categories when it is unknown, there are no experiments or analyses for scenarios in which the number of new categories in each session is indeed unknown. Such a setting would be crucial to validate the effectiveness of the proposed framework in real-world conditions.\n\n3.The framework reduces interference by distinguishing between new and old categories, offering an improvement over purely distance-based methods. However, the paper lacks a more in-depth comparison with open-set recognition and GCD approaches that also differentiate known from unknown classes. (e.g. Extreme Value Theory in open-set recognition and DPN[1] in GCD )\n\nI would be happy to increase my score if the authors can resolve all concerns.\n\n[1] Generalized Category Discovery with Decoupled Prototypical Network", "other_comments_or_suggestions": "None", "questions_for_authors": "Please see weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CfXl6F2x9", "reviewer": "Reviewer_CRqZ", "summary": "This paper studies the task of Continual Generalized Category Discovery (CGCD). It analyzes C-GCD’s forgetting dynamics through a Bayesian lens, revealing that covariance misalignment between old and new classes drives performance degradation. To solve these issues, this paper proposes Variational Bayes C-GCD (VB-CGCD) integrates variational inference with covariance-aware nearest-class-mean classification. VB-CGCD adaptively aligns class distributions while suppressing pseudo-label noise via stochastic variational updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">1. \n\nWe appreciate the reviewer’s valuable feedback. We have regarded HAPPY as an important SOTA but inadvertently missed its discussion in the related work section. We will include the discussion of HAPPY in the revised version, with a particular emphasis on its technique of using Gaussian distributions for prototype sampling, to provide a more comprehensive overview of the relevant work. \n\n>2.\n\nWe reproduced the experimental results of HAPPY and found that our reproduced results outperform those reported in the original paper. We reported the reproduced results in our paper.    \n\n>3. \n\nWe will update the overall process in the caption of Table 1 as follows:  \n\"(a) Offline Stage: Labeled data is used to fine-tune the backbone network. After fine-tuning, features are extracted, and variational inference is performed to obtain and store the class distribution.  \n(b) Online Stage: Unlabeled data is first processed by the feature network to extract features, which are then clustered to assign pseudo-labels. These pseudo-labels are combined with a first round of variational inference to obtain an approximate distribution, then combined with the stored class distribution from the previous session for re-labeling.\"  \n\n>4. \n\nAcross all datasets, we employed the same hyperparameter configuration, including conventional parameters such as epochs and learning rate, as detailed in Appendix C.3.   \nRegarding the fine-tuning hyperparameter λ, we found that the contribution of self-supervision to performance is almost negligible. This is primarily because, compared to self-supervised learning on extremely large-scale pretraining datasets, the fine-tuning dataset has a limited impact on performance, with the cross-entropy loss playing a more crucial role. Therefore, we set λ to a very small value of 0.001.  \n   \nAs for the early stopping hyperparameter, as discussed in Figure 2, we argue that ensuring equal average covariance between new and old classes can effectively mitigate class bias. Consequently, our early stopping strategy dictates training to be stopped once the covariances of new and old classes are equal (i.e., when R equals 0).  \nWe conducted experiments on the CIFAR 100 dataset, focusing on the parameter R, adjusting its value to achieve different trade-offs between model plasticity and stability. \n\n#### Comparisons in overall accuracy as R varies\n\n||S1|S2|S3|S4|S5|\n|----|----|----|----|----|----|\n|R=3|85\\.55|82\\.2|79\\.38|77\\.74|75\\.61|\n|R=2|86\\.85|83\\.3|80\\.2|78\\.41|76\\.13|\n|R=1|88\\.4|85\\.9|83\\.41|82\\.57|81\\.11|\n|R=0|88\\.4|86\\.0|83\\.61|82\\.72|81\\.23|\n|R=\\-1|87\\.03|83\\.75|81\\.31|80\\.18|77\\.91|\n|R=\\-2|87\\.01|83\\.71|81\\.23|79\\.94|77\\.71|\n\n#### Comparisons in the accuracy of old classes as R varies \n||S1|S2|S3|S4|S5|\n|----|----|----|----|----|----|\n|R=3|91\\.62|85\\.31|81\\.98|79\\.17|77\\.44|\n|R=2|91\\.32|86\\.63|83\\.15|80\\.01|78\\.11|\n|R=1|90\\.18|87\\.2|84\\.64|82\\.81|81\\.92|\n|R=0|89\\.46|87\\.06|84\\.75|82\\.85|81\\.84|\n|R=\\-1|87\\.34|84\\.2|81\\.91|79\\.57|77\\.91|\n|R=\\-2|87\\.30|84\\.10|81\\.82|79\\.31|77\\.56|\n\n#### Comparisons in the accuracy of new classes as R varies \n||S1|S2|S3|S4|S5|\n|----|-|-|---|-|-|\n|R=3|55\\.2|63\\.5|61\\.2|66\\.3|59\\.1|\n|R=2|64\\.5|63\\.3|59\\.5|65\\.6|58\\.3|\n|R=1|79\\.5|78\\.1|74\\.8|80\\.7|73\\.8|\n|R=0|83\\.1|79\\.6|75\\.6|81\\.7|75\\.7|\n|R=\\-1|85\\.5|81\\.1|77\\.1|85\\.1|77\\.9|\n|R=\\-2|85\\.6|81\\.4|77\\.1|85\\.0|79\\.0|\n\nThe experimental results indicate that the parameter R serves to balance the accuracy between new and old classes, with the overall accuracy being highest when R equals 0. We will include these details in the updated version of the paper. \n\n>5. \n\nIn continual learning, new data is introduced at each session. If the backbone network is continuously fine-tuned, the features extracted from the same input using different versions of the network (e.g., h₁ and h₂) will inevitably differ, i.e., h₁(x) ≠ h₂(x). We designed an experiment on CIFAR-100, where we continuously perform self-supervised fine-tuning using data from each session. Subsequently, we evaluate feature drift by conducting supervised learning. \n\n|SSL\\-finetuned|overall|S0|S1|S2|S3|S4|S5|\n|--|-|--|---|--|--|--|--|\n|S0|85\\.37|85\\.3|83\\.0|84\\.9|85\\.2|90\\.1|84\\.0|\n|S1|84\\.55|84\\.95|81\\.9|83\\.4|84\\.8|86\\.7|84\\.0|\n|S2|83\\.84|83\\.94|82\\.0|82\\.28|82\\.4|88\\.0|83\\.6|\n|S3|85\\.02|85\\.34|81\\.8|84\\.1|84\\.6|89\\.1|84\\.0|\n|S4|84\\.45|84\\.76|82\\.5|83\\.5|83\\.5|88\\.1|83\\.2|\n|S5|85\\.01|84\\.99|81\\.9|84\\.6|85\\.6|88\\.6|84\\.4|\n\nThe experimental results indicate that as fine-tuning progresses, the accuracy of each session, as well as the overall accuracy, fluctuates within a 2% range, demonstrating the presence of feature drift. Moreover, this drift does not have a consistently positive impact on accuracy and does not lead to significant performance improvements. Since our classification relies on storing the distribution of old classes, any drift may result in inaccurate classifications. Therefore, we chose to avoid continuously fine-tuning the pre-trained network.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This paper derives the method from the perspective of variational bayes, which is clear.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper makes novel contributions of bayes variational inference for continual generalized discovery.", "essential_references_not_discussed": "Please discuss the recent work of C-GCD Happy [R1] in the Related Work.\n\nReferences:\n[R1]. Happy: A Debiased Learning Framework for Continual Generalized Category Discovery. NeurIPS 2024.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper is well-motivated and easy to follow.\n2. The writing and the diagram are clear. I appreciate Figure 1.\n3. The proposed variational Bayes-based method is novel to the community of category discovery.\n4. The performance gains are remarkable.\n\nWeakness:\n1. Discussion of related work Happy [R1] is missing in the Related Work.\n2. Why are some results of Happy in Table1 different from the reported results in the original paper [R1]?\n3. Please include a brief introduction of the method (i.e., spirit, pipeline, etc) in the caption of figure 1.\n4. The experiments of hyperparameters should be added, including $\\epsilon$ in early stopping.\n5. Could the authors provide some experiments that support the claim that self-supervised loss during continual learning introduces feature drift?\n\nReferences:\n[R1]. Happy: A Debiased Learning Framework for Continual Generalized Category Discovery. NeurIPS 2024.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cRal7K8hBg", "reviewer": "Reviewer_9UpM", "summary": "This paper addresses Continual Generalized Category Discovery (C-GCD), or say, iGCD, a task where a model must incrementally learn new classes from unlabeled data streams while preserving knowledge of previously learned classes, a challenge exacerbated by mixed-class data streams and catastrophic forgetting. C-GCD is a task that involves: IL, semi-supervised learning, and class discovery.\n\nIn this work, the authors Variational Bayes C-GCD (VB-CGCD), a Bayesian framework that uses variational inference to model class distributions, align covariances between old and new classes, and mitigate forgetting through a covariance-aware nearest-class-mean (NCM) classifier and an early stopping mechanism.\n\nThe proposed method, VV-CGCD is evaluated on well-established GCD benchmarks: CIFAR-100, TinyImageNet, ImageNet-100, and CUB-200. The results show improvements over the prev. arts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness:\n\n>1. \n\nWe acknowledge that DINO-ViT-B16 was pre-trained on the ImageNet dataset, though its training was based on self-supervised learning using unlabeled data, and it was not trained on CIFAR or CUB. Since all our baselines including HAPPY (SOTA) utilize DINO for feature extraction, VB-CGCD also uses DINO to ensure a fair comparison with other methods. Additionally, to further assess the generalizability of our approach, we have conducted experiments on Stanford Cars, FGVC-Aircraft, CORE50, and Food-101 datasets.\n\n|Datasets|S0|S1|||S2|||S3|||S4|||S5|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||All|All|Old|New|All|Old|New|All|Old|New|All|Old|New|All|Old|New|\n|StanfordCars|72.55|69.41|71.06|45.38|67.08|67.75|56.46|65.45|66.18|53.51|62.35|63.77|37.68|60.92|61.22|55.39|\n|FGVCAircraft|82.32|76.23|79.22|55.47|74.95|75.28|72.34|70.15|71.91|54.51|65.13|67.84|43.67|60.66|62.64|58.03|\n|Core50|99.92|96.50|99.92|79.57|91.11|95.95|61.40|88.91|91.11|73.29|84.93|88.80|54.12|82.28|84.84|58.85|\n|Food101|84.78|81.67|81.80|81.04|79.51|79.87|77.32|78.25|77.85|81.04|75.41|76.30|68.2|73.30|73.76|69.12|\n\nCompared to other baselines, VB-CGCD outperforms them by a decent margin in terms of average overall accuracy, as shown below: \n|Methods|StanfordCar|FGVCAircraft|\n|-|-|-|\n|MetaGCD|54.67|47.16|\n|HAPPY|62.79|53.10|\n|VB-CGCD|65.04|69.42|\n \n> 2.\n\nThank you for your valuable feedback. We will include a discussion on FRoST (Class-iNCD) in the related work section. FRoST employs a Gaussian distribution as the sampling distribution for its replay mechanism, incorporating samples from previously learned classes during training to mitigate forgetting. This represents a typical replay-based approach. In contrast, VB-CGCD does not utilize a replay mechanism but instead relies directly on prototypes for classification. The experimental comparison between FRoST and VB-CGCD is as follows: \n\n|Datasets|S0|S1|||S2|||S3|||S4|||S5|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||All|All|Old|New|All|Old|New|All|Old|New|All|Old|New|All|Old|New|\n|C100|FRoST|90.36|76.87|79.58|63.30|65.31|68.88|43.90|58.01|61.09|36.50|49.27|50.90|36.20|48.03|48.17|46.80|\n|TINY|FRoST|85.86|75.15|78.56|58.10|65.64|67.83|52.50|51.32|54.31|30.40|48.22|52.14|16.90|40.15|42.73|16.90|\n|IN100|FRoST|96.20|87.50|92.96|60.20|79.63|83.37|57.20|76.78|77.00|75.20|66.18|68.65|46.40|63.82|66.40|40.60|\n|CUB|FRoST|90.26|77.03|83.95|43.53|50.77|53.46|34.33|46.42|49.31|26.09|39.40|41.47|23.08|34.55|35.12|29.45|\n\n\nIn terms of final overall accuracy, VB-CGCD outperforms FRoST by an average of 30.76%, demonstrating its superior performance. \n\n>3.\n\nIn unsupervised scenarios, most methods are influenced by the performance of the clustering algorithm. Our strategy is to minimize errors introduced by the clustering algorithm from the classifier's perspective to enhance performance. To ensure fair comparisons with other methods, we have employed the fundamental k-means algorithm. Of course, our approach can also be integrated with more advanced clustering algorithms to further improve performance. \n\n## Q&A\n\n>1. \n\nAcross all datasets and tasks, we employed a consistent set of hyperparameters, including learning rate and training epochs. Specifically, the fine-tuning coefficient λ was uniformly set to 0.001. \nRegarding the early stopping strategy, we posit that equalizing the average covariance between new and old classes effectively mitigates class bias. Therefore, we monitor the covariance of both new and old classes during training and halt training when they are equal, i.e., when R=0. We have conducted a sensitivity analysis on the R. Please refer to Response 4 of Reviewer 2.\n\n>2. \n\nSince our method performs density estimation solely on features, it can be broadly applied to various modalities. We conducted experiments on audio and text classification tasks.  \nFor audio we evaluated our method on esc50, speechcommands, and audiomnist datasets. For text, we evaluated stackoverflow, and clinc. \n\n|Datasets|S0|S1|||S2|||S3|||S4|||S5|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||All|All|Old|New|All|Old|New|All|Old|New|All|Old|New|All|Old|New|\n|ESC50|72.98|75.10|72.03|96.66|78.05|74.68|100.0|78.68|78.05|82.92|80.33|78.05|100.0|78.0|78.93|70.45|\n|Speechcommands|85.25|80.43|82.37|72.11|78.27|79.13|72.67|71.75|76.92|45.38|70.59|70.54|71.01|69.18|69.52|66.44|\n|AUDIO-MNIST|96.22|96.40|96.22|97.26|96.94|96.40|100.0|97.05|96.74|98.85|96.94|96.71|98.68|96.4|96.78|93.75|\n|Clinc|98.66|94.22|98.22|74.22|92.25|94.0|81.77|90.83|91.87|83.55|90.71|90.66|91.11|88.66|90.32|73.77|\n|Stackoverflow|89.0|86.83|87.2|85.0|86.28|86.66|84.0|84.75|85.57|79.0|84.88|84.25|90.0|84.8|84.77|85.0|\n\nThe experimental results indicate that VB-CGCD demonstrates good performance in these tasks as well, further validating its generalizability and robustness to other modalities.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "C-GCD methods can be used to discover novel classes/concepts, which can benefits a spectrum of related fields, such as tissue discoevry and protein discovery.", "essential_references_not_discussed": "There is a significant lack of discussion on related work in this study.  \n\nFirst, the coverage of the Novel Class Discovery (NCD) and Generalized Category Discovery (GCD) literature is very limited, with only two works from each category being discussed. However, a broader spectrum of NCD techniques is highly relevant to this work, as they serve as the foundation for both GCD and Category-wise GCD (C-GCD). A more comprehensive discussion is needed to highlight the technical and methodological similarities between this study and existing NCD and GCD approaches.  \n\nSecond, there is a notable omission of discussions and comparisons regarding Class-incremental NCD, which forms the basis of C-GCD. Foundational works in this domain [1, 2, 3] are neither reviewed nor compared, despite their relevance. Addressing these gaps would strengthen the contextualization of this work within the broader research landscape.  \n\n[1] Joseph K J, Paul S, Aggarwal G, et al. Novel class discovery without forgetting[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 570-586.  \n\n[2] Roy S, Liu M, Zhong Z, et al. Class-incremental novel class discovery[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 317-333.  \n\n[3] Liu M, Roy S, Zhong Z, et al. Large-scale pre-trained models are surprisingly strong in incremental novel class discovery[C]//International Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2024: 126-142.", "other_strengths_and_weaknesses": "Strengths:\n1) the paper is well-written and well-presented. It is easy to read.\n2) the proposed Bayesian C-GCD framework is novel, interesting, and highly generalizable.\n3) the improvements on the evaluated benchmarks look promising and significant.\n4) the abaltion study is sufficient.\n\n\nWeakness:\n1) the benchmarks used for evaluation are significantly limited. The proposed method is based on pre-trained DINO. Note that CIFAR, ImageNet, CUB, are all used in the representation learning of DINO. Although DINO does not use labels to supervise the model, **nearly none of  the \"novel\" classes the model meets at each incremental session are really novel** because these concepts, or even visual content are senn by the encoder during its pre-training. Therefore, I cannot hold high confidence about the effectiveness of the proposed method with current evaluation. Evalution on truely novel benchmarks are needed.\n2) Insufficient discussion and comparison with related work. iNCD methods should be adapted and compared.\n3) dependence on Initial Clustering Quality: The performance of VB-CGCD relies heavily on the quality of the initial clustering used to generate pseudo-labels. Poor clustering results, especially in high-dimensional spaces, could lead to inaccurate pseudo-labels, which may propagate errors and degrade the model's performance over time.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Q1: Sensitivity to Hyperparameters: The method involves several hyperparameters, such as the regularization term and the early stopping threshold. The paper does not provide a detailed sensitivity analysis, leaving it unclear how robust the method is to variations in these hyperparameters across different datasets or tasks.\n\nQ2: How would the method behave in other modalities (non-image data)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyZ8q99tc0", "reviewer": "Reviewer_uAHx", "summary": "This paper introduces Variational Bayes Continual Generalized Category Discovery (VB-CGCD), a Bayesian framework designed to address key challenges in Continual Generalized Category Discovery (C-GCD), including label bias, pseudo-label errors, and the learning-forgetting tradeoff. VB-CGCD comprises four key components: (1) self-supervised offline fine-tuning, (2) self-corrective re-labeling, (3) variational Bayesian distribution estimation with a nearest-class-mean classifier, and (4) covariance-driven early stopping. Experimental results demonstrate that VB-CGCD outperforms existing methods across standard and newly proposed benchmarks with minimal labeled data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Claims And Evidence:\n\n>1.\n\nThe constrained optimization equation formally represents the objective of \"learning new classes without forgetting old ones.\" We observed that maximizing overall accuracy inevitably leads to a decline in the classification performance of previous classes, as illustrated in P2 of Figure 2. Therefore, considering the interplay between old and new classes, we believe that this issue is a multi-objective optimization problem. Accordingly, we relax the original constraint by imposing a covariance constraint, which approximates a balance between preventing forgetting and effectively learning new classes. \n\n>2.\n\nWith advancements in self-supervised pre-training, the effectiveness of self-supervised fine-tuning in small-scale scenarios has diminished. On the other hand, due to the presence of classification errors, employing supervised fine-tuning during online sessions can introduce a significant number of misclassifications, which in turn leads to a decline in overall performance. Meanwhile, we did not entirely freeze the backbone; instead, we performed fine-tuning with labeled data during the offline phase, which was performed using verified ground-truth labels. \n\n>3.\n\nMost methods use the MVN as a sampling distribution for data replay or to assist clustering, rather than directly as a classification prototype. In contrast, VB-CGCD approximates the MVN through variational inference, and we believe that this strategy is the key to our improved performance. \n\n## Experimental Designs Or Analyses:\n\nWe acknowledge that Variational Inference can be computationally intensive. However, our VI process does not need to reach full convergence; only a few steps (1000 epochs in our experiments) are sufficient to distinguish different categories. Additionally, by not requiring fine-tuning of the backbone network during training, we only need to extract features once for each sample, performing a single forward pass through the backbone, which significantly reduces training time. Consequently, compared to other methods that necessitate frequent fine-tuning of the backbone network, our training time is substantially reduced. The running time during training on one RTX-8000 is shown as follows:  \n \n|Datasets|OfflineFine-Tuning (mins)|OfflineSession (mins)|OnlineSession (mins)|Overall (mins)|\n|-|-|-|-|-|\n|C100|67|3.03|1.51|97|\n|TINY|112|2.27|1.58|145|\n|IN100|140|3.36|1.22|169|\n|CUB|12.09|2.01|1.12|39|\n\nCompared to HAPPY (over 25 hours of training) and PromptCCD (over 40 hours of training), VB-CGCD demonstrates significantly superior training efficiency. \nAfter training, VB-CGCD achieves inference computational efficiency comparable to other prototype-based approaches, normally in milliseconds. \n\n## Q&A\n\n>1.\n\nAs a dynamically extensible incremental learning classifier, VB-CGCD can be integrated with advanced clustering algorithms, including those capable of dynamically estimating the number of categories. Since VB-CGCD does not require a preset number of categories, it can be combined with many off-the-shelf methods for estimating unknown category counts. We employed the classical silhouette score to implement a method for estimating the number of unknown categories and integrated it into VB-CGCD. For experimental results, please refer to Response 2 of Reviewer 1. \n\n>2.\n\nVB-CGCD models each category as an independent probability distribution and maintains a collection of all category distributions (i.e., a matrix of latent variables) in memory. After estimating the pseudo-label distribution, we incorporate it into the distribution collection (prototype matrix). During re-labeling, the distances between all samples and this collection are computed to assign labels accordingly. Subsequently, the pseudo-distribution is removed from the collection, and a new distribution is re-estimated. The merge operation is purely an array manipulation, which accounts for VB-CGCD’s inherent scalability, and the reason for being able to handle unknown number of categories.  \n\n>3.\n\nSince the primary goal of continual learning is to enhance overall learning performance, we agree that preventing overfitting to new classes is necessary to mitigate the forgetting of previously acquired knowledge. In fact, many continual learning methods, whether by adding regularization constraints or employing data replay, aim to some extent to slow down the learning of new classes to prevent forgetting old ones. Without imposing any constraints and allowing unrestricted learning of new classes, the model's classification accuracy on new classes may continue to improve, but the accuracy on old classes could drop drastically, as illustrated in phase P3 of Figure 2. Therefore, to maximize overall accuracy, continual learning must strike a balance between stability (preventing forgetting) and plasticity (learning new categories). This trade-off inherently implies that optimizing one aspect will inevitably come at the expense of the other.", "claims_and_evidence": "1. The problem formulation in Eq. (1) requires that the total losses on previous t-1 tasks with parameters leanred after task t-1, does not be increased after learning task t. How is this gaurantee ensured? Does this optimization objective always have a valid solution? It seems that VB-CGCD could not fulfill this constraint, as shown in Figure 2 (a), the accuracy of previous tasks decreases after learning new tasks.\n\n2. The paper states: “To mitigate feature drift, we perform fine-tuning exclusively during the offline stages and keep the backbone network parameters frozen throughout all sessions, including offline and online classification learning.” However, freezing the backbone may hinder the model’s plasticity, particularly when significant distribution shifts occur between sessions. Further discussion on the trade-off between stability and adaptability would strengthen the paper.\n\n3. One of the core ideas, i.e., modeling each class as a multivariate normal distribution (MVN), has been explored in previous continual learning works [1-4].\n\n[1] Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NIPS2023 \n\n[2] Learning semi-supervised gaussian mixture models for generalized category discovery. ICCV 2023\n\n[3] Steering prototype with prompt-tuning for rehearsal-free continual learning. WACV 2024\n\n[4] Happy: A Debiased Learning Framework for Continual Generalized Category Discovery, NIPS 2024", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper presents the theoretical upper bound (Table 9) of model accuracy at the final session, but does not provide formal proofs. This upper bound might come from experiments.", "experimental_designs_or_analyses": "Variational inference-based methods are generally computationally expensive. However, the paper does not compare the computational cost of VB-CGCD with other methods, focusing only on parameter sizes. A computational efficiency analysis would enhance the evaluation.", "relation_to_broader_scientific_literature": "The key contributions of this paper aim to address the challenge of learning from both known and novel categories in an evolving data stream. By providing a practical solution, it aligns with broader efforts in the field to advance continual generalized category discovery.", "essential_references_not_discussed": "Several continual learning works [5-9] that leverage variational inference should be discussed in the related work section for a more comprehensive comparison. \n\n[5] Variational continual learning, ICLR 2018\n\n[6] Variational auto-regressive gaussian processes for continual learning, ICML 2021\n\n[7] Generalized variational continual learning, ICLR 2021\n\n[8] Continual learning via sequential function-space variational inference, ICML 2022\n\n[9] Continual variational autoencoder learning via online cooperative memorization, ECCV 2022", "other_strengths_and_weaknesses": "Other stengths.\n\nThe improvement over previous methods is substantial.", "other_comments_or_suggestions": "1. It is recommended to include punctuation in each numbered equation.\n\n2. Please ensure the correct quotation marks are used throughout the paper.", "questions_for_authors": "1. When the number of new categories is unknown, how does VB-CGCD estimate it effectively?\n\n2. Can you further explain how the class merging is conducted? Specifically, how are “the latent variables of old classes and the pseudo-labeled distribution merged into a unified MVN-NCM classifier”?\n\n3. According to the ablation study results in Table 11, early stopping and re-labeling seem to hinder the learning of new classes. Does this suggest that the proposed method prioritizes stability (performance on old classes) over plasticity (performance on new classes) to achieve a tradeoff?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a variational Bayes-based framework for Continual Generalized Category Discovery (C-GCD), designed to address the challenges of incrementally learning new classes from unlabeled data streams while retaining knowledge of old, previously-seen classes. The authors analyze the forgetting dynamics in C-GCD from a Bayesian perspective and identify covariance misalignment between old and new classes as a key driver of performance degradation. Based on this insight, they introduce VB-CGCD, which integrates variational inference of Gaussian parameters and covariance-aware nearest-class-mean classification. Experimental results on standard benchmarks demonstrate that VB-CGCD significantly outperforms state-of-the-art methods. Of particular note is the performance of the proposed approach on a benchmark with only 10% labeled data in the first, offline supervised training session.

Reviewer opinion of the paper was initially mixed. Reviewers appreciated the readability and clarity of the technical narrative, the impressive performance improvement over baselines on standard benchmarks, and the novelty of the Bayesian C-GCD framework. On the other hand, they raised concerns over insufficiently comprehensive experiments, limited discussion of related work on incremental Novel Class Discovery, and the dependence on known number of classes on the quality of clustering.

The authors engaged with all reviewer concerns in their rebuttal, providing additional results and promising to incorporate more comprehensive discussion of related work and new results in the revised manuscript. Given that the standard practice in GCD and CGCD is to assume a known number of classes (and relegate estimation of this to a generic clustering algorithm), this criticism of the work is not well-founded -- although the authors are encouraged to include the comparison with HAPPY using silhouette score in the revised manuscript.

In terms of issues regarding coverage of the literature in the Related Work section, given page limits and need to narrowly zoom in on the real contributions for conference papers, it is very hard to range far afield and include everything even in the last year that has anything to do with variational inference, continual learning, or GCD. The authors articulated in rebuttal their motivations for excluding some of the mentioned works, and will update the Related Work to include some of the more relevant ones.

After addressing these critical issues in rebuttal, the consensus that remains is that this is a solid and novel contribution which yields excellent improvements over the state-of-the-art.
Decision: Accept (poster)