Happy: A Debiased Learning Framework for Continual Generalized Category Discovery

Authors: shijie  ma
fei  zhu
zhun  zhong
wenzhuo  liu
xu-yao  zhang
cheng-lin  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 08:38:55

Abstract: Constantly discovering novel concepts is crucial in evolving environments. This paper explores the underexplored task of Continual Generalized Category Discovery (C-GCD), which aims to incrementally discover new classes from *unlabeled* data while maintaining the ability to recognize previously learned classes. Although several settings are proposed to study the C-GCD task, they have limitations that do not reflect real-world scenarios. We thus study a more practical C-GCD setting, which includes more new classes to be discovered over a longer period, without storing samples of past classes. In C-GCD, the model is initially trained on labeled data of known classes, followed by multiple incremental stages where the model is fed with unlabeled data containing both old and new classes. The core challenge involves two conflicting objectives: discover new classes and prevent forgetting old ones. We delve into the conflicts and identify that models are susceptible to *prediction bias* and *hardness bias*. To address these issues, we introduce a debiased learning framework, namely **Happy**, characterized by **H**ardness-**a**ware **p**rototype sampling and soft entro**py** regularization. For the *prediction bias*, we first introduce clustering-guided initialization to provide robust features. In addition, we propose soft entropy regularization to assign appropriate probabilities to new classes, which can significantly enhance the clustering performance of new classes. For the *harness bias*, we present the hardness-aware prototype sampling, which can effectively reduce the forgetting issue for previously seen classes, especially for difficult classes. Experimental results demonstrate our method proficiently manages the conflicts of C-GCD and achieves remarkable performance across various datasets, e.g., 7.5% overall gains on ImageNet-100. Our code is publicly available at https://github.com/mashijie1028/Happy-CGCD.

Main paper: 
 introduction: 
 In the open world [1, 2, 3] , visual concepts are infinite and evolving and humans can cluster them with previous knowledge. It is also important to endow AI with such abilities. In this regard, Novel Category Discovery (NCD) [4, 5, 6] and Generalized Category Discovery (GCD) [7, 8, 1, 9, 10] endeavor to transfer [4, 11] the knowledge from labeled classes to facilitate clustering new classes. However, they are constrained to static settings where models only learn once, which contradicts the ever-changing world. Thus, extending them to the temporal dimension is important. In the literature, Continual Novel Category Discovery (C-NCD) [12, 13, 14] and Continual Generalized Category Discovery (C-GCD) [15, 16, 17, 18] aim to discover novel classes continually. C-NCD assumes all data come from new classes, while C-GCD further considers the coexistence of old and new ones. However, C-GCD still has some limitations, e.g., some works [15, 17] store labeled data of old classes, causing storage and privacy [19] issues. Others [16, 18] consider limited incremental stages and novel categories or assume a prior ratio of known samples [15] , failing to reflect practical cases.In this paper, we tackle the task of C-GCD, but with more realistic considerations: (1) More learning stages with more new classes. (2) At each stage, data from previous stages are inaccessible [20] for storage and privacy concerns. (3) Unlabeled data contain samples from old classes but are fewer than new ones in each class, and the ratio of them is unknown. The C-GCD setting is illustrated in Figure 1 with two phases: (1) Initial supervised learning (Stage-0). The model is trained on labeled classes to acquire general knowledge. (2) Continual unsupervised discovery (Stage-1 ∼ T ). At each stage, the model learns from unlabeled data containing both new and old classes. Note that, old classes include initially labeled classes as well as those discovered in previous stages. The core challenge is managing the conflicts between discovering new classes and preventing forgetting old ones.To explore the nature of the conflicts, we conducted preliminary experiments (Section 3.2) which reveal two issues: Models (1) tend to misclassify new classes as old, leading to collapsed accuracy of new classes, and (2) exhibit catastrophic forgetting of old classes. We summarize them as underlying issues to be addressed: (1) Models display overconfidence in old classes and severe prediction bias.(2) The features of old classes are disrupted when learning novel classes. Meanwhile, the similarity between clusters varies, leading to biased hardness across classes for classification.To address these issues, we propose a debiased framework, namely Happy, which is characterized by Hardness-aware prototype sampling and soft entropy regularization. Specifically, on the one hand, to better discover new classes during incremental stages, we utilize clustering-guided initialization for new classes, ensuring a reliable feature distribution. More importantly, to mitigate the prediction bias between new and old classes, we introduce soft entropy regularization to allocate necessary probabilities to the classification head of new classes, which is essential for new class discovery. On the other hand, to prevent catastrophic forgetting in rehearsal-free C-GCD, we model the class-wise distribution in the feature space for old classes, and sample them when learning novel classes, which significantly mitigates catastrophic forgetting. Furthermore, we devise a metric to quantify the hardness of each learned class, and prioritize sampling features from categories with greater difficulty. This helps the model to consolidate difficult knowledge accordingly and thus improves the overall performance. Consequently, these designs enable our model to specifically address the challenges in C-GCD, i.e., effectively discover new classes while preventing catastrophic forgetting of old classes.In summary, our contributions are: (1) We extend Continual Generalized Category Discovery (C-GCD) to realistic scenarios. In addition, we propose a debiased learning framework called Happy, which excels in effectively discovering new classes while preventing catastrophic forgetting with reduced bias in the introduced C-GCD settings. (2) We propose cluster-guided initialization and soft entropy regularization for collectively ensuring stable clustering of new classes. On the other hand, we present hardness-aware prototype sampling to mitigate forgetting. (3) Comprehensive experiments show that our method remarkably discovers new classes with minimal forgetting of old classes, and outperforms state-of-the-art methods by a large margin across datasets.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the abstract and introduction, we present the studied task, and motivation, together with the proposed method and contributions of the paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Category Discovery. Novel Category Discovery (NCD) [4, 5, 21] is firstly formalized as deep transfer clustering [4] , i.e., transferring the knowledge from labeled classes to help cluster new ones.Early works employ robust rank statistics [5, 22] for knowledge transfer. UNO [6] proposes a unified objective with Shinkhorn-Knopp algorithm [23] . Later works [24, 25, 26] exploit relationships between samples and classes. NCD assumes unlabeled data only contain new classes. Instead, Generalized Class Discovery (GCD) [7, 27] further permits the existence of old classes. Thus models need to classify old classes and cluster new ones in the unlabeled data. Recent works handle GCD with non-parametric contrastive learning [8, 28, 10] or parametric classifiers with self-training [9, 29, 30] . More recent works explore GCD in other settings, e.g., active learning [31] and federated learning [32] . In summary, both NCD and GCD are limited to static settings where models only learn once.Continual Category Discovery. Pioneer works [12, 13, 14] study the incremental version of NCD, assuming unlabeled data only contain new classes, and we call them C-NCD. Recent works [15, 16, 17, 18] explore the incremental version of GCD, we collectively refer to them as Continual Generalized Category Discovery (C-GCD). GM [15] proposes a framework of growing and merging.In the growing phase, the model performs novelty detection and implements clustering on the novelties. Then GM integrates the newly acquired knowledge with the previous model in the merging stage. Kim et al. [16] utilize noisy label learning and the proxy and anchor scheme to split the data in C-GCD. Zhao et al. [17] propose a non-parametric soft nearest-neighbor classifier and a density-based sample selection method. Orthogonally, Wu et al. [18] argue that the initial labeled data are not fully exploited and present a meta-learning [33] framework to learn a better initialization for continual discovery. Despite effectiveness, C-GCD settings studied by the above methods still have some limitations, e.g., the number of stages is very few with limited new classes, and the assumption of prior ratio of old classes or storing previously labeled samples is unrealistic [19, 34] . In this paper, we extend C-GCD to more pragmatic scenarios, as shown in Figure 1 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations including the confidence calibration issues and the scope regarding the classification task in the Conclusion, i.e., Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 We first formalize Continual Generalized Category Discovery (C-GCD) (Section 3.1). To delve deeper into the issues, we conduct preliminary experiments (Section 3.2). Results reveal that models are susceptible to two types of bias, which significantly degrade the performance and motivate us to propose the debiased learning framework in Section 4.Task Definition. As shown in Figure 1 , C-GCD has two phases: (1) Initial supervised learning (Stage-0). The model is trained on labeled data D 0 train = {(x l i , y i )} N 0 i=1 of initially labeled classes C 0 old = C 0 init , to learn general knowledge and representations. We denoteC 0 new = None. (2) Continual unsupervised discovery (Stage-1 ∼ T ). At Stage-t (1 ≤ t ≤ T ), the model is fed with an unlabeled dataset D t train = {(x u i )} N t i=1, which contains both old and new classes. We denote the categories in D t train asC t = C t old ∪ C t new . K t old = |C t old |, K t new = |C t new | and K t = K t old + K tnew denote the number of "old", "new" and "all" classes respectively. Note that, after the first stage, i.e., when t ≥ 2, "old" classes include initially labeled classes C 0 init and all new classes discovered in previous stages, i.e.,C t old = C 0 init ∪ {C i new } t-1 i=1, and "new" classes refer to the classes unseen before. At the next stage, new classes from the current stage become the subset of old classes, i.e., C t old = C t-1 old ∪ C t-1 new . The number of novel classes K t new at stage t is known a-prior or estimated using off-the-shelf methods [5, 7, 35] in advance. After training of each stage, the model will be evaluated on the disjoint test setD t test = {(x i , y i )} N t test i=1 containing all seen classes C t old ∪ C t new .Realistic Considerations. Our C-GCD is more realistic than prior arts [15, 16, 18] in that: (1) More stages with more new classes to be discovered. (2) Rehearsal-free. Previous samples are inaccessible for storage and privacy issues. (3) At each continual stage, old classes have fewer samples per class than new classes in the unlabeled data, and the proportion of old samples is unknown.Notations. At Stage-t, we decompose the model into encoder f t θ (•) and parametric classifierg t ϕ = [{ϕ old i } K t old i=1 ; {ϕ new j } K t new j=1] with head of old and new classes. The classifier is ℓ 2 -normalized without bias term, i.e., ∥ϕ t i ∥ = 1. The encoder maps the input x i to a feature vectorz i = f t θ (x i ) ∈ R d .Here, we use ℓ 2 -normalized hyperspectral feature space, i.e., z i = z i /∥z i ∥. The classifier finally produces a probability distributionp i = σ(g t ϕ (z i )/τ p ) ∈ R K t using softmax function σ(•).We conduct preliminary experiments on CIFAR100 [36] using the model described in Section 3.1, which is initially trained on D 0 train and continually discovers new classes on D t train using unsupervised self-training scheme [9] . Results reveal that models are prone to the following two types of bias.Prediction bias in probability space. As illustrated in Figure 2 (a), the model's accuracy for new classes has collapsed. The reason is that old classes C 0 old are trained under full supervision while new classes are under unsupervised self-training [9, 37] , which brings about overconfidence [38, 39, 40] in old classes, as in (b). In this case, prediction bias could occur, where some new classes are incorrectly predicted as old ones, which motivates us to constrain the model to give necessary attention and predictive probabilities to new classes to compensate for this intrinsic gap, as discussed in Section 4.2.Hardness bias in feature space. After adding constraints to ensure learning new classes (Section 4.2), their accuracies significantly fluctuate across incremental stages, leading to unstable performance, as shown in Figure 2 (c ). The underlying cause is that some clusters are more similar to others in the feature space, resulting in lower accuracy of these difficult classes. As in (d), hardness bias (defined in Section 4.3) is obvious across classes. This paper focuses on the hardness of previously learned categories C t old , and addresses how to avoid these biases in preventing forgetting in Section 4.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our code is publicly available at https://github.com/mashijie1028/ Happy-CGCD.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
the proposed framework: happy: 
 Overview of the Method. As shown in Figure 1 , C-GCD has two phases: (1) Initial supervised learning (Stage-0). The model is trained on labeled samples of C 0 init (Section 4.1). Our contribution mainly lies in (2) Continual unsupervised discovery (Stage-1∼ T). Motivated by the conflicts between new class discovery and the forgetting of old classes, as well as the two types of bias discussed in Section 3.2, we propose the debiased learning framework Happy as illustrated in Figure 3 . Specifically, for category discovery, we propose initialization of new heads and soft entropy regularization to resist prediction bias (Section 4.2). To mitigate forgetting, we consider hardness bias and present hardness-aware prototype sampling (Section 4.3). The overall objective is derived in Section 4.4.At Stage-0, the model is trained on labeled data D 0 train from a large number of classes C 0 init to learn general representations, which serves as the foundation for subsequent continual category discovery. We use standard supervised cross-entropy loss on the batch B: L cls = 1 |B| i∈B -y i log p i , where p i = σ(g 0 ϕ (f 0 θ (x i ))/τ ) denotes the prediction. To reduce overfitting, we further employ supervised [41] and self-supervised contrastive learning [42] in the ℓ 2 -normalized projection space: where P(i) is the positive set with the same label and τ c is temperature. The overall loss function is:EQUATIONEQUATIONContinuously discovering unlabeled new classes is challenging, as prediction bias towards old classes could collapse new class accuracy (Section 3.2). Therefore, we need to constrain the model to pay more attention to new classes to ensure effective category discovery.Clustering-guided Initialization. Randomly initialized classifiers bring about unstable training. We argue that clustering could provide a good initialization for new classes. Specifically, at Stage-t, we employ KMeans [43] on D t train and obtain K t = K t old + K t new ℓ 2 -normalized cluster centroids {c i } K t i=1 . Among them, the K t new centroids least similar to old heads, as measured by maximum cosine similarity with them, serve as the potential initialization for new class heads:EQUATIONGroup-wise Soft Entropy Regularization. Entropy regularization [9, 29, 30] is common to avoid trivial solutions of clustering in static settings. However, at each stage of C-GCD, there are generally more old classes. Directly employing it equally across all classes will allocate most of the probability to old classes, leading to prediction bias and collapsed performance (as in Figure 2 (a, b)). To address this, we need to constrain the model explicitly. Considering that at each stage, there are fewer new classes but more samples per new class, and old classes have been well-learned previously, we propose to treat all old classes as a whole and the new classes as another, and derive C-GCD as binary classification. Specifically, we first compute the marginal probability in the batch c) are scalars indicating the marginal distribution on old and new classes respectively, where the superscript (c) denotes class indices and p old + p new = 1. Then we propose soft entropy regularization on the marginal distribution of the old and the new:EQUATIONL old,new entropy = p old log p old + p new log p new .(4) In this way, the model could focus more on each new class, ensuring reliable learning in new classes. We also employ entropy regularization within the new and old classes to avoid trivial solutions:EQUATIONTo sum up, the soft entropy regularization is employed in a group-wise manner on three groups, i.e., "inter old-new" (Eq. ( 4)), "intra old" and "intra new" (Eq. ( 5)), and we add them together:EQUATIONThe soft regularization ensures effective learning of new classes. See Section 5.4 for more discussions.Overall Loss for New Class Discovery. To achieve self-training on unlabeled data, we perform self-distillation [9, 37] . Specifically, we use another augmented view x ′ i to produce sharpened q ′ i with smaller temperature τ t < τ p and employ cross-entropy loss to supervise the prediction p i :L self-train = 1 2|B| i∈B ℓ(q ′ i , p i ) + ℓ(q i , p ′ i ).The overall objective for new category discovery is:EQUATIONwhere λ 1 controls the importance of the proposed regularization loss.Modeling Learned Classes. Catastrophic forgetting [44, 45, 46] is a notorious problem in continual learning, especially when previous samples are inaccessible. Instead of storing seen samples, we can model the feature distribution for learned classes. Since the data in each incremental stage are unlabeled, at the end of each incremental stage, we perform class-wise Gaussian distribution in the feature space using models' predictions on D t train :EQUATIONwhere ŷi = arg max c pi denotes the prediction, µ c and Σ c are mean and covariance. Note that, for Stage-0, we directly use the ground-truth labels instead of predictions in Eq. ( 8). We call µ c as prototypes. When learning new knowledge, one can sample features from old classes N (µ c , Σ c ), and classify them correctly to mitigate forgetting. We find a shared diagonal matrix [47] empirically works fine, i.e., Σ c = rI, where r is computed at Stage-0 asr 2 = 1 K 0 c∈C 0 init Tr(Σ c )/d.Incorporating Hardness to Learned Classes. As in Figure 2 (c), accuracy fluctuations across classes are significant, and treating all classes equally leads to hardness bias and suboptimal results. Intuitively, difficult classes should receive more attention during sampling. Here, we propose an unsupervised metric, considering the samples with higher similarity to others are more prone to be confused and therefore more difficult. We define hardness h i and obtain hardness distribution as:EQUATIONwhere i = 1, • • • , K t old and p hardness is the categorical distribution to sample classes. Those with higher hardness are more likely to be sampled, which better suppresses the forgetting of hard classes. Sequential Sampling. We first sample categories from categorical distribution c ∼ p hardness and then sample class-wise features from Gaussian distribution of the sampled classes z c ∼ N (µ c , rI) for classification. The loss for hardness-aware prototype sampling is :EQUATION)Overall Loss for Mitigating Forgetting. As training proceeds, the feature space becomes outdated for previous prototypes, we thus apply knowledge distillation [48] using the last stage model and current training dataset, i.e.,L kd = 1 |B| i∈B 1 -cos(f t θ (x i ), f t-1 θ (x i )). The overall loss is:EQUATIONwhere λ 2 controls the weight of the knowedge distillation.To continually discover new classes without forgetting old ones, we combine the losses for new (Eq. ( 7)) and old classes (Eq. ( 11)), and contrastive learning (Eq. ( 1)) to formulate the final objective: Datasets. We construct C-GCD on four datasets: CIFAR100 [36] (C100), ImageNet-100 [49] (IN100), Tiny-ImageNet [50] (Tiny) and CUB [51] , each is split into two subsets:EQUATION(1) Stage-0, where 50% of classes serving as C 0 init constitute initial labeled data. (2) Stage-1 ∼ T (T = 5 by default). At each stage, the remaining classes are evenly sampled as new classes, along with all previously learned classes to constitute continual unlabeled data. Detailed dataset statistics are shown in Table 2 The accuracy is calculated using ground truth y i and models' predictions ŷi as: ACC = max p∈P(Ct)1 M M i=1 1(y i = p(ŷ i )), where M = |D t test | and P(C t ) is the set of all permutations across all classes C t old ∪ C t new . The optimal permutation could be computed once using Hungarian algorithm [52] on all classes, and we report "All", "Old" and "New" accuracies as evaluate metrics.Implementation Details. Following the convention [7, 10, 18, 31] , we use ViT-B/16 [53] pretrained by DINO [37] as the backbone, and fine-tune only the last transformer block for all experiments. The output [CLS] token is chosen as feature representation. At Stage-0, models are trained  
comparison with state-of-the-arts: 
 We compare our methods with (1) Kmeans [43] on pre-trained features, (2) GCD methods: Vanil-laGCD [7] , SimGCD [9] , SimGCD+LwF [44] , and (3) recent continual category discovery works: FRoST [12] , GM [15] and MetaGCD [18] . Since GM [15] requires storing exemplar samples, we adjust it to sampling features. For a fair comparison, all methods use the same objective (Eq. ( 2)) to pre-train the model at Stage-0. Results are reported in Table 1 , Table 3 , and Table 4 .Happy outperforms prior methods by a large margin. For example in Table 1 , on IN100, compared to MetaGCD [18] and GM [15] , our approach achieves an improvement of 11.90% and 7.50% for 'All' accuracy, respectively. On C100, Happy improves the previous state-of-the-art by 3.45% and 13.60% for old and new classes across 5 stages. Besides, our method produces more balanced accuracy between 'Old' and 'New'. These improvements benefit from our consideration of underlying bias in the task of C-GCD and the tailor-made debiased components in Happy.Happy effectively balances discovering new classes with mitigating forgetting old classes. To decouple and analyze the two conflicting objectives, we use M f and M d in [15] to evaluate the overall forgetting of labeled classes and the discovery of new classes respectively. Table 3 shows that VanillaGCD [7] and MetaGCD [18] struggle with category discovery due to the weak supervision of contrastive learning. In addition, FRoST [12] focuses solely on new classes, at the expense of old class performance. In contrast, our method effectively balances both, achieving improvements of 6∼12% in two metrics.C-GCD with more continual stages. To explore more realistic and challenging scenarios, we conduct C-GCD with 10 continual stages. Results in Table 4 demonstrate that Happy consistently outperforms other counterparts, showcasing Happy is a competent long-term novel category discoverer.Here, we conduct extensive ablations on each main component (Table 5 ) and analyze how our method handles the conflicting goals between discovering new classes and mitigating forgetting old ones. Finally, we delve into the mechanism of hardness in our framework. How does Happy mitigate catastrophic forgetting? (f) includes hardness-aware sampling based on (d), which improves 'Old' ACC by 2.59% on CIFAR100. However, without L kd , the feature space could drift significantly when learning new classes and become misaligned with the learned classifier, which degrades the performance. As a whole, (g) incorporates L kd to remarkably improve 'Old' by 4.43% and 1.95% on CIFAR100 and CUB. Similarly, better clustering of new classes also benefits old ones because incorrectly classifying new as old can hinder the learning of old classes. In this sense, the learning of new and old classes is mutually reinforcing. How does hardness-awareness help C-GCD? To delve into the effectiveness of hardness-aware modeling, we conduct ablations with and without it. Results (average 'All' accuracy across 5 stages) in Figure 4 show that hardnessawareness consistently improves performance across various datasets. We also present sensitivity analysis on temperature τ h in Eq. ( 9). As Table 6 shows, τ h = 0.1 is a proper choice. When τ h is too large, p hardness convergences to the uniform distribution, which is similar to the one without hardness modeling. A small τ h also brings suboptimal results. In such cases, p hardness becomes overly sharp, resulting in the sampling of only a very limited number of hard classes, which exacerbates the forgetting of remaining categories.Does incorporating class prior into regularization necessarily improve results? Without any prior knowledge about the proportion of new and old class samples, we employ soft entropy regularization in Eq. ( 4) to prevent bias. A natural question arises: Can the introduction of information about the ratio of new to old class samples at each stage further enhance performance? To explore this issue, we directly use the ground truth ratio of old and new samples p gt old , p gt new as a prior and modify Eq. ( 4) as L old,new prior = -p gt old log p old -p gt new log p new . That is, using cross-entropy to supervise the model's predictive probabilities p old , p new , which surprisingly degrades performance as shown in Figure 5 . The reason lies in the gap between the model's predicted ratio of new class samples (pred new ratio) and the prior ratio of new classes p gt new (gt ratio), as revealed in Figure 6 , which is caused by the confidence gap between old and new classes (Figure 2 ). This gap ultimately causes the predicted ratio of new samples to exceed p gt new , bringing about degraded performance than using Eq. ( 4) without any prior. Unknown class number scenarios. Previous experiments assume the number of new classes K t new is known, which often does not hold in reality. At the start of each stage, we need to first estimate the number of new classes before instantiating the classifier. Prior arts [5, 7] query some labeled data when estimating the class number, which is not applicable in the purely unsupervised setting of C-GCD. Instead, we employ off-the-shelf silhouette score [35] to estimate K t new in an unsupervised manner. Specifically, we compute silhouette score using mean intra-cluster distance and mean nearest-cluster distance and select the number of classes corresponding to the highest score value as the estimation. Then we utilize the estimated number for training and evaluation. Average accuracies across 5 stages on CIFAR100 are reported in Table 7 . Our method outperforms others when K t new is not known a-prior. Happy could effectively mitigate two types of bias in C-GCD. As elaborated in Section 3.2, models in C-GCD are susceptible to prediction bias and hardness bias. To validate the effectiveness of the proposed method in bias mitigation, we design metrics to quantitatively measure these biases. Specifically, for prediction bias, we provide two metrics: (1) ∆p(↓) = p old -p new denotes the difference in marginal probabilities between old and new classes (see Section 4.2). ( 2) ∆r(↓) denotes the proportion of new classes' samples misclassified as old classes. Both ∆p and ∆r are calculated on the test data after Stage-1. The results in Table 8a from two datasets demonstrate that L entropy-reg effectively reduces prediction bias, with a significantly lower marginal probability gap and fewer new class samples misclassified as the old. For hardness bias, we also present two metrics: (1) V ar 0 (↓) denotes the variance in accuracy of the initial labeled classes C 0 init . (2) Acc h (↑) denotes the accuracy of the hardest class in C 0 init . Both metrics are calculated after 5 stages. Results in Table 8b demonstrate that hardness-aware sampling effectively reduces hardness bias, with lower accuracy variance and higher hardest accuracy. In this regard, the proposed modules competently alleviate both types of bias, which is consistent with our motivation. 
conclusive remarks: 
 We tackle the pragmatic but underexplored task of Continual Generalized Category Discovery (C-GCD), which involves conflicting goals of continually discovering unlabeled new classes while preventing forgetting old ones. We further identify prediction bias and hardness bias hinder the effective learning of both old and new classes. To overcome these issues, we propose a debiased framework namely Happy. The clustering-guided initialization and soft entropy regularization collectively alleviate prediction bias and ensure the clustering of new classes. On the other hand, by modeling the hardness of learned classes, we propose hardness-aware prototype sampling to dynamically place more attention on difficult classes, which significantly prevents the forgetting of old classes. Overall, our method achieves better discovery of new classes with minimal forgetting of old classes, which is validated by extensive experiments across various scenarios.Limitations and Future Works. Due to the imbalanced labeling conditions between the initial and continual stages in C-GCD, the model's confidence is not calibrated and there is an obvious confidence gap between old and new classes, in these cases, incorporating prior information even degrades performance (Section 5.4). Future work should incorporate confidence calibration [38] into C-GCD to further mitigate potential biases. Another promising direction is to devise competent class number estimation methods for C-GCD, because in the unsupervised setting, class number estimation becomes significantly challenging. Additionally, this paper primarily discusses classification tasks, while future works could extend the C-GCD learning paradigm to object detection [54] , segmentation [55] and multi-modal learning [56, 57, 58] .In this section, we first provide a detailed explanation of the task of Continual Generalized Category Discovery (C-GCD) and a comparison with class-incremental learning. Then we illustrate the practicality of C-GCD studied in this paper through some examples.The core differences between C-GCD and Class-incremental Learning [20, 44] (CIL) lie in that training data is fully unlabeled at each stage of C-CGD, by contrast, conventional CIL adopts a fully-supervised setting. On the other hand, at each stage of C-GCD, the unlabeled training data D t train contains samples from previously seen classes, which makes the task more challenging because models need to implicitly or explicitly split the samples from old and new classes and then discover novel categories. While in rehearsal-free CIL, at each stage, the labeled training dataset typically does not contain samples of previous classes, otherwise it becomes the replay-based sitting and will simplify the problem, because the training data is fully labeled.As mentioned in the main manuscript, we study a more pragmatic setting of C-GCD, whose specific manifestations of realistic considerations are listed as follows:More continual stages with more novel categories to be discovered. Prior works [15, 16, 18] mainly implement C-GCD with 3 stages given nearly 70% of all the classes serving as labeled classes. This simple setting does not reflect real-world scenarios. Humans are lifelong learners over the course of their entire lives, and our setting closely aligns with this situation. Specifically, the default setting in this paper has 5 continual stages with 50% of all the classes serving as novel classes.Rehearsal-free setting without storing previous samples. Several works [15, 17] in C-GCD require the storage of previous samples to construct a non-parametric classifier or mitigate catastrophic forgetting. This store-and-replay manner could cause privacy and storage issues, especially in cases with very long learning periods. While we study the rehearsal-free C-CGD.The ratio of new class samples is unknown. Some works study C-GCD by assuming that the proportion of new class samples per stage is known, which facilitates the design of novelty detection, owing to the fact that novelty detection [15, 39] typically relies on a threshold to determine whether a sample is from novel classes. In our setting, we lift this restrictive assumption and our framework Happy does not rely on the ratio. Instead, our method does not explicitly perform novelty detection, but instead implicitly learns with soft entropy regularization and self-distillation.At each stage, the number of samples of each old class is significantly fewer than the number of samples in each new one. If at each stage, the number of class-wise samples of old and new classes is roughly the same or both have plenty of samples, then C-GCD degenerates to the static setting of GCD where the catastrophic forgetting is inherently avoided because there are plenty of samples for each class, which bring about desirable outcomes even using baseline methods. This also contradicts the reality. Imagine a scenario where a student is growing up and entering different stages of learning. For example, he is currently in college where he needs to self-learn many new subjects like calculus and linear algebra. However, he occasionally encounters some old knowledge from his high school days, such as trigonometry and plane geometry. In this case, new knowledge is mixed with old knowledge, but the quantity of old knowledge is quite small.Fair training in Stage-0. We train all the methods using similar objectives at Stage-0, specifically, for methods with parametric classifiers [9, 12, 15] , we employ L init in Eq. ( 2), while for methods with contrastive learning and non-parametric classifiers [7, 18] , we employ supervised and self-supervised contrastive learning on the labeled data, i.e., the last two terms in Eq. ( 2). The results of different methods at Stage-0 are similar, as shown in Table 1 , ensuring fair comparisons of subsequent continual learning stages.Model Details. Following the convention of the literature [7, 10, 18] , we use ViT-B/16 [53] pre-trained with DINO [37] as the encoder, and fine-tune only the last transformer block for all experiments. The output [CLS] token is chosen as feature representation. For the parametric classifier, we use ℓ 2 weight normed prototypical classifier [9, 30] without the bias term. The dimensionality of feature space and projection space for contrastive learning is 768 and 65,536, respectively. Note that all the feature vectors in the 768-dimensional feature space are ℓ 2 -normalized, i.e., hyperspherical feature space, including the feature representation z i of each sample x i , the head of each class in the classifier ϕ i , the KMeans [43] cluster centroids c i in Eq. ( 3), the class-wise prototypes µ c in Eq. ( 8) and the sampled features z c in Eq. (10) .Training Details. We train the models in Stage-0 for 100 epochs with a learning rate of 0.1, and 30 epochs with a learning rate of 0.01 for each of the continual stages. We use a cosine annealed schedule for the learning rate.Hyper-parameters and implementation details of Happy. For the weights of loss terms, we empirically set λ 0 = 0.35 and λ 1 = λ 2 = 1, and detailed hyper-parameter analysis is elaborate in Section E.5. For the temperature, we set the main temperature τ p = 0.1 in model predictive probability p i and the τ t in the sharp soft q i . We set τ h in hardness distribution p hardness as 1. As for the temperature in the contrastive learning term, we follow prior arts [7, 9] and set τ c as 0.07 and 1 for supervised and self-supervised contrastive learning, respectively. When we compute L old,in entropy and L new,in entropy in Eq. ( 5), the distribution within old p (c) , c ∈ C t old and new classes p (c) , c ∈ C t new should be firstly normalized whose summation across the class indices equals to 1.In this section, we give a detailed algorithm of Happy in Algorithm 1, including both (1) Initial supervised learning (Stage-0) and (2) Continual unsupervised discovery (Stage-1 ∼ T ). Train the model h 0 on D 0 train using the loss function Linitial in Eq. ( 2). ▷ Train the model h t on D t train using the overall loss function LHappy in Eq. (12) 14:Input: Initial labeled dataset D 0 train = {(x l i , yi)} N 0 i=1 of K 0 classes C 0 init ,end for 15:▷ Compute class-wise prototypes µc (c = K t-1 + 1, • • • , K t ) in Eq. ( 8) using model predictions 16:▷ Append new prototypes to the existing set 17:▷ Model hardness distribution phardness on all the prototypes using Eq. ( 9) 18: end for Output: The trained model h T = g T ϕ • f T θ (•) that could perform classification on all seen classes. C-GCD is essentially a clustering problem, specifically for the unlabeled new classes. Following [7, 5, 9, 10, 18] , the accuracy is calculated using ground truth y i and models' predictions ŷiEQUATIONhere, M = |D t test | is the number of samples in the test dataset and P(C t ) represents the set of all permutations across all classes C t old ∪ C t new . The optimal permutation could be computed once using Hungarian algorithm [52] , and subsequently 'All', 'Old' and 'New' are computed on corresponding indices of classes. C-GCD utilizes inductive evaluation, i.e., models are evaluated on a disjoint test dataset containing all of the seen classes.To decouple and analyze the objectives of novel class discovery and preventing forgetting, GM [15] designed new metrics, i.e., the maximum forgetting metric M f and the final discovery metric M d . They are defined as follows:EQUATIONEQUATIONHowever, old classes at different stages are changing and expanding. in the above definitions, M f does not truly quantify the forgetting of the initial classes. On the other hand, M d only measures the category discovery performance at the last stage, which overlooks measuring the accuracy of new categories throughout the process. As a result, we re-define these two metrics as follows:EQUATIONEQUATIONIn our metrics, M f quantify the forgetting of fixed classes set C 0 init which is more reasonable, and M d measures category discovery of each new classes, which could more comprehensively reflect the ability to cluster new classes. In the main manuscript, we use the re-defined M f and M d to evaluate models in Table 3 .Here, similar to the preliminary experiments in Figure 2 , we train baseline models and provide more metrics, i.e., maximum softmax probability, maximum logit value, margin and negative entropy, of confidence distribution on new and old classes, as illustrated in Figure 7 . The results consistently reveal the severe confidence gap between old and new classes, which is the underlying cause of prediction bias.In the main paper, we conduct experiments with 5 continual stages by default. To evaluate models in more realistic scenarios with longer continual learning stages, we provide more detailed results of over five continual stages are reported in Table 11 . Happy also achieves remarkable performance on these fine-grained datasets.We fix the weights of L self-train and L hap as 1, considering they are the main objectives for new and old classes. As a result, our method mainly contains three loss weights λ 1 , λ 2 , λ 3 for L entropy-reg , L kd and L u con , respectively. Here, we give a sensitivity analysis on CIFAR100 and report average 'All' Acc in Table 12 . As shown above, the model is relatively insensitive to λ 3 , whereas λ 1 and λ 2 have a more significant impact. Overall, the optimal values for each hyper-parameter are close to 1. In our experiments, we simply set all weights to 1, which shows remarkable results across all datasets. Thus, our method does not require complex tuning of parameters and exhibits strong generalization capabilities and practicability.Table 12 : Sensitivity analysis of hyper-parameters λ 1 , λ 2 and λ 3 . This paper focuses on Continual Generalized Category Discovery (C-GCD) and primarily addresses the classification issues. From a more intrinsic perspective, it represents a paradigm of transferring existing knowledge to continuously generalize and learn new information. Therefore, it can be applied to a wide range of tasks and scenarios, such as reasoning abilities in LLMs [62] and multimodal models [58, 63] , continuous pre-training and instruction tuning, and large generative models' generalization abilities to novel concepts. In the fields of biology and health sciences, the principle of C-GCD can assist the discovery of new species and drugs, which will help human beings understand the ecosystem better and facilitate timely diagnosis and treatment of new diseases.At its core, C-GCD involves leveraging and transferring knowledge learned from old categories to learn new information better, embodying the principle of applying learned concepts to new situations. From this perspective, old knowledge significantly determines the model's ability to discover new knowledge. If biases or unfairness are learned from old knowledge, these issues can also manifest in the newly discovered knowledge. As a result, future works should pay attention to the bias and fairness issues, specifically when learning old classes, and the scrutiny of newly learned knowledge.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present the dataset splits in Table 2 , and give details about the experimental details in Section 5.1 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We report all the experimental results as the average over 5 runs, but we do not report the error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have cited all the comparative methods and necessary adjustments in Section 5.2.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We are currently organizing the codes and will release them as soon as possible.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This paper studies a new task called Continual Generalized Category Discovery (C-GCD), which aims to incrementally discover more new classes from unlabeled data over a long period while maintaining the ability to recognize previously learned classes without storing samples of past classes. There are two challenges in C-GCD, including the prediction bias in probability space and the hardness bias in feature space. To address these challenges, this paper proposes cluster-guided initialization and soft entropy regularization to mitigate prediction bias, and proposes hardness-aware prototype sampling to mitigate hardness bias and forgetting. Experimental results demonstrate the effectiveness of the proposed method.

This paper finally received 1x Borderline reject, 2x Borderline accept, 1x Weak accept. Reviewer TDQJ has raised concerns about the negative impacts of the proposed method, lack of theoretical results, the limited applicability, and the stability of the method. After the authors' rebuttal, Reviewer TDQJ indicated that some concerns were addressed. Reviewer TLvc thinks that the setting is over-claimed, there are too many hyper-parameters and loss terms, more evidence should be provided to show that the two issues are mitigated. The authors have provided corresponding feedback, and Reviewer TLvc confirmed that these concerns were resolved and would keep the original score (Borderline accept). Reviewer jUDg mainly has concerns about the C-GCD setting and the potential information leakage issue. After discussing with the authors, Reviewer jUDg acknowledged that these concerns were addressed and increased the score from 6 (Weak accept). Reviewer cVL6 (Borderline reject) has concerns about the complexity of the framework, scalability concerns, evaluation metrics, generalizability to other domains, and adaptation to state-of-the-art methods. Reviewer cVL6 has also been involved in the discussion with the authors, but indicated that most of the concerns were not addressed. Reviewer cVL6 still has concerns about the hyperparameters and theoretical contributions. I have checked all the comments from Reviewer cVL6 and the authors, I would agree with Reviewer cVL6 regarding the two concerns, but I feel that the hyperparameter issue is not a fatal problem. Besides, the lack of theoretical contributions also may not be a very strong reason to reject this paper, because this paper studies a new practical task with an innovative method achieving satisfactory performance. I personally think that whether a paper should be accepted should not simply depend on whether this paper has some minor problems. We should focus on the key merits of the paper. As most reviewers acknowledged, this paper indeed has merits in a new task, an innovative method, and good performance.

The current average score of this paper is 5.0, which means that the decision of this paper is clearly on the borderline. I have carefully checked the paper, all the comments, and all the discussions. I noticed that the relatively low score is actually caused by some minor problems. After much consideration, I feel that this paper could be accepted.

Reviews: review: paper_summary: paper_summary: The article presents a method for Continual Generalized category discovery. A de-biasing learning framework for the Category Discovery (C-GCD) task is designed to address the challenge of continuously discovering new concepts in an ever-changing environment while maintaining recognition of known categories. Traditional C-GCD studies have some limitations, such as storage and privacy issues caused by storing samples of past classes, and only considering limited incremental stages or assuming a proportion of known samples, which are not in line with practical application scenarios. Therefore, the study focuses on a more realistic C-GCD setup that includes more learning phases, more new categories, and after each phase, data from previous phases is not accessible.
strength_weakness: strength_weakness: 1. Although the authors acknowledge and briefly discuss the social impact of technology, the paper does not detail specific negative impacts that a "Happy" framework could bring, such as potential bias transmission, fairness issues, privacy violations, or security risks. For an algorithm intended for application in the open world, the lack of a comprehensive social impact analysis may limit its acceptance and ethical application in practice.
2. It is clearly stated that this study does not include theoretical results, which means that a complete hypothesis set and theoretical proof are not provided to support the validity of the proposed method. While empirical research has shown the effectiveness of "Happy, "the lack of a solid theoretical foundation may diminish its persuasive power in academia.
3. The authors acknowledge that there are some assumptions and limitations to the study, such as overfitting the model, performance in noisy environments, and testing on specific datasets. These factors may limit the method's general applicability and ability to generalize, especially in different data distributions or more complex real-world environments.
4. It is pointed out that even after the introduction of the debias mechanism, the recognition accuracy of the new class still fluctuates at different incremental stages, which indicates that the model may have stability problems when processing data of different class difficulty. This volatility can affect model reliability and user trust in real-world deployments.
questions: questions: 1. In the experimental part, how do you make sure that in the unsupervised increment phase, the model not only finds new categories, but also accurately distinguishes them without confusing them with the old ones? Are there specific metrics or experimental Settings to measure this ability to differentiate?
2. How does the soft entropy regularization realize the reasonable distribution of the probability of the new class? Can you explain this process in detail and how it is combined with cluster-guided initialization to improve the clustering performance of new categories?
3. In the hardness-aware prototype sampling strategy, how do you define and quantify the "hardness" of a class, and how do you sample effectively against this hardness to mitigate forgetting? Are there concrete examples of how this strategy helps the model remember old categories that are difficult to classify?
4. The paper mentions that in preliminary experiments it was found that the model tends to misclassify new categories into old ones, and that the features of the old categories are disturbed when learning the new ones. In addition to the proposed solutions, have other technologies, such as meta-learning or memory enhancement networks, been considered to further improve these problems?
limitations: limitations: The authors discussed the limitations of the work in the paper, including the confidence calibration problem and the scope of the study mainly focused on the classification task.
Confidence Calibration problem: In the Continuous Generalized Category Discovery (C-GCD) task, the confidence of the model is not calibrated due to an imbalance in the labeling conditions between the initial phase and the sustained phase. This leads to a clear gap between the old category and the new category, and even degrades performance when incorporating prior information.
The scope of application is limited to classification tasks: Although this paper mainly discusses the C-GCD learning paradigm under classification tasks, the application of this method has not been extended to other fields, such as object detection and image segmentation.
The authors note that future work should consider how confidence calibration techniques can be incorporated into C-GCD to further reduce potential bias. At the same time, they encourage the extension of the C-GCD learning framework to more types of visual tasks in order to expand its applicability and impact. These discussions reflect a clear understanding of the limitations of the current findings and suggest possible directions for future research.
paper_summary: paper_summary: The paper points out two-bias issues in CGCD: prediction bias in probability space and hardness bias in feature space. To tackle those two issues, they propose cluster-guided initialization and soft entropy regularization to mitigate prediction bias, and they propose hardness-aware prototype sampling to mitigate hardness bias and forgetting.
strength_weakness: strength_weakness: 1. Overclaim. The paper argues that they extend CGCD to realistic scenarios. But from the experiment setup, I am not convinced that the scenarios they consider are more realistic than existing work [16,18]. The stage and new classes are still limited.
2. Too many hyper-parameters and loss terms. There are too many hyper-parameters to adjust different loss weights, hindering its generalization. In addition, there are too many loss terms, hindering the understanding of the method. For example, what is the effect of the proposed Eq 4? 
3. In Table 5, why KD improve New class by a large margin? What does the new class mean? The total novel class or the new class in the last task?  
4. As the paper points out two issues: prediction bias and hardness bias, and proposes methods to mitigate the two issues, they should provide evidence to show that two issues are mitigated.


Minor:
1. The rightmost one in Figure 2 is (d) instead of (b).
2. $L_{hap}$ in eq 10 and 11 should be consistent.
3. citation 23 is a little strange. It seems that it was published in ICCV2023. The citation should be corrected.
questions: questions: See weaknesses, especially 2 and 4. I will raise my score if the 2 and 4 are well resolved.
limitations: limitations: None
paper_summary: paper_summary: The paper proposes a novel method for the Continual Generalized Category Discovery task, addressing the challenges of discovering new classes and preventing forgetting. The approach introduces Clustering-guided Initialization and Group-wise Soft Entropy Regularization for class discovery, as well as Hardness-aware Prototype Sampling for mitigating forgetting. Rigorous experimentation across several datasets demonstrates a significant improvement in performance.
strength_weakness: strength_weakness: 1. In the C-GCD setting described in the paper, during the continuously discovering stage, the samples of old classes are available. Why not use these old class data directly instead of the Hardness-aware Prototype Sampling method? Some explanation or experimental validation should be added.

2. How is the margin probability computed in line 173 if all the labels are not available?

3. The proposed method uses DINO pretrained ViT-B/16 as the backbone, which, as I understand, is pretrained on the ImageNet dataset. This results in information leakage for CIFAR-100, ImageNet-100, and TinyImageNet datasets in the continual learning process, as all classes are already known in the pretrained model.
questions: questions: Please see the Weaknesses
limitations: limitations: No
paper_summary: paper_summary: The paper presents a novel approach to the task of Continual Generalized Category Discovery (C-GCD). The proposed framework, named Happy, aims to address the challenges of continuously discovering new classes from unlabeled data while preventing the forgetting of previously learned classes. The authors identify two primary issues in C-GCD: prediction bias and hardness bias. To mitigate these, they introduce clustering-guided initialization, soft entropy regularization, and hardness-aware prototype sampling. Experimental results show that the Happy framework significantly improves performance across various datasets, including notable gains on ImageNet-100.
strength_weakness: strength_weakness: Complexity of the Framework: While the combination of multiple techniques (clustering-guided initialization, soft entropy regularization, and hardness-aware prototype sampling) is innovative, it also adds complexity to the framework. This could make it challenging to implement and tune in practice.

Scalability Concerns: The paper does not extensively discuss the scalability of the proposed framework. As the number of classes and stages increases, the computational and memory requirements could become prohibitive.

Evaluation Metrics: The paper primarily focuses on accuracy improvements but does not delve deeply into other potential evaluation metrics, such as computational efficiency, memory usage, or robustness to different types of data distributions.
questions: questions: Generalizability to Other Domains: While the framework is validated on multiple vision datasets, it is unclear how well it would generalize to other domains, such as natural language processing or other types of sequential data.

Adapt to the SoTA methods: Is it possible to adopt the method to several significant recent works in the field, such as SPTNet, TIDA, and InfoSieve?
limitations: limitations: NA

Comments: title: Further response to Reviewer cVL6
Further Response to Reviewer TLvc
Further Response to Reviewer TDQJ
Further Response to Reviewer cVL6
Response to Reviewer cVL6
Thanks for your recognition
Response to Reviewer jUDg
comment: Dear Reviewer cVL6,

Thank you for your feedback. We will respond to your concerns point by point.

* Lack of a solid theoretical foundation.

  * Actually **we have included the theoretical foundation of our method**. Please refer to **Further Response to Reviewer TDQJ**, we also include this point as below.
  * The theoretical foundation of our method is the **InfoMax principle** [R1], namely **maximizing the mutual information (MI) between the inputs and outputs of a system.** Here inputs and outputs denotes the feature $Z$ and model predictions $Y$.
  * For the self-training problem of learning new classes in C-GCD, we employ the maximizing the mutual information (MI) theory. Specifically, the objective can be expressed as $\max I(Y, Z) = H(Y) - H(Y|Z)$, where $Y$ represents the labels and $Z$ the features, with $I$ and $H$ denoting mutual information and entropy, respectively.
  * The proposed entropy regularization on model predictions $Y$ is the minus of first term $-H(Y)$, while the self-distillation objective aims to encourage model to predict more confidently, resulting in lower prediction entropy, namely the second term of minimizing $H(Y|Z)$, namely maximizing $-H(Y|Z)$.
  * Overall, our method follows the Infomax principle and aims to maximize mutual information, to ensure the quality of self-learning.

* Not quite stable debias mechanism affect model reliability.
  * Instability is an inherent issue in C-GCD; however, our approach has effectively mitigated this instability, exhibiting significantly improved stability compared to previous methods.
  * **Example 1:** On CIFAR100, our method improves the hardest accuracy from 65.10% to 70.23%, and the variance decreases from 23.04% to 10.33%, which means our method greatly mitigates the forgetting of hard old classes and alleviates the extent of imbalance caused by hardness bias.
  * **Example 2:** Our method exhibits significantly less variability compared to others,On CIFAR100, the maximum new accuracy difference across stages is $\Delta_{new}$=6.3 (56.1-49.8), which is less than the 17.3 (48.9-31.6) of **previous sota** MetaGCD. Similar patterns can be observed in other datasets as well.
  * Our method is robust and generalizes well to 15 unseen distributions, please see **Rebuttal PDF**.
  * Overall, our method is a relatively robust method which is much more stable than previous methods.

* Hyper-parameter tuning.
  * As we responded in **General Response to Common Concern 1** and **Further Response to Reviewer TLvc**, we did not claim that the hyper-parameters are all optimal at 1, but rather, they are **near-optimal**, which is sufficient to achieve SOTA  results across 6 datasets and 15 unseen distributions, without the need for meticulous tuning, thus saving on computational costs.
  * For $L_{new}$ and $L_{old}$, our results show that a 1:1 weight ratio is optimal, because both old and new classes matter and we need to maintain a balanced focus on both new and old classes as a whole (other weight ratios of $L_{new}$ and $L_{old}$ will result in a 5% to 10% performance degradation). So keep the weight ratio of $L_{new}$ and $L_{old}$ to 1, and the only parameter for tuning is $\lambda$ for $L_{con}^u$. We found that $\lambda$ equals 0.7$\sim$1 works fine. Therefore, It is simple to choose hyper-parameters for our method, just tune $\lambda$ for $L_{con}^u$, which generalizes well on all datasets.
  * Overall, just keep the weights of old and new loss to 1, and the weight for $L_{con}^u$ be slightly less than 1 (e.g, 0.7), our methods consistently achieves SOTA on all datasets. As a result, the hyper-parameter tuning is relatively simple.



**References:**

[R1]. Self-organization in a perceptual network. Ralph Linsker. 1988.



Thanks for your feedback. If you still have questions, please feel free to continue the discussion.
Thanks for the authors' response. The lack of a solid theoretical foundation and not quite stable debias mechanism affect model reliability. I'm also not quite sure how much hyper-parameters tuning would further affect its stability. Even though they're all ones, but these may coincidentally be optimal.
Dear Reviewer cVL6,

Thanks for your feedback. As Reviewer TLvc and jUDg mentioned, our responses have addressed their concerns.

We were wondering if our latest responses have addressed your concerns. If you have further specific questions on the weaknesses of the method, feel free to raise them and we are willing to discuss them with you. We look forward to and appreciate your feedback.

Thank you once again.
Dear Reviewer TLvc,

We were wondering if our latest responses have addressed your concerns about weakness 2 and 4. If you have any other specific questions, feel free to raise them and we are willing to discuss them with you. We look forward to and appreciate your feedback.

Thank you once again.
Dear Reviewer TDQJ,

We were wondering if our latest responses have addressed your concerns. If you have any other specific questions, feel free to raise them and we are willing to discuss them with you. We look forward to and appreciate your feedback.

Thank you once again.
Dear Reviewer TLvc,

We are glad that our responses have resolved your concerns. Regarding the issues **W2 and W4**, which you are particularly concerned about, we have further organized our responses, summarized as follows:

* W2: Too many hyper-parameters and loss terms.

  * **Understanding of the method.** Our method consists of three parts: $L_{Happy}=L_{new}+L_{old}+\lambda L_{con}^u$.

    1. Cluster new classes: $L_{new}$, with self-training and mitigation of prediction bias for new class.

    2. Mitigate forgetting old classes: $L_{old}$, with hardness-aware prototype sampling to alleviate hardness bias.

    3. Contrastive learning: $L_{con}^u$ to generally ensure feature representations.

  * **Hyper-parameter tuning.** For $L_{new}$ and $L_{old}$, our results show that a 1:1 weight ratio is optimal, because both old and new classes matter and we need to maintain a balanced focus on both new and old classes as a whole (other weight ratios of $L_{new}$ and $L_{old}$ will result in a 5% to 10% performance degradation). So keep the weight ratio of $L_{new}$ and $L_{old}$ to 1, and the only parameter for tuning is $\lambda$ for $L_{con}^u$. We found that $\lambda$ equals 0.7$\sim$1 works fine. Therefore, It is simple to choose hyper-parameters for our method, just tune $\lambda$ for $L_{con}^u$, which generalizes well on all datasets.

  * **Overall.** The three objectives are complementary and non-conflicting, where each is essential, and they collectively improve classification across all classes.

* W4: Provide evidence to show that two issues are mitigated.

  * Here, we provide more intuitive examples.
  * **Prediction bias** refers to the bias that the model tends to predict new class samples to the old ones. **Metric:** We compute the proportion of new classes' samples misclassified as old classes. **Example:** In our method, the ratio of new samples misclassified to old ones decreases from 63.25% to 10.20%, which means that Happy could largely mitigate prediction bias, with better separation between old and new classes.
  * **Hardness bias** refers to the bias that models have weaker classification and more severe forgetting on more hard old classes. **Metric:** We compute the accuracy of the hardest class and the accuracy variance among all old classes. **Example:** On CIFAR100, our method improves the hardest accuracy from 65.10% to 70.23%, and the variance decreases from 23.04% to 10.33%, which means our method greatly mitigates the forgetting of hard old classes and alleviates the extent of imbalance caused by hardness bias.



We hope our further responses regarding **W2 and W4** have clearly resolved your concerns. We were wondering whether our paper could be re-evaluated considering these further explanations on **W2 and W4**.

If you have any other questions, we would be happy to further discuss with you. We really appreciate your feedback.
Dear Reviewer TDQJ,

Thank you very much for your feedback. Regarding the weaknesses you've highlighted, we have further organized our responses and summarized them as follows:

* W1: Lack of a comprehensive social impact analysis.
  * Here, We have included a more detailed discussion on societal impacts, supplemented with specific examples.
  * **Medical risks:** Our method could be applied in the medical field, where biases in learned categories might affect the discovery or diagnosis of new diseases. Over time, these errors could increase, potentially delaying critical medical interventions.
  * **Fairness issues:** Regarding different genders and populations, our model may learn unfair biases from the labeled data, which could then be transferred to new domains. This could perpetuate inequalities within the model's predictions.

* W2: Lack of a theoretical foundation.
  * The theoretical foundation of our method is the **InfoMax principle** [R1], namely **maximizing the mutual information (MI) between the inputs and outputs of a system.** Here inputs and outputs denotes the feature $Z$ and model predictions $Y$.
  * For the self-training problem of learning new classes in C-GCD, we employ the maximizing the mutual information (MI) theory. Specifically, the objective can be expressed as $\max I(Y, Z) = H(Y) - H(Y|Z)$, where $Y$ represents the labels and $Z$ the features, with $I$ and $H$ denoting mutual information and entropy, respectively.
  * The proposed entropy regularization on model predictions $Y$ is the minus of first term $-H(Y)$, while the self-distillation objective aims to encourage the model to predict more confidently, resulting in lower prediction entropy, namely the second term of minimizing $H(Y|Z)$, namely maximizing $-H(Y|Z)$.
  * Overall, our method follows the Infomax principle and aims to maximize mutual information, to ensure the quality of self-learning.
* W3: There are assumptions that may limit the general applicability, especially in different distributions.
  * We need to clarify here that our method does not rely on explicit assumptions or predefined ratios of new to old categories in previous methods, thereby enhancing its generalizability. This has been demonstrated in our experiments, including the distribution shift experiments detailed in the Rebuttal PDF.
* W4: The accuracy of the new class still fluctuates, which can affect model reliability and user trust in real-world deployments.
  * Our method significantly mitigates fluctuations. For example, in CIFAR 100, the gap in accuracy among classes has been reduced from 17.3% to 6.3%. This ensures the reliability and stability of our approach in real-world deployments.



**References:**

[R1]. Self-organization in a perceptual network. Ralph Linsker. 1988.



Hope these further responses address your concerns about the weaknesses part. If you have further questions, please let us know. We are willing to have more discussions with you. Thanks for your participation.
Dear Reviewer cVL6,

Thanks for your feedback. As Reviewer TLvc and jUDg mentioned, our responses have addressed their concerns. Additionally, we have further explained some of your concerns in our previous responses. If you still have some specific questions, please let us know. We are willing to discuss further.

Thank you for your participation.
Dear Reviewer cVL6,

Thanks for your feedback. Considering the "weaknesses of the other Reviewers", we have given responses point by point in the rebuttal. Here, we summarize our responses to the weaknesses pointed out by all reviewers:

* **Hyper-parameters tuning.** Please refer to **General Response to Common Concern 1** and **response to you in W1**. Our method requires three hyper-parameters, and **we only need to set all weights to 1**, which is generalizable and could perform well across all datasets. Therefore, the parameter tuning is simple and easy to implement.

* **Complexity of the method.**

  * Firstly, we have demonstrated the necessity of each module, as shown in Table 5 of the main manuscript.
  * Secondly, we have provided the computational overhead for our proposed modules (see **responses to you in W2 and W3**). The cluster-guided initialization, entropy regularization, and hardness modeling that we propose consume less than 1% of time and memory resources, which means that they are very efficient and effective.

* **Scalability of the method.** All continual learning methods experience a linear increase in computational demand with the number of classes and stages. Compared to previous replay-based methods, our approach saves disk storage by a factor of 1/1000, as detailed in the **response to you in W2**.

* **Generalization abilities of the method.**

  * We have conducted evaluations on 15 corrupted and shifted distributions of CIFAR100-C, as in the **General Response to Common Concern 1** and **Rebuttal PDF**.

  * Furthermore, previously, we have also conducted experiments on two more fine-grained datasets: **Stanford Cars** and **FGVC Aircraft** datasets, with the results as follows:

    | Standford Cars |    All    |    Old    |    New    |
    | :------------: | :-------: | :-------: | :-------: |
    |      GCD       |   47.00   |   47.73   |   42.61   |
    |    MetaGCD     |   54.67   |   55.28   |   50.95   |
    |  Happy (Ours)  | **62.79** | **63.68** | **57.34** |

    | FGVC Aircraft |    All    |    Old    |    New    |
    | :-----------: | :-------: | :-------: | :-------: |
    |      GCD      |   42.95   |   44.35   |   33.38   |
    |    MetaGCD    |   47.16   |   48.61   |   38.23   |
    | Happy (Ours)  | **53.10** | **53.81** | **48.71** |

  * To conclude, our method achieves remarkable performance compared to the previous sota on 6 datasets and 15 unseen distributions, showcasing the strong generalization abilities of Happy.

* **Adapting to SoTA  methods.** We have adapted our method to two sota methods: SPTNet and InfoSieve. Please refer to **response to you in Q2** for more details. We will add all results of adapting Happy to all three methods you listed in the final version. **We will also add citations of the three methods: SPTNet TIDA, and InfoSieve in our reference list.**



Thank you again for your feedback. We hope our responses and extensive experiments have addressed your concerns. If you have any further questions, we welcome continued discussion. Thank you once again for your review.
Dear Reviewer jUDg,

We are pleased that our responses have addressed your concerns. Thank you very much for your insightful suggestions and valuable efforts, which are crucial for enhancing the quality of our paper.

Thank you once again.
Thank you for your response. It has addressed some of my concerns, so I will raise my rating to 6.
Thanks to the author for answering my question, but it failed to solve my doubts about the weakness of the method
I appreciate author's feedback. My concern are resolved. I keep my original score.
Dear Reviewer TLvc,

Thanks very much for your time and valuable comments.

In the rebuttal period, we have provided detailed responses to all your comments and questions point-by-point for the unclear presentations.

Any comments and discussions are welcome! 



Thanks for your attention and best regards,

Authors of Submission 5010.
Dear Reviewer TDQJ,

Thanks very much for your time and valuable comments.

In the rebuttal period, we have provided detailed responses to all your comments and questions point-by-point for the unclear presentations.

Any comments and discussions are welcome! 



Thanks for your attention and best regards,

Authors of Submission 5010.
I read all the comments of the other reviewers. I agree with all weaknesses of the other Reviewers. I hold the same opinions.
Dear Reviewer jUDg,

Thank you very much for your feedback. Here, we provide a detailed explanation of W2: how to compute the marginal probability:

* In this paper, marginal probability for old $\overline p_{old}$ and new classes $\overline p_{new}$ refer to the sum of  $\overline{\boldsymbol{p}}\in R^{K}$ (marginal probability of all samples in a batch) at the indices of new and old classes, which is calculated **along the class dimension**, **rather than along the sample dimension**.
* Here both $\overline p_{old}$ and $\overline p_{new}$ are **scalars**, while $\overline{\boldsymbol{p}}\in R^{K}$ is a K-dim **vector**, $K=K_{old}+K_{new}$ denotes the number of total classes of the current stage.
* As a result, **we do not need to separate old class samples from new class samples when we compute $\overline p_{old}$ and $\overline p_{new}$**.
* Specifically, for a batch of samples, let $\boldsymbol p_i$ denotes the prediction of the i-th sample. We first average the model predictions for **all samples** along the batch dimension, **without distinguishing between new and old classes samples**, namely $\overline{\boldsymbol{p}}=\frac{1}{|B|}\sum_{i\in B}\boldsymbol p_i$. Here, both $\overline{\boldsymbol{p}}$ and $\boldsymbol p_i$ are $K$ dim vectors. Then we calculate the marginal probabilities for old and new classes **along the class dimension**, i.e., $\overline p_{old}=\sum_{c=1}^{K_{old}}\overline{\boldsymbol{p}}[c]$ and $\overline p_{new}=\sum_{c=K_{old}+1}^{K}\overline{\boldsymbol{p}}[c]$ (so both $\overline p_{old}$ and $\overline p_{new}$ are scalars and $\overline p_{old}+\overline p_{new}=1$). Note that  $\overline{\boldsymbol{p}}[c]$ represents the c-th index of the vector $\overline{\boldsymbol{p}}$.
* **Here is one example**. On the first stage of CIFAR100, there are 50 old classes and 10 new classes, so the dimension of model predictions is $K=60$. We first compute the average of model predictions for all samples in a batch and obtain the **vector** $\overline{\boldsymbol{p}}$. Then we sum the first 50 dimensions of $\overline{\boldsymbol{p}}$ to obtain the **scalar** $\overline p_{old}$ and sum the last 10 dimensions to obtain the **scalar** $\overline p_{new}$.
* To conclude, $\overline p_{old}$ and $\overline p_{new}$ are marginal probabilities of **all** samples **along the dimension of old/new classes**, rather than the marginal probability of **old/new samples**. So we do not need to determine whether a sample belongs to an old class or a new class.

We really appreciate your feedback. If you have any additional questions, please let us know. We greatly respect your insights and enjoy the discussions with you. Thanks again!
Thank you for your response. Some of my concerns have been addressed. However, regarding W2, the author may have misunderstood my question. What I want to know is how to determine whether a sample belongs to an old class or a new class within a session when all labels are unavailable. The marginal probability for old and new classes can only be computed individually if there is a way to accurately separate old class samples from new class samples in each session in an unsupervised manner.