DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discovery

Authors: Yuanpei Liu, Kai Han
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Generalized Category Discovery, Semi-supervised Learning, Out-of-distribution Detection

Abstract:
In this paper, we tackle the problem of Generalized Category Discovery (GCD). 
Given a dataset containing both labelled and unlabelled images, the objective is to categorize all images in the unlabelled subset, irrespective of whether they are from known or unknown classes. 
In GCD, an inherent label bias exists between known and unknown classes due to the lack of ground-truth labels for the latter. 
State-of-the-art methods in GCD leverage parametric classifiers trained through self-distillation with soft labels, leaving the bias issue unattended. 
Besides, they treat all unlabelled samples uniformly, neglecting variations in certainty levels and resulting in suboptimal learning. 
Moreover, the explicit identification of semantic distribution shifts between known and unknown classes, a vital aspect for effective GCD, has been neglected. 
To address these challenges, we introduce DebGCD, a Debiased learning with distribution guidance framework for GCD. 
Initially, DebGCD co-trains an auxiliary debiased classifier in the same feature space as the GCD classifier, progressively enhancing the GCD features. Moreover, we introduce a semantic distribution detector in a separate feature space to implicitly boost the learning efficacy of GCD. Additionally, we employ a curriculum learning strategy based on semantic distribution certainty to steer the debiased learning at an optimized pace. 
Thorough evaluations on GCD benchmarks demonstrate the consistent state-of-the-art performance of our framework, highlighting its superiority. Project page: [https://visual-ai.github.io/debgcd/](https://visual-ai.github.io/debgcd/)

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730460677864

thread_type:
official_review

<thread_object>
creator: reviewer_q4nm

<content>
summary:
The authors propose a novel framework called Debiased Learning with Distribution Guidance (D2G) for the GCD task, which introduces a debiased learning paradigm to optimize the clustering feature space and learns a semantic distribution detector to enhance the learning effect of GCD. Besides, D2G propose a curriculum learning mechanism that steers the debiased learning process to effectively mitigate the negative impact of uncertain samples. The authors evaluate the method on the public GCD benchmarks to demonstrate the effectiveness.

strengths:
D2G considers both label bias and semantic shift to address the challenging GCD task. It’s a novel idea to mark the first exploration of these aspects.
D2G effectively incorporates all components into a unified framework and can be trained in a single stage without any additional computation burden.
The authors conduct extensive experimentation on public GCD benchmarks to demonstrate its effectiveness.

weaknesses:
1. The reason for using OOD techniques to solve GCD task is not clear because the objectives of these two tasks are different. The motivation for using MLP projection network to solve this problem needs further explanation.
2. From the experimental results, the performance improvement of the method is not significant, especially on the CUB dataset. Besides, there are few comparison methods on the ImageNet-1K dataset, which can lead to unreliable comparison results.
3. Some hyperparameters lack ablation experiments to verify that the experimental method is optimal, including the number of layers in MLPs, the loss weights, and so on.

questions:
1. I wonder whether the GCD method is sensitive to certain categories, resulting in limited performance improvement on some datasets. Perhaps the authors can design some experiments to test it.
2. Is it better to use vision-language pre-trained large models such as CLIP to solve label bias problems, as CLIP contains a lot of pre-trained knowledge for new categories.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532905599

thread_type:
official_review

<thread_object>
creator: reviewer_59sw

<content>
summary:
This paper presents the D2G (Debiased Learning with Distribution Guidance) framework for addressing the Generalized Category Discovery (GCD) problem. GCD is challenging due to label biases and semantic shifts between known and unknown categories. The D2G framework introduces a debiased learning paradigm, a semantic distribution detector, and a curriculum learning approach based on distribution certainty to address these issues. Extensive experiments demonstrate D2G’s superiority over existing GCD methods on various benchmarks.

strengths:
1. The paper is well-organized, with clear explanations of technical details.
2. The introduction of a debiased learning framework specific to GCD with a multi-feature distribution approach is innovative.
3. The technical contributions are well-structured and effectively evaluated. The integration of auxiliary debiased learning, semantic detection, and curriculum learning reinforces the model's performance.

weaknesses:
1. The variation in results from the GCD benchmarks can be very large, so it is important to report all results as well as the error bars from the three independent runs, as SimGCD does in its Supplementary Information.
2. While the authors claim that D2G does not introduce additional computational burdens during inference, a more detailed analysis of the training time and computational costs associated with the auxiliary components would be valuable.

questions:
1. What strategies do authors envision to reduce potential overfitting during assisted debiasing learning, especially when utilizing limited unlabeled data on fine-grained datasets?
2. Based on Table 4, it can be concluded that the effect of label debiasing is not very good. Have the authors considered not using the label debiasing strategy? For example, what are the results for "w/o debiased learning, w/ auxiliary classifier, w/o semantic dist. learning, w/o dist. guidance"?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607922920

thread_type:
official_review

<thread_object>
creator: reviewer_m5tq

<content>
summary:
This paper propose D2G a novel framework that addresses the challenging GCD task. Several new paradigms and mechanisms like debias learning in this framework enhance the model’s performance. Combined with these, the method proposed demonstrates its effectiveness and achieves superior performance on broad benchmark.

strengths:
1. The paper writing is clear and easy to understand.
2. The topic of the article is of significant theoretical and practical importance, addressing a gap in the existing literature.
3. The paper clearly outlines the shortcomings of previous studies and results section is logically organized.
4. The proposed framework for DCG is novelty. Various incremental mechanisms make sense to me.

weaknesses:
1. There is a error in fig.1 (d), the brown dish line is invisible.
2. The hyperparameters in Eq. 14 are empirical values or obtained through experiments? If latter, I believe the authors should include some ablation studies for clarification.
3. In Tab. 2, the performance of D2F is suboptimal compared to InfoSieve, but it lacks a specific analysis. Could the authors provide further details on this?
4. The impact of the various method proposed, such as Debias learning and Auxiliary Classifier, should be evaluated through ablation studies on a broader dataset. The paper currently reports results only on the Stanford dataset. Do authors validate only on this dataset?

questions:
please refer to weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631337950

thread_type:
official_review

<thread_object>
creator: reviewer_ycci

<content>
summary:
The paper claims that existing GCD methods suffer from label bias, fail to account for differences in uncertainty, and do not address semantic distribution shifts. To address these issues, the author proposes D2G framework, which comprises Semantic Distribution Detection and Auxiliary Debiased Learning. The Semantic Distribution Detection module treats each labeled category as a separate binary classification, using the prediction confidence score obtained to filter and scale the debiased loss. The additional loss introduced by these components can be directly integrated with SimGCD and these modules can be entirely discarded during inference.

strengths:
1. The motivation behind addressing label bias is sound to me. Previous methods apply soft supervision to unlabeled data, which results in weaker supervision for unknown classes. The proposed method aligns well with this motivation.
2. The approach achieves performance improvements demonstrating its effectiveness.
3. The framework is efficient in inference.
4. The writing is clear and easy to follow.

weaknesses:
1. The distribution detector functions as multiple independent binary classification, so there is no competition between categories. It serves two purposes: first, it uses negative class confidence scores to filter out likely unknown classes in the final $L^u_{adl}$; second, it imposes stronger supervision on samples with higher uncertainty. For the first purpose, is there a significant difference in effectiveness compared to using self-entropy to filter unknown samples? Self-entropy would seem a more natural and straightforward metric, yet the author does not analyze the benefits of this one-vs-all design. For the second, Equation 10 imposes stronger pseudo one-hot supervision on samples deemed uncertain by the distribution detector. For example, if an unknown class is close to a known class, the loss will be reduced by $d_i$. The ablation study indicates that this yields significant performance gains, but lacks detailed analysis and discussion.
2. The D2G framework finetunes more parameters than SimGCD, which only trains the last block. Since D2G builds on SimGCD, it would be more meaningful to compare performance under the same training setup. The authors did not provide this.
3. There is a performance drop compared to the baseline on Herbarium19.
4. Since all introduced modules can be discarded during inference, I think the key to performance improvement likely lies in the enhancement of the discriminability of DINO CLS token. However, the authors provide minimal discussion on this aspect.
5. The description of the ablation studies is not sufficiently clear, and there is a lack of discussion between experiments. Specifically, regarding debiased learning, all debiased losses could theoretically be applied directly to the original classifier in SimGCD. It is unclear why the addition of a second classifier is necessary for effective performance. Additionally, I would like to know the impact on performance of removing the MLP prior to the OVA module.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732559484960

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response(1/2)

comment:
We thank all the reviewers for their insightful comments and positive feedback. Reviewer YCCi marked that **"the motivation behind addressing label bias is sound to me,"** and noted that **"the proposed method aligns well with this motivation"** and that **"the framework is efficient in inference."** Reviewer M5TQ commented that our work is **"addressing a gap in the existing literature"** and highlighted that **"the proposed framework for GCD is novel."** Additionally, Reviewer 59SW stated that **"the introduction of a debiased learning framework specific to GCD with a multi-feature distribution approach is innovative"** and affirmed that **"the technical contributions are well-structured and effectively evaluated."** Reviewer Q4NM also noted that **"it's a novel idea."** Furthermore, the reviewers agreed that our paper is **"clear and easy to follow"** (Reviewer YCCi), **"clear and easy to understand"** (Reviewer M5TQ), and **"well-organized, with clear explanations of technical details"** (Reviewer 59SW).

We have carefully addressed all concerns raised by the reviewers. First, we provide a **general response** to the shared concerns and critical points. We then address the individual concerns of each reviewer following their comments. We will also further strengthen the final manuscript based on the reviewers' concluding feedback.

**Code:** A well-documented code together with all trained models will be made public.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732559599499

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response(2/2)

comment:
### **Q1. Motivation and  ablation about the MLP before OVA (Reviewers YCCi and Q4NM)**
We appreciate the reviewers' suggestions and have incorporated the requested experiments. In the context of OOD, our objective is not to differentiate between multiple distinct unknown categories, as in GCD; rather, we aim to distinguish all unknown samples from the known classes, effectively framing this as a binary classification problem. This requirement led us to introduce a different embedding space that is more suitable for this task, achieved by simply adding an MLP projection head. 
To validate the impact of the number of layers in the MLP, we conducted an ablation study on the SSB benchmark regarding GCD and OOD performance, as shown in the Table A1 and Table A2. We observe that the average GCD performance across all categories of D2G gradually improves as the number of MLP layers increases from 0 to 5. A similar trend is evident in the OOD performance. However, extending the MLP to 7 layers results in little to no improvement in performance. In our implementation, we adopted a 5-layer MLP in our framework.


**Table A1. The GCD performance using different number of MLP layers.** 
| | CUB | SCars | Aircraft|Average
|--------------|--------------|--------------|--------------|--------------
**MLP layer**| **All/Old/New** | **All/Old/New**| **All/Old/New** |**All**
0|63.6/75.2/57.8|62.3/76.2/54.1|59.6/62.2/58.3|61.8
1|64.9/71.6/61.6|63.9/80.2/56.0|60.7/63.7/59.2|63.1
3|66.0/**73.5**/62.3|64.7/**82.2**/56.2|61.1/64.2/59.5|63.9
5|**66.3**/71.8/**63.5**|**65.3**/81.6/**57.4**|61.7/63.9/**60.6**|**64.4**
7|65.8/72.0/62.7|64.8/80.5/57.3|**61.9**/**65.2**/60.3|64.1

**Table A2. The OOD performance using different number of MLP layers.** 
| | CUB | SCars | Aircraft|Average
|--------------|--------------|--------------|--------------|--------------
**MLP layer**| **AUROC** | **AUROC**| **AUROC** |**AUROC**
0|80.1|81.3|79.0|80.1
1|84.2|85.1|83.5|84.2
3|86.9|87.4|85.9|86.7
5|86.8|**89.6**|86.3|87.6
7|**87.2**|89.4|**87.1**|**87.9**


### **Q2. Further ablation about the loss weights (Reviewers M5TQ and Q4NM)**
We appreciate the suggestions from the reviewers and have included the requested experiments. Indeed, we do not extensively tune the hyperparameters but intuitively set them based on existing literature and our hypothesis. Our rationale for selecting values for the loss weights is as follows:
For $\lambda_{sdl}$, we take inspiration from the previous literature using OVA classifier[1]. In the paper, the model is fine-tuned with a learning rate of $10^{-3}$ , while the learning rate in the SimGCD baseline is 0.1 (which is 100 times larger than $10^{-3}$). To achieve a similar learning effect, as validated in OVA, we scale our $\lambda_{sdl}$ value from 1 down to (1/100). Therefore, we set $\lambda_{sdl} = 0.01$ by default. 
For $\lambda_{adl}$, the weight of the debiased classifier, we expect it to play an important role similar to that of the original GCD classifier (where the loss weight is set to 1). Thus, we have defaulted this value to 1.0. 
After determining the default values, we conducted experiments on the SSB benchmark regarding the two loss weights by exploring values around the defaults. For $\lambda_{sdl}$, the range was (0.005, 0.01, 0.02). As for $\lambda_{adl}$, the range was (0.5, 1.0, 2.0). The impact of $\lambda_{sdl}$ is detailed below in Table A3, with $\lambda_{adl}$ set to 1.0. The impact of $\lambda_{adl}$ is illustrated below in Table A4, with $\lambda_{sdl}$ set to 0.01. The results are in line with our hypothesis, indicating that our selected hyperparameters are indeed reasonable.

**Table A3. The ablation results about $\lambda_{sdl}$ on the SSB benchmark.**
||CUB | SCars | Aircraft|Average
|--------------|-------------|--------------|--------------|--------------
|$\lambda_{sdl}$| **All/Old/New** | **All/Old/New**| **All/Old/New** |**All**
|0.02|65.5/**73.2**/61.6|64.3/79.2/57.1|60.6/63.5/59.1|63.5
|0.01|**66.3**/71.8/**63.5**|**65.3/81.6/57.4**|61.7/63.9/**60.6**|**64.4**
|0.005|65.8/72.4/62.5|64.9/81.2/57.0|**62.1/65.4**/60.3|64.3


**Table A4. The ablation results about $\lambda_{sdl}$ on the SSB benchmark.**
||CUB | SCars | Aircraft|Average
|--------------|-------------|--------------|--------------|--------------
|$\lambda_{adl}$| **All/Old/New** | **All/Old/New**| **All/Old/New** |**All**
|0.5|64.3/**72.2**/60.3|63.6/79.3/56.1|60.2/63.5/58.6|62.7
|1.0|**66.3**/71.8/**63.5**|**65.3**/81.6/**57.4**|**61.7/63.9/60.6**|**64.4**
|2.0|65.5/70.8/62.8|64.1/**83.0**/55.0|60.4/63.5/58.8|63.3


*[1] Saito, Kuniaki, and Kate Saenko. "Ovanet: One-vs-all network for universal domain adaptation." ICCV 2021.*
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the challenges of inherent biases and overlooked semantic distribution shifts in Generalized Category Discovery (GCD). It introduces a novel model incorporating a debiased auxiliary classifier, a semantic distribution detector, and a curriculum learning strategy to enhance the learning process. Experimental evaluations demonstrate that the proposed D2G model achieves state-of-the-art performance on GCD benchmarks. Reviewers provided positive feedback and expressed satisfaction with the authors' responses and revisions.