HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts

Authors: Hongjun Wang, Sagar Vaze, Kai Han
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Generalized Category Discovery

Abstract:
Generalized Category Discovery (GCD) is a challenging task in which, given a partially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones. In this paper, we challenge a remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method to handle GCD when the unlabelled data also contains images from different domains to the labelled set. Our proposed `HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a specialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach. Finally, we construct a benchmark from corrupted fine-grained datasets as well as a large-scale evaluation on DomainNet with real-world domain shifts, reimplementing a number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730089876547

thread_type:
official_review

<thread_object>
creator: reviewer_jy36

<content>
summary:
This paper introduces a new problem setting: Generalized Category Discovery (GCD) with domain shift. The authors leverage techniques from domain adaptation and curriculum learning to propose a new method called HiLo. Comprehensive experiments on the proposed benchmark demonstrate substantial improvements.

strengths:
The paper proposes a new problem setting and proposes a HiLo, which combines multiple techniques from domain adaption and achieves better results.

weaknesses:
1. The novelty of the method appears limited, as it seems to combine various techniques from different domains.

2. The comparison with UniOT should be included in the main results. Since the proposed setting is similar to universal domain adaptation, it is essential to compare methods from both domains in the main results.

Minor： 

Missing citation for the following important paper

[1] Rastegar et al. Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery. NeurIPS 2023.

[2] Gu et al. Class-relation Knowledge Distillation for Novel Class Discovery. ICCV2023.

questions:
Please clarify the novelty of the proposed method, and include more comparisons with UinOT in the main results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540320207

thread_type:
official_review

<thread_object>
creator: reviewer_l4ea

<content>
summary:
The paper introduces the HiLo framework, a learning method aimed at tackling Generalized Category Discovery (GCD) under domain shifts. HiLo addresses challenges in categorizing both seen and unseen categories across distinct domains within partially labeled datasets, leveraging a multi-faceted approach: mutual information minimization to separate domain and semantic features, PatchMix for augmented domain adaptation, and a curriculum learning strategy. The proposed method is evaluated on synthetic and real-world domain-shift datasets, showing substantial improvements over existing GCD models.

strengths:
1. The paper presents an innovative GCD approach by combining mutual information minimization with domain-specific data augmentation and curriculum learning to handle domain shifts effectively.
2. Extensive evaluation on both synthetic (SSB-C) and real-world (DomainNet) benchmarks demonstrates the model's robustness and its superiority over baseline GCD models, especially under domain-shifted conditions.

weaknesses:
1. In the "Problem statement," the following sentence is unclear: "The objective of GCD is ... with singleton cardinalities for the latter." The author needs to differentiate between the GCD task setting and the domain shift GCD task setting, so this statement should be revised for clarity and precision.
2. The font sizes of the tables are not standardized, and the font in table 2 is too small.
3. I'm curious as to how many runs each of the authors' experimental results were derived from, and given that the differences in the results of the GCD benchmark tests can be very large, the authors should have listed the error lines generated by the three independent runs.

questions:
1. For the experimental results of the ORCA method, what is the backbone used by the authors?
2. Were any curriculum learning alternatives considered, such as adaptive weighting based on difficulty or dynamic sample weighting? A brief discussion on these choices would clarify why the current approach was favored.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684786804

thread_type:
official_review

<thread_object>
creator: reviewer_rnzq

<content>
summary:
Generalized Category Discovery (GCD) is a challenging task where, given a partially labeled dataset, the model must classify all unlabeled instances. This paper introduces a new task and method to handle the GCD problem when the unlabeled data contains images from different domains. In terms of the method, the HiLo architecture and learning framework involves extracting "low-level" (early layers) and "high-level" (late layers) features from a vision Transformer and decoupling domain and semantic features by minimizing the mutual information between the two sets of features. The PatchMix contrastive learning method is introduced into the GCD task, with its objective function extended to enable the utilization of both labeled and unlabeled data for training. Curriculum learning is adopted, gradually increasing the sampling probability weight of samples predicted to be from unknown domains to enhance the model's robustness to domain shifts. Experiments are conducted on the DomainNet and the SSB-C benchmark datasets constructed based on the Semantic Shift Benchmark (SSB). The experimental results show that HiLo significantly outperforms existing category discovery models, validating the effectiveness of the method.

strengths:
1. The HiLo architecture extracts features from different layers of the vision Transformer and decouples domain and semantic features by minimizing mutual information. This feature processing method, based on the neural network hierarchical structure and information theory, provides a more effective feature representation for category discovery in the presence of domain shifts and avoids the problem of feature confusion in traditional methods.

2. The PatchMix method is introduced into the GCD task and innovatively extended. By adjusting its objective function, it can adaptively utilize labeled and unlabeled data for training. This extension not only combines the advantages of data augmentation but also flexibly adjusts the learning process according to the nature of different data, enhancing the model's ability to learn data from different domains and categories.

3. The curriculum learning method is employed, which dynamically adjusts the sampling probability weights according to the difficulty of samples and the unknown degree of domains. This strategy of gradually introducing samples from easy to difficult conforms to the learning law, enabling the model to better adapt to the challenges brought by domain shifts and improving the model's convergence speed and robustness to complex data distributions.

4. In terms of method design, innovative technical architectures and learning strategies are used, as well as theoretical analyses to verify their effectiveness. From the theoretical derivation of the target error to the analysis of the roles of different components, a solid theoretical foundation is provided for the innovation of the method, demonstrating the advantage of the close integration of theory and practice.

weaknesses:
1. In HiLo, features are disentangled by assuming that features from different layers represent domain and semantic information, respectively and minimizing the mutual information based on this assumption. However, this assumption may oversimplify the complexity of feature representation in neural networks. In fact, features from different layers may be a mixture of multiple types of information. Simply defining the early layers as domain features and the late layers as semantic features may not be entirely accurate, which may lead to incomplete feature disentanglement in some complex data distributions and affect the performance and generalization ability of the model.

2. The introduction and extension of PatchMix in the GCD task is an innovation, but it also brings some problems. The adjustment of its objective function and its application on different data increases the complexity of the model. When dealing with data with large domain differences, it is a challenge to determine the mixing proportion and application method accurately. If not handled properly, it may introduce too much noise or incorrect information, which may instead interfere with the learning process of the model and reduce the classification performance.

3. In the curriculum learning method, the adjustment parameters of the sampling probability weights need to be selected through the validation set, which increases the dependence of the model on specific datasets. Moreover, for different datasets and tasks, the optimal values of these parameters may vary greatly, and the model cannot adaptively determine these parameters. If these parameters cannot be correctly selected in a new dataset or task, curriculum learning may not be able to play its intended role. It may even have a negative impact on the learning of the model.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713284469

thread_type:
official_review

<thread_object>
creator: reviewer_6eyx

<content>
summary:
This paper introduces a new challenge for Generalized Category Discovery, which requires model to categorize unlabeled data in the presence of domain shifts. Traditional GCD methods assume all images come from the same domain, which leads to a significant performance drop under domain shifts. The proposed HiLo framework explicitly disentangles semantics and domain, achieving domain adaptation in GCD through patchmix and curriculum learning. Experimental results show performance improvements, validating the effectiveness of the approach.

strengths:
1. The paper presents a new, practically meaningful, and challenging setting, and constructs corresponding datasets.
2. The domain-semantic disentangled design is well-reasoned, clearly aligning the motivation.
3. The proposed approach demonstrates significant performance improvement on SSB-C.
4. The writing is clear and easy to follow.

weaknesses:
1. The performance gain on DomainNet is considerably smaller than on SSB-C, and the improvement over methods like SimGCD, which does not account for domain shift, is modest. This indicates limited robustness on various domain shifts and fails to highlight the advantages of the proposed approach.
2. The method is sensitive to certain hyperparameters, and $r^{'}$ does not exhibit consistent performance across the original and new domains.
3. The approach of decoupling domain and semantics is derived from [1], and the use of patchmix for domain adaptation is adapted from [2]. The curriculum learning strategy is also straightforward. Overall, the method seems to be an assembly of prior works, lacking substantial novelty.
4. There is no analysis of the disentangled domain and semantic features, such as distribution visualizations. This would help illustrate the effectiveness of the disentanglement.
5. In line 287, same representation loss $L^{rep}_s$ on both domain and semantic features is confusing. This approach may lead domain features to capture information beyond true domain characteristics. It would be valuable to see t-SNE visualizations of domain features, semantic features, and their combination. The author does not provide a corresponding discussion.
6. Line 313 mentions using pre-trained DINO to obtain $z_d$, but previously $z_d$ is associated with a projection head. If the projection head is discarded, then $z_d$ will always be identical in different time steps. If it is retained, the term “pretrained” is confusing. This needs clarification.
7. The ablation study is somewhat unclear. For instance, in row (5) where only deep features are used, does this mean all other designs related to the shallow feature $z_d$ are also omitted? This also needs clarification.

Reference

[1] Learning deep representations by mutual information estimation and maximization

[2] Patch-mix transformer for unsupervised domain adaptation: A game perspective

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732515080255

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank reviewers for their constructive and valuable feedback. We are encouraged that the reviewers find our paper to be **"clear and easy to follow"** (Reviewer 6eyX), with a **"well-reasoned"** (Reviewer 6eyX) and **“innovative approach”**, while presenting **"innovative technical architectures"** with **"solid theoretical foundation"** (Reviewer RNZQ). The reviewers agreed that our contributions are significant, noting our **"new, practically meaningful, and challenging setting"** (Reviewer 6eyX, jy36) and **"effective feature representation"** learning method (Reviewer RNZQ). We are glad that reviewers also found our evaluation **"demonstrates the model's robustness"** (Reviewer L4eA) and that reviewers acknowledge our **"significant performance improvement on SSB-C"** (Reviewer 6eyX) and effectiveness in handling **"domain shifts"** (Reviewer jy36, L4eA).

We have carefully addressed all concerns raised by the reviewers. First, we provide a **general response** to shared concerns or critical points. We also address the reviewers’ individual concerns after their comments. We have also revised our manuscript based on the comments from the reviewers.


**Novelty of proposed model (Reviewer 6eyX, jy36)**

In this paper we have proposed both a novel (but intuitive) **problem setting** as well as a new **solution** to tackle it. The setting is a challenging and practical image classification problem, as noted by Reviewers 6eyX, RNZQ and jy36. Here, a model must jointly learn from labeled and unlabeled images, with the goal of clustering all unlabeled images into distinct categories. Notably, the unlabeled images may come from different **categories** and **domains** to the labeled set. Though simple and practical, only subsets of this problem have been addressed in prior literature: particularly in the Unsupervised Domain Adaptation (UDA) and Generalized Category Discovery (GCD) fields. 

As such, we are transparent in the paper that aspects of our solution have been introduced in prior work (Line 247-248), notably the PatchMix approach. However, we also show that simply ensembling SoTA methods from UDA and GCD does not yield substantial gains in our challenging setting (see Table 5). 

Particularly, we find that it is critical to find the optimal recipe for the current task, and that without the following innovations the method does not work:
- A new PatchMix formulation for new classes: PatchMix was developed for UDA and naive application of it does not allow the method to discover new classes. Instead, we must devise a PatchMix-based contrastive learning method to address the challenge of GCD in the presence of domain shift (see Section 3.2.2, and ablation in Table 4). Our approach properly leverages all available samples, including both labelled and unlabelled data, from both in-domain and out-of-domain sources, encompassing both old and new classes. By incorporating these diverse samples, our technique aims to improve the model's ability to handle domain shifts and effectively generalize across different classes.
- A curriculum learning strategy: To our knowledge, the use of curriculums is still underexplored in the GCD literature. We find that in our challenging setting, it is critical for an appropriate curriculum to be introduced. 
- Disentangling domain and semantic features: Though the loss formulation has been previously explored, it has not been applied to category discovery before, where it finds a natural fit in the presence of the domain shift problem.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces HiLo, a novel framework for Generalized Category Discovery (GCD) under domain shifts. It disentangles semantic and domain features using mutual information minimization, enhances learning with PatchMix-based contrastive learning, and integrates curriculum learning. Extensive evaluations on SSB-C and DomainNet benchmarks show substantial improvements over baselines.

The paper's strengths include: 1) Innovative problem setting combining GCD and domain adaptation. 2) Effective disentanglement of domain/semantic features. 3) Robust experimental results with strong theoretical underpinnings.

However, the reviewers raised concerns about the novelty and insufficient analysis of the claimed disentangled feature.