HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts
Authors: Hongjun Wang
Sagar Vaze
Kai Han
Venue: ICLR 2025 Poster (2025)
Keywords: Generalized Category Discovery
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: we extracts domain and semantic information independently and minimize their mutual information while incorporating contrastive learning for robust representations with pseudo-labelling strategies

Abstract: Generalized Category Discovery (GCD) is a challenging task in which, given a partially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones. In this paper, we challenge a remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method to handle GCD when the unlabelled data also contains images from different domains to the labelled set. Our proposed `HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a specialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach. Finally, we construct a benchmark from corrupted fine-grained datasets as well as a large-scale evaluation on DomainNet with real-world domain shifts, reimplementing a number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations.

Published as a conference paper at ICLR 2025
HILO: A LEARNING FRAMEWORK FOR GENERALIZED CATEGORY DISCOVERY ROBUST TO DOMAIN SHIFTS
Hongjun Wang'
Sagar Vaze²
Kai Han !M
1 Visual AT Lab, The University of Hong Kong 2 Visual Geometry Group, University of Oxford
h jjjannecennect.... .hku.h hk
sagare roooto.ooooooo. ox ac. ukk
ka maanxhhu... hk
ABSTRACT
Generalized Category Discovery (GCD) is a challenging task in which, given a par- tially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones. In this paper, we challenge remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method handle GCD when the unlabelled data also contains images from different domains to the labelled set. Our proposed 'HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a spe- cialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach. Finally, we construct a benchmark from corrupted fine- ane-grained datasets as well as a large-scale evaluation on DomainNet with real- well-wrldd domain shifts, reimplementing number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations. Project page: ht nttps: : //visualllaeeeeeeeeeee ai. git github. io/hilo/////
INTRODUCTION
Category discovery Han et al. (2019) has recently gained substantial interest in the computer vision community Han et al. (2020; 2021); Fini et al. (2021); Wen et al. (2023); Jia et al. (2021); Zhao & Han (2021). The task is to leverage knowledge from a number of labelled images, in order discover and cluster images from novel classes in unlabelled data. Such a task naturally occurs in many practical settings; from products in a supermarket, to animals in the wild, to street objects for an autonomous vehicle. Specifically, Generalized Category Discovery (GCD) Vaze et al. (2022) has recently emerged as a challenging variant of the problem in which the unlabelled data can contain both instances from 'seen' and 'unseen' classes. As such, the problem is succinctly phrased as: "given a dataset, some of which is labelled, categorise all unlabelled instances (whether or not they come from labelled classes)".
In this paper, we challenge a key, but often ignored, assumption in this setting: GCD methods still assume that all instances in the unlabelled set come from the same domain as the labelled data. In practise, unlabelled images may not only contain novel categories, but also exhibit low-level covariate shift Sun et al. (2022); Yan et al. (2019). It has long been established that the performance of image classifiers degrades substantially in the presence of such shifts Ganin et al. (2016); Tzeng al. (2014); Zhang et al. (2019) and, indeed, we find that existing GCD models perform poorly in such a setting, Compared to related literature in, for instance, domain adaptation Du et al. (2021); Chen et al. (2022b); Zhu et al. (2023) or domain generalization Shi et al. (2022); Harary et al. (2022) the task proposed here presents a dual challenge: models must be robust to the lewel covariate shift while remaining sensitive to semantic novelty.
Concretely, we tackle task in which model is given access to labelled data from source domain. is further given access to a pool of unlabelled data, in which images may come from either the source domain or new domains, and whose categories may come from the labelled classes or from new ones (see Figure 1). Such setting may commonly occur if, for example, images are taken with different cameras or under different weather conditions. Moreover, such setting often observed
Corresponding author.

---

Published as a conference paper at ICLR 2025
on the web, in which images come from many different domains and with innumerable concepts. We suggest that the ability to cluster novel concepts while accounting for such covariate shift will be an important factor in fully leveraging web-scale data.
To tackle these problems, we introduce the 'HiLo' architecture and learning framework. The HiLo architecture extracts both 'low-level' (early layer) and 'high-level" (late layer) features from a vision transformer Dosovitskiy et al. (2020). While extracting features at multiple stages of the network has been performed in domain adaptation Bousmalis et al. (2016); Peng et al. (2019b); Liu al. (2020), we further introduce an explicit loss term to minimise mutual information between the two sets of features (Section 3.2.1).
The intuition is that the covariate and se- mantic information in the data is (by def- inition) independent, and that the induc- tive bias of deep architectures is likely to represent low-level covariate informa- tion in early layers, and abstract seman- tic information in later ones Olah et al. (2017); Zhou al. (2021). Next, we take inspiration from a strong method from the domain adaptation field, Patch- Mix Zhu et al. (2023), which works by performing mixup augmentation in the embedding space of pretrained trans- former. While naive application of this method does not account for semantic novelty in unlabelled data, we extend the PatchMix objective allow training with both self-supervised contrastive objective (Section 3.2.2), and seman- tic clustering loss (Section 3.2.2). With these changes, the PatchMix style aug- mentation is tailored to leverage both the labelled and unlabelled data avail- able in the GCD setting. Our 'HiLo' feature design in our framework enables the model to disentangle domain and se- mantic features, while patch mixing allows the model to bridge the domain gap among images and focus more on determining the semantic shifts. Therefore, we introduce the patch mixing idea into our 'HiLo' framework, equipping it with strong capability to discover novel categories from unlabelled images in the presence of domain shifts.
Seen Class
Unseen Class
Lirswes
Domain
Food
Cat
Gat
Plane
Bird
Strewberry
Sementics
Figure 1: We present new task where model must categorize unlabelled instances from both well as seen and novel domains. In the example above, models are given labels only for the images in green boxes. The models are tasked with categorizing all unlabelled images. including those from different domains (top two rows) and novel categories (rightmost three columns an orange background).
and unseen categories, as
Finally, we find that curriculum learning Bengio et al. (2015); Zhou et al. (2020); Wu & Vorobeychik (2022) is particularly applicable to the setting introduced in this work (Section 3.2.3). Specifically, the quality of the learning signal differs substantially across different partitions of the data: from a clean supervised signal on the labelled set; to unsupervised signals from unlabelled data which may or may not come from the same domain and categories. It is non-trivial to train GCD model to discover novel categories in the presence of both domain shifts and semantic shifts in the unlabelled data. To address this challenge, we introduce a curriculum learning approach which gradually increases the sampling probability weight of samples predicted as from unknown domains, training proceeds. Our sequential learning process prioritizes the discovery of semantic categories initially and progressively enhances the model's 's ability to handle covariate shifts, which cannot be achieved by simply adopting existing domain adaptation methods.
To evaluate out models, we construct the 'SSB-C' benchmark suite- based on the recent Semantic Shift Benchmark (SSB) Vaze et al. (2021) - with domain shifts introduced by synthetic corruptions following ImageNet-C Hendrycks & Dietterich (2019). On this benchmark, as well as on a large-scale DomainNet evaluation with real data Peng et al. (2019a), baselines from the category discovery literature. We find that, on both benchmarks, our method substantially outperforms all existing category discovery models Vaze et al. (2022); Wen et al. (2023); Han et al. (2019); Fini et al. (2021).
also reimplement range of performant
In summary, we make the following key contributions: (i) We formalize a challenging open-world task for category discovery in the presence of domain shifts: (ii) We develop a new method, HiLo, which disentangles covariate and semantic features to tackle the problem, extending state-of-lue the
2

---

Published as a conference paper at ICLR 2025
Mulual information
minimization
PatchMix
Transformal
1-0
Book
Dataset
Projector
Projector
Curtcullun poretimnn
(-T12
TiMe
1sT
Projection
Figure 2: Overview of HiLo framework. Samples are drawn through our proposed curriculum sampling approach, considering the difficulty of cach sample. Labelled and unlabelled samples are paired and augmented through PatchMix which we subtly adapt in the embedding space for contrastive learning for GCD. The mixed-up embeddings are then processed by our network with a high-level (for semantic) and low- lovelevel (for domain) feature design, allowing for the semantic disentangled feature learning via mutual information minimization.
methods from the domain adaptation literature: (iif) We reimplement a range of category discovery models on a benchmark suite containing both dine-grained and coarse coarse-grained datasets, with real and synthetic corruptions. (iv) We demonstrate that, on all datasets, our method substantially outperforms current state-of-the-arr category discovery methods with finetuned hyperparameters.
RELATED WORK
Category discovery was firstly studied as novel category discovery (NCD) Han et al. (2019) and recently extended to generalized category discovery (GCD) Vaze al. (2022). GCD extends NCD by including unlabelled images from both labelled and novel categories. Many successful NCD methods have been proposed (e.g.. DTC Han et al. (2019), RankStats Han et al. (2020; 2021). WTA Jia et al. (2021), DualRank Zhao & Han (2021), OpenMix Zhong et al. (2021b), NCL Zhong al. (2021a), UNO Fini et al. (2021), class-relation knowledge distillation Gu et al. (2023)), they do not address domain shifts. Recent work Zang et al. (2023) considers domain shifts in NCD with labelled target domain images. We focus on GCD without any labelled instances from new domains, where unlabelled images may come from multiple novel domains. There is a growing body of literature for category discovery, including non-parametric methods (e.g., GCD Vaze et al. (2022), CiPR Hao et al. (2023)); parametric methods (e.g.. SimGCD Wen et al. (2023)), ORCA Cao et al. (2021), μ-GCD Vaze et al. (2023); prompt based techniques (e.g., PromptCAL Zhang al. (2023), SPTNet Wang et al. (2024a), PromptGCD Cendra et al. (2024)), debiased learning (e.g., DebGCD Liu & Han (2025)), and other specialized methods (e.g.. GPC Zhao et al. (2023), DCCL Pu et al. (2023), InfoSieve Rastegar et al. (2024)). However, existing GCD methods do not consider domain shifts in unlabelled data.
Semi-supervised learning (SSL) aims to develop robust classification models using both labelled and unlabelled data, assuming instances belong the same class set. Consistency- based approaches, such as Mean-teacher' Tarvainen & Valpola (2017), Mixmatch Berthelot et al. (2019), and Fixmatch Sohn et al. (2020), have demonstrated effectiveness in SSL. Recent methods Chen et al. (2020b;c; 2021) have enhanced SSL performance by incorporating contrastive learning (c.g.. Chen al. (2020a), He et al. (2020)). Several studies Wang et al. (2022): Rizve et al. (2022); Wang et al. (2024b); Sun et al. (2024) have extended standard SSL to open-world settings.
Unsupervised domain adaptation (UDA) adapts models from source domain to target domain, with labelled data from the former and unlabelled data from the latter. UDA methods are categorized into moment matching Tzeng et al. (2014); Long et al. (2015; 2017); Zhang et al. (2019) and adversarial learning Ganin et al. (2016); Gao al. (2021); Tang & Jia (2020) methods. DANN FGDA, DADA are popular examples using a min-max game. MCD and SWD implicitly use adversarial learning with L, distance and sliced Wasserstein discrepancy, respectively. CGDM Du al. (2021) leverages cross-domain gradient discrepancy, while Chen et al. (2022b) couples NWD with a single task-specific classifier with implicit K- K-Lipschitz constraint. PMTrans Zhu et al. (2023) aligns the source and target domains with the intermediate domain by employing semi-supervised mixup losses in both feature and label spaces. MCC Jin et al. (2020) minimizes between-class
3

---

Published as a conference paper at ICLR 2025
confusion and maximizing within-class confusion, while NWD Chen et al. (2022b) uses single task-specific classifier with implicit K-Lipschitz constraint obtain better robustness for all the domain adaptation scenarios.
HILO NETWORKS FOR GCD WITH DOMAIN SHIFTS
In this section, we start with the problem statement of GCD with domain shifts. Subsequently, we introduce the SimGCD baseline in Section 3.1, which serves as robust GCD baseline upon which our method is built. Finally, we introduce HiLo networks for GCD with domain shifts in Section 3.2.
Problem statement. We define Generalized Category Discovery with domain shifts as the task of classifying images from mixed domains = Q2 U (where Q n = @ and S2 may contain multiple domains in practise), only having access to partially labelled samples from domain S2", The goal is to assign class labels the remaining images, whose categories and domains may be or unseen in the labelled images. Formally, let D be an open-world dataset consisting of a labelled
set D' ((,, 31) ): C X x yd and an unlabelled set D { 1 C X The label space for labelled samples is y = C₁ and for unlabelled samples is y = C C- U C2. where C, C₁, and C2 represent the label sets for 'All', 'Old', and 'New' categories, respectively. It is important to note that y c y The objective of GCD with domain shifts is to classify all unlabelled images in D" (from either Ω2 or (29) using only the labels in D'. This different from the setting of NCD with domain shift Yu et al. (2022), which assumes ) n 0, and GCD, which assumes Ω Ω with [S21| |021 For notation simplicity, hereafter we omit the subscript for each image 2;-
3.1 BACKGROUND: SIMGCD
SimGCD Wen et al. (2023) is a representative end-to-end baseline for GCD, which integrates two primary losses for representation learning and parametric classification: (1) contrastive loss C based on InfoNCE Oord al. (2018) is applied for the representation learning of the feature backbone; and (2) a cross-entropy loss Lels for training cosine classification head Gidaris & Komodakis (2018), utilizing different image views as pseudo-labels for one another. Following Vaze et al. (2022), SimGCD employs the ViT model as the backbone containing m Transformer layers. Let F be the feature extractor consisting of these m layers and H be projection head. For an input image æ, a ég-normalised feature can be obtained by z H(F((p(x)) where şo a standard embedding layer before the multi-head attention layers in the ViT model. The representation loss is
(T)
Σ P(x)) E
logo ( =+;T),
where (:: T) is the softmax operation with a temperature r for scaling and P(xx) denotes the positive feature set for each æ. Suppose we sample a batch B, which contains labelled images and unlabelled images, denoted as B' and 8", respectively. For each æ E B (either a labelled or unlabelled image), P(xr) contains only the feature of different view of the same image. For each æ E B', an additional P(ar) including features of other images from the same class and the feature of a different view of the same image is also used for supervised contrastive learning. Likewise, the classification loss can be written as
Lale (m)
Σ q logo (2 w);)) WEW
where W is set of prototypes and each vector w in W represents Eg-normalised learnable class prototype. & is the fo-normalised vector of F()((x)). For each x E B, q is the pscudo-label from a sharpened prediction of different view of the same image. For each x E B², an additional q as the one-hot ground-truth vector is also used for supervised learning. Let e be the summation of Leep and Cals for simplification, the overall loss can then be written as:
Lxim = x X (x) (1 Σ (x) +
BEB
BEB
where Be denotes the subset of labelled samples in the current mini-batch, and is an entropy maximization term to prevent pseudo-label collapse Assran et al. (2022). Finally. X and are hyperparameters, and we refer the original work for further details Wen et al. (2023).
4

---

Published as a conference paper at ICLR 2025
Despite achieving strong performance on the standard single-domain GCD task. SimGCD struggles in the more realistic scenario in which the unlabelled data exhibits domain shifts. Next, we present our HiLo framework, which builds upon SimGCD and introduces three key innovations to effectively handle domain shifts in GCD.
3.2 HILo: HIGH AND LOW-LEVEL NETWORKS
The architecture of our HiLo framework is outlined in Figure 2. Firstly, propose method to disentangle domain features and semantic features using mutual information minimization. Secondly, we introduce patch-wise mixup augmentation in the image embeddings, facilitating knowledge transfer between labelled and unlabelled data across different domains. Lastly, employ currieu- lum sampling scheme that gradually increases the proportion of samples from the unseen domain during training. This curriculum-basee baasd approach aids the learning process by initially focusing on casier single-domain discrimination and gradually transitioning to more challenging cross-domain discrimination.
3.2.1 LEARNING
DISENTANGLED FEATURES FOR GCD
As covariate shift observed by new domains Ω in D" degrades performance, we aim to learn two distinct feature sets encoding domain and semantic aspects by minimizing their mutual information. For each image ac, thus consider that its feature can be partitioned into two parts, depicting domain-specific (e.g.. real, sketch) and semantic information (e.g., cat, dog), respectively. However, it is intractable to estimate the mutual information between random variables of semantic and domain in finite nigh-dimensional space without parametric assumptions Zhao et al. (2018); Song & Ermon (2019). Instead of calculating the exact value, assumptions based on convex conjugate Nguyen et al. (2010) and GAN Nowozin et al. (2016) are utilized for estimation. Belghazi et al. (2018); Hjelm et al. (2018) further demonstrate that this estimation can be achieved without such assumptions. We thus adopt the approach from Hjelm et al. (2018) based on estimate the mutual information. For each image, instead of considering single feature vector as = H(F((((a))) here we consider two feature vectors, z, and 2,, for domain and semantic information respectively. Inspired by the fact that deeper layers the model give higher-level features and the shallower layers of the model give lower-level features Sze et al. (2017); Zhou et al. (2021), use the feature from the very first layer of the ViT as za and that from the very last layer as z,-
divergence to
Specifically, we obtain [za, ze] H (())))) where H. consists of two projection heads, one on the first layer feature F and the other on the last layer feature of (see Figure 2). Therefore, the mutual information between domain and semantic features can be approximated by estimator:
L. (zd, 2x) Ep(xa...) log (1 + e =+(=0t2))) -ppesa)((s)) [log (1 + ((
(4) where & is an MLP and an output dimension of 1. & takes the concatenation of 2, and za as input and predict a single scalar value. We aim to minimize the expected log-ratio of the joint distribution concerning the product of marginals. Note that here z, and Σạ may come from two different images. In practice, we tile the domain and semantic features of all the images in the mini-batch, and concatenate them, before applying on all the concatenated features. We then extract the diagonal entries (which are from the joint distribution) as the first term and the other entries (which are from the marginals) as the second term in Equation (4).
3.2.2 PATCHMIX CONTRASTIVE LEARNING
Mixup Zhang et al. (2018b) is a powerful data augmentation technique that involves blending pairs of samples and their corresponding labels to create new synthetic training examples. It has been shown to be very effective in -unerispeee learning Hataya & Nakayama (2019), long-tailed recognition Xu al. (2021), etc. In the presence of domain shifts, Mixup has also been shown to be effective in unsupervised domain adaptation Na et al. (2021) and domain generalization Zhang et al. (2018a); Yun et al. (2019); Zhou et al. (2021). Recently, PMTrans Zhu et al. (2023) introduced PatchMix, which is a variant of Mixup augmentation by mixing up the embeddings of images in the Transformer-bat based architecture for domain adaptation. Particularly, for an input image x with label y, PatchMix augments its j-th embedding patch by
( Bg (z()), + +1 (1-
(1- B3) (p)=)))))
where x' is an unlabelled image with or without domain shift, B, E [0. is the random mixing proportion for the j-th patch, sampled from Beta distribution, and denotes the multiplication
5

---

Published as a conference paper at ICLR 2025
operation. A one-hot vector derived from y is then smoothed based on 3g supervise the cross- entropy loss to train the classification model. However, this works under the assumption that the out-of-domain samples share the same class space with the in-domain samples, restricting its application to the more practical scenarios where the out-of-domain samples may come from new classes as we consider in the problem of GCD with domain shift. Hence, we devise PatchMix-baad contrastive learning method to address the challenge of GCD in the presence of domain shift. Our approach properly leverages all available samples, including both labelled and unlabelled data, from both in-domain and out-of-domain sources, encompassing both old and new classes. By incorporating these diverse samples, our technique aims to improve the model's s ability to handle domain shifts and effectively generalize across different classes.
When incorporating the PatchMix into our problem setting, the unlabelled sample æ'' in Equation (5) may have both domain and semantic shifts. With the new PatchMix augmented embedding layer ş and the two projection heads of H. wc can obtain [ =.] = (=)(((())) We separately consider the learning of domain and semantic features. For semantic features, we introduce a factor which takes the portion semantic of the sample x: into account, after mixing up with æ'. In specific, the contrastive loss in Equation (1) is now modified as:
(T)(())
Σ
a lo, ET;T7),
[P(x)
(6)
E E(((()
where a =
B.a+(1-B)-8" 3 denotes the vector consisting of all 33 as in Equation (5). 8 and s' are two vectors storing the attention scores for all the patches for æ and x' respectively. The attention scores, computed following Chen et al. (2022a); Zhu et al. (2023), account for the semantic weight of each patch. To train the semantic classification head, we adopt the loss as in Equation (2). Differently, if as is a labelled sample, inspired by Szegedy al. (2016), we replace q with q = a q + Tcl
3.
where q the one-hot vector derived from the label y of æ.. If x: is unlabelled, similar to Equation (2), is a pseudo-label from a sharpened prediction of another mixed-up view. Aside from the label q. we also need to learn another set of semantic prototypes by replacing W with W Let the modified classification loss be Lsla and L be the summation of LTeP and Lgls, The loss for PatchMix-bass semantic representation and classification learning is:
L. = A Σ (x) +1 BEB
Σ 2. (x)
ZEB
Next, for domain-specific features, employ the same loss Equation (6), except that we now train on Σz. for representation learning. We denote this loss as LT For training the classification head, we again adopt the loss as in Equation (2) but modify the label q and learn set of domain prototypes Wd. Therefore, if is a labelled sample, q = a q + 1-a 1. where q is the domain label. Note that
we only assume that the labelled samples are from the same domain and do not assume that domain labels are available for any unlabelled samples, which is more realistic and challenging. Therefore, the only known domain label is typically 1. To obtain pseudo-labels for the unlabelled samples, we run the semi-supervised k-means as in Vaze et al. (2022) on the current mini-batch. We denote this modified classification loss as Cgle and the summation of C and Cole as L Therefore, the loss for representation and classification learning can be written as
Ld Σ BEB
( (1 A) Σ (x). #EB
(8)
Overall loss. We apply the mean-entropy maximization regularizer, as described in Equation (3), to both semantic and domain feature learning. These regularizers denoted as A. and Ag respectively. Let A A. Aa and be the balance factor. The overall loss for our HiLo framework can then be written as
L = Lm + L. + Cs + AeA.
3.2.3 CURRICULUM SAMPLING
As curriculum sampling Bengio et al. (2015) can effectively enhance the generalization capability of models by gradually increasing the difficulty of the training data, which is also a natural fit to the GCD with domain shift problem. Here, we also introduce curriculum sampling scheme further

---

Published as a conference paper at ICLR 2025
enhance the learning of our HiLo framework. We expect the training to start by focusing on samples from the same domain to learn semantic features and leverage more samples containing the additional challenge of domain shifts in the later training stages. To this end, we devise a difficulty measure Pes ( ()) for each sample x at training time step (i.e., epoch), by considering the portion of samples belonging to each domain. As the unlabelled samples are from multiple domains and we do not have access to the domain label, we run the semi-supervised k-means on all the domain features extracted using the DINO pretrained backbone. Let the resulting clusters along the domain axis be Dd and D), which corresponds to domains 12 and S2' respectively and Dr = D U D'D With the above, we then define the sampling probability weight Pcs (x) for each sample as follows:
1,
x D
D
Pos (2)
=
(Da
x E EDo
(10)
ro + (r' - (t)) f);
x E D
where (-)) is an indicator function, L' is constant epoch number since which we would like to increase the portion of samples from unknown domains, ry and r' are constant probabilities for samples from unknown domains to be sampled in the earlier stages (i.e., < (') and latter stages (i.e., > ('), / indicates the current training time step. In our formulation, (1) if is a labelled sample, its Pes (z+(t is set 1, without any discount; (2) if x an unlabelled sample and is in D7 (i.e.. predicted as from the seen domain), Pcs tt is set to [D'L
[2D1| (i.e.. proportional to the labelled and unlabelled samples from the same domain, as per the sampling strategy used in the conventional GCD without domain shifts Vaze et al. (2022)); and (3) if æ is an unlabelled sample and is in Do (i.e., predicted as from the unseen domain). its Pea ( (x)t) will increase along with the training after epoch t'. We investigate choices of ro. r' and E' in Appendix M.
In Appendix D, we provide an approximated theoretical analysis for our method with two theorems. Theorem suggests (1) that learning on the original domain data first can effectively lower the error bound of category discovery on D" and (2) the domain head that can reliably discriminate original and new domain samples can further reduce this error bound. Theorem 2 suggests that minimizing the mutual information between domain and semantic features can further lower the error bound of category discovery on D" These theorems further validate the effectivvnness of our method from a theoretical perspective.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. To validate the effectiveness of our method, perform various experiments on the largest public datasets with domain shifts, DomainNet Peng et al. (2019a), containing about 0.6 million images with 345 categories distributed among six domains. Moreover, based on the Semantic Shift Benchmark (SSB) Vaze et al. (2021) (including CUB Welinder et al. (2010), Stanford Cars Krause et al. (2013b), and FGVC-Aireraft Maji et al. (2013)), we construct a new corrupted dataset called SSB-C (i.e.. CUB-C, Scars-C, and FGVC-C) following Hendrycks & Dietterich (2019), We exclude unrealistic corruptions and corruptions that may lead to domain leakage to ensure that the model does not see any of the domains in SSB-C during training (see Appendix we introduce 9 types of corruption and 5 levels of corruption severity for each type, resulting in dataset 45x larger than SSB. For the semantics axis, on both DomainNet and SSB-C, following Vaze
for details). Overall,
Table Statisties of the evaluation datasets.
Labelled
Unlabelled
#Class D
#Domain [
#Class |
#Domain
Dataset
#Image
#Image
[|]
DomainNet
39.1K
172
1
547.5K
345
6
CUB-C
1.5K
100
1
45K
200
10
Scars-C
2.0K
98
1
61K
196
10
FGVC-C
1.7K
50
1
50K
100
10
et al. (2022), we sample a subset of all classes as the old classes and use 50% of the images from
these labelled classes construct D/z-- The remaining images with both old classes and new classes
7

---

Published as a conference paper at ICLR 2025
treated as the unlabelled data Dir.. For the domain axis, DomainNet, select images from the 'real' domain as Da= and pick one of the remaining domains as Das in turn (or include all the remaining domains as D.s)). While on SSB-C, we use each dataset in SSB as Dn= and its corresponding corrupted dataset in SSB-C as Dis. Statistics of the datasets are shown in Table 1.
Evaluation protocol. For DomainNet, Ω2- = {wi} and f2 = {w2). where wi stands for different domains. We also experiment with the case where S = {w2, ...,,.... w.]. We train the models on De= (hee, D/z= Dij)) and Dis of all classes without annotations. For SSB-C, Q2 = {wi) } and I² = =fa2.... [1o] since we have nine types of corruptions. During evaluation, we compare the ground-truth labels y; with the predicted labels Oi and measure the clustering accuracy as ACC - D Eler 11 [yy = (()))( where & is the optimal permutation that matches the predicted cluster assignments to the ground-truth labels. We report the ACC values for 'All' classes (i.e., instances from )), the 'Old' classes subset (i.e., instances from )²),, and "New" classes subset (i.e., instances from )²m) for Dj and Dir separately.
Implementation details. Following the common practice in GCD, we use the DINO Caron et al. (2021) pre-trained ViT-B/16 as the feature backbone and the number of categories is known in Wen et al. (2023) for all methods for fair comparison. When the category number unknown, one can employ existing methods (e.g.. Han et al. (2019); Vaze et al. (2022); Hao et al. (2023); Zhao et al. (2023)) to estimate it and substitute it into the category discovery methods (see Table 24). The 768-dimensional embedding vector corresponding to the CLS token is used as the image feature. For the feature backbone, we only fine-tune the last Transformer layer. We train each dataset for 7 = 200 epochs using batch size of 256. We follow the protocol in Vaze et al. (2022); Wen et al. (2023) to select the optimal hyperparameters for our method and all baselines, based on the 'All' accuracy on the validation split of Dun. The initial learning rate for our approach is 0.1 for CUB and 0.05 for other datasets, and the rate is decayed using a cosine schedule. e' is set to the 80-th epoch. To is assigned as 1D1//1(2) for DomainNet and 0 for SSB-C. r' is set to 1 for DomainNet and 0.05 for SSB-C. is set to 0.1. Following Vaze et al. (2022); Wen et al. (2023), we set O.35. See Appendices M and N for choices of hyperparameters for HiLo components and learning rates for all methods.
4.2 MAIN COMPARISON
We compare our method with ORCA Cao et al. (2021), GCD Vaze et al. (2022) and SimGCD Wen et al. (2023) in generalized category discovery, along with two strong baselines RankStats+ Han et al. (2021) and UNO+ Fini et al. (2021) adapted from novel category discovery, on DomainNet (Table 2) and SSB-C (Table 3), respectively. Additionally, we provide results by incorporating UDA techniques in Section 4.3 and the strong CLIP model in Appendix G.
In Table 2. we present results on DomainNet considering one domain as Ω2 each time. Our method consistently outperforms other methods for 'All' classes (even better for 'New" classes) in both domain f2 and Ω2' by a large margin. For example, for the 'Real' and 'Painting' pair, it outperforms the GCD SoTA method, SimGCD, by nearly 5% and 19% in proportional terms, which is remarkable considering the gap between different methods. RankStat+ performs well on 'Old' categories in the unseen domain 52*h In Appendix B, we present results on DomainNet considering all domains except 'Real' as Ω2ºh It can be seen, in such a challenging mixed domain scenario, our method still substantially outperforms other methods. breakdown evaluation of each domain shift for both datasets can be found in Appendices H and I.
Table 2: Evaluation the DomainNet dataset. The model is trained on the 'Real' (i.e., 2*) + Paint- ing' / /'Seeehh 7 /"uickdraww" /'Clipart// / I'ffogrrah'' (i.e., 22") domains turn.
1800
46.
73.3
261
361
45
173
45
725
26
13
116
93
314
666
161
43
58
44
465
461
333
ST8
22
16J
4.
644
36
160
153
B3
61.3
196
528
244
38
322
624
31
866
11
414
416
224
41
816
200
312
N60
41
610
468
M66
15
664
216
41
45
40
43
M1
88
21
151
96
741
R1
96
258
267
M6E
361
RM
MAT
164
118
In Table 3, we show the results on SSB-C. We can see that HiLo significantly outperforms other methods across the board. For example, on CUB-C, HiLo outperforms SimGCD nearly 43.8% in proportional terms within Ω8 and 51.4% on unlabelled samples within Ω SimGCD shows good performance for new categories, while UNO+ demonstrates good performance for old categories.
Comparing the results with the single domain results in Vaze et al. (2022); Wen et al. (2023), we find that including corrupted data during training impairs the performance the original domain. SSB-C
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation SSB-C datasets. We report results of baselines in the seen domain (i.e., Original) and the overall performance of different corruptions (i.e., Comupted). On 'Corrupted' our model provides hetween 20 % and 80% relative gains over SimGCD Wen al. (2023).
CUB-C
Sears-C
FGVC-C
Original
Corrupted
Original
Corrupted
Original
Corrupted
Methoos
All
New
All
New
All
New
All
Old
New
All
Old
New
All
Okd
New
RankStats+
19.3
22.0
15.4
13.6
23.9
4.5
14,8
20.8
7.8
11.5
22.6
1.0
14.4
16.4
14.5
8.3
15.6
5,0
UNO
25.9
40.1
21.3
21.5
33.4
8.6
22.0
41.8
7.0
16.9
29.8
4.5
22.0
33.4
13.8
16.5
25.2
8.8
ORCA
18.2
22.8
14.5
21.5
23.1
18.9
19.1
28.7
11.2
15.0
22.4
8.3
17.6
19.3
16.1
13.9
17.3
10.1
GCD
26.6
27.5
25.7
25.1
28.7
22.0
22.1
35.2
20.5
21.6
29.2
10.5
25.2
28.7
23.0
21.0
23.1
17.3
SimGCD
29.0
28.8
26.7
22.1
26.1
28.9
22.3
21.4
UniOF
27.5
26.8
27.3
22.5
24.3
22.9
31.4
13.7
27.3
29.8
22.5
21.6
23.5
19.6
(Ours)
56.8
54.0
60.1
52.0
53.6
50.5
39.5
44.8
37.0
35.6 42.9
28.4
44.2
50.6
47.4
31.2
29.0
A3.4
is 45x larger than SSB, posing a significant challenge and resulting in unsatisfactory performance for existing methods. However, our method, Hil.o, continues to demonstrate promising results, further validating its effectiveness.
Table 4: Influence of different model components. We select the 'Real' and 'Painting' domains from DomainNet to train the DINO model with the techniques introduced above as the baseline. Rows 2-4 indicate our main conceptual methodological contributions and rows 5-7 represent the careful ablation of enginecring choices.
Real
Painting
Methods
All
Old
New
All
Old
New
Reference
SimGCD Wen et al. (2023)
61.3
77.8
52.9
34.5
35.6
33.5
(I)
SimGCD + PatchMix in Zhu ct al. (2023)
62.5
76.3
54.2
34.8
36.0
33.8
(2)
SimGCD + PatchMix for CL
63.5
75.0
57.6
36.6
39.6
33.6
(3)
SimGCD Disentangled Features
66.4
79.2
59.8
35.6
36.7
34.2
(4)
SimGCD Curriculum Sampling
63.6
78.6
55.9
38.4
39.9
35.9
Reference
HiLo
64.4
77.6
57.5
42,1
42.9
41.3
(5)
3.d. 7, from deep features only
28.2
40.3
22.7
13.6
20.0
11.0
(6)
Zd. 2, from shallow features only
10.1
18.1
6.4
5.7
5.7
Self-dist. for domain head
63.2
76.8
56.1
40.2
40.5
39.8
4.3 ANALYSIS
Effectiveness of different components. We validate the effectiveness of different components and design choices for our method in Table 4. As our method is built upon SimGCD, the effectiveness of each component can be observed by comparing its performance with that of SimGCD, We combine SimGCD with the original PatchMix in Zhu et al. (2023) (row 1) as strong baseline for task since these are SoTA methods for GCD and UDA respectively. Rows 2-4 indicate our main conceptual methodological contributions. As can be seen, simply combining SimGCD with the original PatchMix developed for UDA leads relatively small influence on the results. The original PatchMix focuses mainly on bridging the domain gap of labelled classes through a semi-supervised loss, which limits its capability on the unseen classes from new domains. After subtly adapting PatchMix into contrastive learning for GCD (row 2), the unlabelled data containing both domain shifts and semantic shifts can be properly utilized for training. leading to an obvious performance boost on 52', Furthermore, when we disentangle semantic features from domain features (row 3), the model significantly improves performance on both Ω2 and Ω2, demonstrating dissociation of spurious correlations. Appendix E also shows the efficacy of MI regularization in two distinct scenarios. Curriculum sampling further enhances performance on S2' (row 4).
Incorporating various techniques for GCD with domain shifts. We study the effectiveness of incorporating the SoTA UDA techniques (MCC and NWD) into the baseline methods. Differently to our task, domain adaptation does not consider discovering categories in the unlabelled images from the unseen domain. Results are shown Table 5. By comparing the results with those of 'Real + Painting' in Table 2, we can see that the results for each method are marginally improved by introducing these techniques. This reveals that simply adopting the UDA techniques to the GCD methods is not sufficient to handle the challenging problem of GCD with domain shifts. Moreover, HiLo again notably outperforms all other methods after introducing these UDA techniques, despite the gain by these techniques being relatively marginal, further demonstrating the effectiveness and significance of our Hiloo design. We also extend our analysis by incorporating various UDA techniques (e.g.. EFDM Zhang et al. (2022), SFA Li ct al. (2021), UniOT Chang ct al. (2022)), data augmentation methods (e.g., Mixstyle. Mixup, Cutmix) and curriculum learning (e.g.. CL B Bengio al. (2009), SPI. Kumar et al. (2010)) into baseline models in Table 6. Our findings reveal that while some UDA techniques and data augmentations offer improvements, they fall short of addressing the full complexity of GCD with domain shifts. Specifically, EFDM improves SimGCD's 's performance only on Ω2 likely due its reliance explicit source-target domain alignment, which is not

---

Published as a conference paper at ICLR 2025
available in task formulation. The performance of different augmentation methods has clear drop when compared with our proposed PatchMix for CL (Table 4 row 2). Notably, HiLo consistently outperforms all tested UDA baselines and data augmentation techniques. These results underscore the necessity of our tailored approach for the challenging task of GCD with domain shifts, demonstrating that simply adopting existing UDA or data augmentation methods is insufficient to address this complex problem.
Table 5; Evaluation on the DomainNet dataset by introducing SoTA UDA techniques.
Real
Painting
Methods
All
Old
New
All
Old
New
RankStats+
37.3
62.1
23.4
31.0
51.2
9.2
UNO+
46,9
72.4
32.8
32.1
47.6
17.7
ORCA
33.4
50.1
26.7
30.0
41.1
9.1
GCD
+MCC+NWD
50.6
54.0
48.4
34.0
43.1
22.7
SimGCD
63.1
77.1
56.9
35.7
39.0
32.4
Hil.o (Ours)
65.0
77.8
58.0
42.5
43.1
42.0
Table 6: Influence of different UDA techniques (e.g-. Mixstyle, EFDM), data augmentations (e.g.. Mixup, Cutmix), and curriculum learning (e.g., CI., SPL.) We select the 'Real' and 'Painting' domains from DomainNet train the DINO model with the SoTA GCD method SimGCD as the baseline and compare with one baseline method UniOT from universal domain adaptation.
Real
Painting
Methods
All
Old
All
Old
Reference
SimOCD
61.3
77.8
52.9
34.5
35.6
33.5
SimGCD Mixstyle
62.3
76.8
54.0
35.0
36.1
34.0
SimGCD BFDM
62.6
76.0
34.1
34.8
33.8
SimGCD Mixup
62.7
76.5
54.3
34.9
37.2
32.5
SimGCD Cutmix
62.5
76.3
54.1
33.2
36.0
31.6
Reference
HilLo
77.6
42.1
42.9
HiLo + Mixup- PatchMix
63.7
77.1
56.8
39.8
39.9
38.7
IiLa Cutmix PaichMix
62.9
76.8
56.0
37.4
38.0
36.7
Hil a CT. curriculum sampling
75.9
34.7
35.8
HiLo + SPI. curriculum sampling
62.8
76.5
54.5
35.0
36.1
34.0
Reference
HiLo
64.4
77.6
37.3
42.1
42.9
41.3
SFA
60.1
73.4
38.0
UniOT
63.3
77.4
57.4
35.3
38.7
32.0
Importance of domain-semantic feature disentanglement, To validate the necessity of extracting
domain and semantic features from different layers, we experiment on two variants of the model, by attaching both heads in H either to the deepest layer or to the shallowest layer. As shown in rows 5-6 in Table 4, both variants are significantly inferior to our approach using features from different layers. In addition, we further carry out controlled experiments by fixing the layer for one of the two heads while varying the other. In Figure 3 (a), we fix the semantic head to the last layer and vary the 'Shallow' layer for the domain head, from layer 1 to layer 4. As can be seen, attaching the domain head to the earlier layers gives better performance, which also validates that lower-level features are more nomain-oriented Similarly, in Figure 3 (b), we fix the domain head the first layer and vary the 'Deep' layer for the semantic head, from the last layer to the fourth last layer. We can see that the last layer is the best choice for the semantic head. These results corroborate the importance of domain-semantic semantic feature disentanglement and our design choice of using lower-level features for domain-s -omain-specific information and higher-level features for semantic camantic-specifiee information.
Figure 3: To investigate the effect of features extracted from different layers, we fix the layer for one of the two heads while varying the other on the CUB-C C dataset. Features from the first and last layers yield the best performance.
5 CONCLUSION
In this paper, we study the new and challenging problem of generalized category discovery under domain shifts. To tackle this challenge, we propose the HiLo learning framework, which contains three major innovations, including domain-semantic disentangled feature learning, PatchMix contrastive learning, and a curriculum learning approach. We thoroughly evaluate HiLo on the DomainNet dataset and our constructed SSB-C benchmark, and show that HiLo outperforms SoTA GCD methods for this challenging problem.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements This work is supported by National Natural Science Foundation of China (Grant No. 62306251). Hong Kong Research Grant Council Early Career Scheme (Grant No. 27208022), and HKU Seed Fund for Basic Research.
REFERENCES
Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang. and Ping Chen, Generalized category discovery with decoupled prototypical network. In AAAI, 2023.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. ECCV, 2022.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In ICML, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. NeurIPS, 19, 2006.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. NeurIPS, 2015.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In NeurIPS, 2009.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. NeurlPS, 32, 2019.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.
Domain separation networks. NeurlPS, 2016.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICLR, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Fernando Julio Cendra, Bingchen Zhao, and Kai Han. Promptced: Learning gaussian mixture prompt pool for continual category discovery. In ECCV, 2024.
Wanxing Chang, Ye Shi, Hoang Duong Tuan, and Jingya Wang. Unified optimal transport frame- work for universal domain adaptation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NeurlPS, 2022.
Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, Alan Yuille, and Song Bai. Transmix: Attend to mix for vision transformers. In CVPR, 2022a.
Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task-specific classifier as discriminator: Discriminator-free adversarial domain adaptation. In CVPR, 2022b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In /CML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. ArXiv e-prints, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. ArXiv e-prints, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless systems. Cambridge University Press, 2011.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics, 1983.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu. Cross-domain gradient discrepancy minimization for unsupervised domain adaptation. In CVPR, 2021.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In ICCV, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 2016.
Zhiqiang Gao, Shufei Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong. Gradient
distribution alignment certificates better adversarial domain adaptation. In ICCV, 2021.
Spyros Gidaris and Nikos Komodakis, Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
Peiyan Gu, Chuyu Zhang, Rui Xu, and Xuming He. Class-relation knowledge distillation for novel class discovery. In ICCV, 2023.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCV, 2019.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. ICLR, 2020.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI. 2021.
Shaozhe Hao, Kai Han, and Yean-Yee K Wong. Cipr: An efficient framework with cross-instance positive relations for generalized category discovery. TMLR, 2023.
Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, et al. Unsupervised domain generalization by learning a bridge across domains. In CVPR, 2022.
Ryuichiro Hataya and Hideki Nakayama. Unifying semi-supervised and robust learning by mixup. ICLR workshop, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In ICLR, 2019.
Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
Karan Grewal, Phil Bachman, Adam
Xuhui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single- and multi-mon multi-modal data. In ICCV, 2021.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In ECCV, 2020.
Daniel Kifer, Shai Ben-David, and Johannes Gehrke. Detecting change in data streams. In VLDB, volume 4, pp. 180-191. Toronto, Canada, 2004.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013a.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshop, 2013b.
12

---

Published as a conference paper at ICLR 2025
M Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NeurIPS, 2010.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV, 2021.
Yuanpei Liu and Kai Han. Debgod: Debiased learning with distribution guidance for generalized category discovery. In ICLR, 2025.
Ziwei Liu, Zhongqi Miao, Xingang Pan, Xiaohang Zhan, Dahua Lin, Stella X Yu, and Boqing Gong. Open compound domain adaptation. In CVPR, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In ICML, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ArXiv e-prints, 2013.
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
Jaemin Na Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In CVPR, 2021.
XuanI Long Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS, 2016.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv e-prints, 2018.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019a.
Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled representations. In ICML, 2019b.
Nan Pu. Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In CVPR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
Sarah Rastegar, Hazel Doughty, and Cees Snoek. Learn to categorize or categorize to learn? self- coding for generalized category discovery. In NeurlPS, 2024.
Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openidn: Learning to discover novel classes for open-world semi-supervised learning. In ECCV, 2022.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. 2022.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying cuni-supervised learning with consistency and confidence. NewrIPS, 2020.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. ArXiv e-prints, 2019.
13

---

Published as a conference paper at ICLR 2025
Tao Sun, Mattia Segu, Janis Postels, Yuxuan Wang, Luc Van Gool, Bernt Schiele, Federico Tombari, and Fisher Yu. Shift: synthetic driving dataset for continuous multi-task domain adaptation. In CVPR, 2022.
Yiyou Sun, Zhenmei Shi, and Yixuan Li. A graph-theoretic framework for understanding open-world semi-supervised learning, NeurIPS, 2024.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: tutorial and survey. Proceedings of the IEEE, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In AAAI, 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. NeurIPS, 33:18583- 18599, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS, 2017.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. ArXiv e-prints, 2014.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good
closed-set classifier is all you need. In ICLR, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR, 2022.
Sagar Vaze, Andrea Vedaldi, and Andrew Zisserman. No representation rules them all in category discovery. In NeurlPS, 2023.
Hongjun Wang, Sagar Vaze, and Kai Han. Spinet: An efficient alternative framework for generalized category discovery with spatial prompt tuning. In ICLR, 2024a.
Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang. Linyi Yang, Zhi Zhou, Lae Guo, al. Usb: A unified semi-supervised learning benchmark for classification. NeurIPS, 2022.
Yu Wang,
Zhun Zhong, Pengchong Qiao, Xuxin Cheng, Xiawu Zheng, Chang Liu, Nicu Sebe, Rongrong Ji, and Jie Chen. Discover and align taxonomic context priors for open-world semi- supervised learning. NeurIPS, 2024b.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Xin Wen, Bingchen Zhao, and Xiaojuan Qi. A simple parametric classification baseline for general- ized category discovery. ICCV, 2023.
Junlin Wu and Yevgeniy Vorobeychik. Robust deep reinforcement learning through bootstrapped opportunistic curriculum. In ICML, 2022.
Zhengzhuo Xu, Zenghao Chai, and Chun Yuan. Towards calibrated model for long-tailed visual recognition from prior perspective. NeurIPS, 2021.
Wenjun Yan, Yuanyuan Wang. Shengjia Gu. Lu Huang, Fuhua Yan, Liming Xia, and Qian Tao. The domain shift problem of medical image segmentation and vendor-adaptation by unet-gan. In MICCAI, 2019.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Self-labeling framework for novel category discovery over domains. AAAI, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. arXiv preprint arXiv: 1905. 04899, 2019.
14

---

Published as a conference paper at ICLR 2025
Zelin Zang, Lei Shang. Senqiao Yang. Fei Wang, Baigui Sun, Xuansong Xie, and Stan Z Li. Boosting novel category discovery over domains with soft contrastive learning and all in one classifier, In ICCV, 2023.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018a.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pat Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018b.
Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Prompteal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. In CVPR, 2023.
Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact feature distribution matching for arbitrary style transfer and domain generalization. In CVPR, 2022.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In ICML, 2019.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statisties and mutual knowledge distillation. NeurIPS, 2021.
Bingchen Zhao, Xin Wen, and Kai Han. Learning cemi-supervised gaussian mixture models for generalized category discovery. In ICCV, 2023.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon.
Bias and generalization in deep generative models: An empirical study. NeurIPS, 2018.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In CVPR, pp. 10867-10875, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories an open world. In CVPR, 2021b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: From clean label detection to noisy label self-correction. In ICLR, 2020.
Jinjing Zhu, Haotian Bai, and Lin Wang. Patch-mix transformer for unsupervised domain adaptation: A game perspective. In CVPR, 2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A SSB-C benchmarks
17
Multiple unseen domains for DomainNet
19
C PatchMix contrastive learning
20
Theoretical analysis
D.1 Proof of bounds for the target error
21
D.2 A tighter bound for the target error
24
Effect of mutual information minimization on different datasets
28
F Novel category discovery in the presence of domain shifts
29
Investigation of CLIP for GCD with domain shifts
30
H Detailed evaluation of SSB-C datasets
31
Additional experimental results on DomainNet
32
J HiLo on the vanilla GCD setting
34
Effects of different output dimensions for the semantic and domain heads
35
L Unknown category number
36
M Hyperparameter choices for HiLo components
37
N Effects of learning rates
38
Stability of different methods
39
Qualitative Results
40
Broader impacts and limitations
42
16

---

Published as a conference paper at ICLR 2025
A SSB-C BENCHMARKS
As demonstrated in Section 4.1 in the main paper, construct the SSB-C benchmark to evaluate the robustness of algorithms to diverse corruptions applied to validation images of the original SSB benchmark (including CUB Welinder et al. (2010), Stanford Cars Krause et al. (2013a) and FGVC- Aircraft Maji et al. (2013)), adopting the corruptions following Hendrycks & Dietterich (2019). We introduce 9 types of corruption (see Figure 4) in total. Each type of corruption has 5 severity levels. Therefore, SSB-C x5x larger than the original SSB.
We exclude similar (i.e., defocus blur, glass blur and motion blur) or unrealistic corruptions (i.e., pixel noise and JPEG mosaic). We also exclude corruptions (i.e., bright noise, contrast noise) that may lead to domain leakage (since these corruptions have been adopted during DINO pretraining) to ensure that the model does not see any of the domains in SSB-C during training on the GCD task with domain shifts.
Here the list of the 9 types of corruption we applied following Hendrycks & Dietterich (2019):
Gaussian noise often appears in low-lighting conditions.
Shot noise, also known as Poisson noise, results from the discrete nature of light and is form of electronic noise.
Impulse noise occurs due to bit errors and similar to salt-and anp-pe noise but with color variations.
Frosted blur appears on windows or panels with frosted glass texture.
Zoom blur happens when the camera moves rapidly toward object.
Snow obstructs visibility while frost forms on lenses or windows coated with ice crystals.
Fog shrouds objects and can be rendered using the diamond -quuree algorithm.
Speckle noise is granular texture that occurs in coherent imaging systems, such radar and medical ultrasound. It results from the interference of multiple waves with the same frequency.
Spatter occurs when drops blobs splash, spot, or soil the images.
Though synthetic, SSB-C incorporates extra challenges and unique values over existing datasets like DomainNet Peng et al. (2019a). Particularly, SSB-C includes (I) fine-grained recognition challenges under domain shifts and (2) more types of domain shifts that are not covered in DomainNet.
17

---

Published as a conference paper at ICLR 2025
Gaussian Noise
Shot Noise
Impulse Noise
Zoom Blur
Snow
Frost
Fog
Speckle
Spatter
Figure 4: Our SSB-C dataset includes 45 distinct corruptions that are algorithmically generated from 9 types of corruptions, covering noise, blur, weather, and digital corruptions. Each type has 5 severity levels.
18

---

Published as a conference paper at ICLR 2025
B MULTIPLE UNSEEN DOMAINS FOR DOMAINNET
Due to the large scale of DomainNet (over 587K images), which is significantly larger than SSB-C, is difficult utilize all the samples from all remaining domains other than the 'Real' domain in the experiments. Nonetheless, we conduct experiments with multiple domains in DomainNet by subsampling instances while balancing classes and images per class. The total number of unlabelled images from different domains remains the same as the single domain experiment in the paper. Specifically, we randomly select 20% samples from each category in each domain without replacement. Putting all these selected samples from all domains gives subset which has approximately equivalent number of samples to the total number of samples in 'Painting' domain as the single domain experiment (see Section 4.2 the main paper). In this challenging multi-domain experiment, HiLo continues to demonstrate promising results, further validating its effectiveness.
Table Experiments on multiple domains in DomainNet. computation, while ensuring class and image balance. The total number of unlabelled images across different domains is kept consistent with the single domain experiment mentioned in the main paper.
subsample instances for ease of
Real
Painting
Skooth
Quickdraw
Clipurt
Infograph
Methods
All
Old
New
All
Old
New
Old
New
All
Old
New
Old
New
All
Old
New
UNO+
43.1
72.0
28.6
30.3
43.7
17.4
12.0
16.3
2.1
2.3
1.8
22.8
37.4
9.5
12.4
20.3
ORCA
32.1
40.9
23.5
23.0
38.8
17.0
11.6
14.7
7.6
2.8
3.6
2.1
20.1
33.4
10.3
8.4
17.8
6.8
GCD
47.8
53.5
45.1
32.9
40.3
26.9
17.0
22.7
11.3
1.9
2.4
1.8
24.3
31.2
15.1
10.5
12.0
9.9
SimGCD
62.2
77.3
54.3
36.6
42.9
30.3
18.2
22.6
15.0
2.2
2.0
2,4
25.0
34.7
16.4
11.8
13,8
10.5
(Ours)
49.0
23.6
19

---

Published as a conference paper at ICLR 2025
C PATCHMIX CONTRASTIVE LEARNING
Specifically, PatchMix consists of a patch embedding layer that transforms input images from labelled and unlabelled data into patches. As outlined in the main paper, PatchMix augments the data by mixing up these patches the embedding space (as shown in Figure 5 (a)). We randomly sample from Beta distribution to control the proportion of patches from images. Subsequently, we compute the loss for representation learning (Figure 5 (b)) and classification learning (Figure 5 (c)) based on the augmented embeddings and predictions, respectively. The confidence factor a is determined by the overall proportion of known semantics in the mixed samples (i.e., B for all the patches) and the attention scores for all the patches the input image. is then assigned based on the similarity score or the actual label to guide the training (see Equation (6) in the paper).
Set-supervied
Supersiedd
Gog
(org))
X1
A
Xi
Porj)
(hrrg)
(
(b) Representation leaming
X2
Unsuparvised
Superv
(a) PatchMix
(c) Classification leaming
Saneriiiiearr
Figure 5: Illustration of PatchMix and loss functions. (a) PatchMix augments the data by mixing up image patches in the embedding space with sampled from Beta distribution. (b) The similarity matrix for representation learning and (c) mixed embedding patches for classification learning are adjusted according to the actual semantic components within the mixed patches, determined by
20

---

Published as a conference paper at ICLR 2025
D THEORETICAL ANALYSIS
Recall that 'D is an open-world dataset consisting of a labelled set DD Na and an unlabelled set D {æ;} j=1, C We also define the mapping function 9 parametrized by deep neural network as one of the hypotheses from G.
v:) C X x ) {(x;
For terminology convenience, here, we term D' as the source domain data, distributed according to the density Px,X))) while D the target domain data, with density (X). Note that D contains the unknown mixture of (X) and (r(X). The objective of our task is to leverage measurable subsets Q and 52 under Dj and D22 to find a hypothesis E that minimizes the target error eD=, as defined by a zero-one loss function f: y x y R.
eDn (g) D [[(g(z),)) g)].
(11)
More generally, if y is determined by labelling function g' given the input x, have
eD- (g.9)) = EE-D= ([(g(z),
(12)
Similarly, the source error ep (g) and epi (g.g') can be defined by ex) Eu.grDi [e(g(x), v)]
and e (g. g') == (f_gx)((((()) g (x)].
When the source domain does not adequately cover the target domain, the target risk of a learned hypothesis cannot be consistently estimated without additional assumptions. Nonetheless, an upper bound on the target risk can be estimated and then minimized. Ben-David et al. (2006) introduce the A-distance (also known as H-divergence) assess the worst-case loss when extrapolating between domains for hypothesis classes. The A-distance between any two distributions Dj and D2 is defined as
ds(Di, sup Pr(g(a) 1] Pr(g(x) = 11 gEG D1
D2
D.1 PROOF OF BOUNDS FOR THE TARGET ERROR
Lemma 1. Consider symmetric hypothesis class G defined on the space X, with VC dimension d. Let le and D be collections samples under domains DD and Dzz de( ( is the empirical A-distance between these sample sets. For any (0, 1), with probability least 8,
do(D1, S min gEG
Σ ==g(=)==
1(x E (2) +
1
1(x E 22)) x;y(z)=1
Σ
|
d dloo(2 S2
log d logg(2/22))
log 2 8
4 max
[2a
[
Proof. Recall the definition of the A-distance for hypothesis class G:
ds( = sup
(r(T())
-Pr(7(n)]
(13)
geG Q
520
where
Pr[[(g)]
Σ (g((x) 1),
12
[2a
#Ef2
Pr[[(()
(g(()) 1).
ZEΩ
For any hypothesis and corresponding set [(g), we have
[r]]
(r)
Σ 1(((x)
1)
1 Σ 1(g(x) 1).
2
[2a
#en*
21

---

Published as a conference paper at ICLR 2025
The empirical A-distance is then
do 12") = sup gEG
(g(x)) = 1)
Σ
[(g(z) 1)
=Ef
xEN
sup gEG
Σ
1 (())
Σ
1(x E
[20
122
z:g(z)=1
x:g(2) m1
To simplify this, we consider the complement set where g(x)) =
Pr([(g)]
Pr(1())]=
1
Σ
1(x
=2))
Σ
1(x E 2²))
Q2
52"
[
|
#;y(z)=1
219(#)=1
Σ
(
Σ
1 E
==9(z)=0
=10(0))=
Thus, the empirical A-distance can be expressed as:
de(S29,52*)
de(S2
sup gEC
Σ
1 EQ2)
1
Σ
1(x (
|021
x=g(=)==
x;g(x)=1
sup gEC
1
Σ
[ E
+
Σ Ma 219(0)==
$220
S2
#=0(z)=0
To find the minimum value, need to consider the complement of the set (g), which leads to minimizing the expression inside the supremum. This gives us:
1 E (05)) ))
=2 ds( -=)==" 1 min DEC
Σ
1(x (()))= +
Σ
n
[2
==((===
x;g(=)=1
From Theorem 3.4 of Kifer al. (2004), we know that:
|| [ds( de D2)1 > (21091)2--1000 120 -2)
We a union bound to handle the terms separately:
(2m))e-He 18 e/116 < and
2
For the first inequality:
(2 100
Taking the natural logarithm both sides:
12
log((2 (2
log
16
10
d dloo(2(
< log
16
16
dloog(2(221) - llog 2
2
8
dloo((((20)) - log
4
|224|
22

---

Published as a conference paper at ICLR 2025
Similarly, for the second inequality:
dlog(2| log 2
[29
To ensure that both inequalities hold, we take the maximum of the two derived values:
(loo((2(22))
log?
(2 log 3
2 max
4
[2a9]
[22
Thus, we have
dlog(2|
log 2
[log(2[22))
log /
(D, s do((2" + max
[02a1
[S28|
(xr 126) E 1)
1 min 2
1 E +
Σ
[200
1226
x=g(===0
x;g(z)=1
[dlog(2 12
log
dooo (25
log
+ max
[200
[26
Theorem Consider a symmetric hypothesis class defined on the space X, with a VC dimension d. Let Q and D be collections of samples under domains Dj and D:2- For any (0, 1), with probability at least 8.
11 E 52%) ))
(D=(g) S (g) +
1 min gEG
Σ
1(x E (2))
Σ 2;g(x)=1
[2
1220
z=g(z)=0
d (o (2
log 3
d dloo((1(22)) log 2
max
[2a0|
[2261
Proof. Let g E G be a hypothesis such that g(x) = 1 if and only if gi (x) + g2(x) for some 91.92 G, indicating disagreement between 91 (z) and g2(x). Based on the definition of A- distance, have
de (D(,,DD2) = suup Pr[g(z) =1] - --rrl(((T) 11 D.
.eG
2 sup
eeD (91, .2) - (91 92)l
gi.92G
2/eD, (g1, 92) eD2 (01.g2))
Consider an ideal joint hypothesis g", which is the hypothesis which minimizes the combined error (ideally zero). By using the triangle inequality Ben-David et al. (2006), have:
de (D,, D2) 2 (e (91) eD, (g")) (eD, eD,
> 2/eD, (91) ED. (91)).
As D contains samples from both Dj and D2, we immediately know that:
ds(DD, Dd) de D2)
23

---

Published as a conference paper at ICLR 2025
By Lemma I, we have that
2(eD= (g) - eD (1)) S
1
min ge
Σ
1(x 22) +
Σ
[x E (P))
[20
|22
m:g(a)=o
x:g(>)-1
dloo(()
log $ [aaa]
d d log
4 max
|S28|
1 M60E
cD- (g) S (g) + 1
min gEG
Σ amg(ar)=
1 E (2)
Σ
[22n]
|
(an(a)
d (2
log
dloo (2
log
2 max
[2
Theorem demonstrates that the upper bound of the error on D" depends on the error on D' and the domain classification performance of g. It is evident that all components involved in Equation (8) minimize the misclassification eггоr the second item. For the first item, curriculum sampling
ensures reduced error of on D during the early training stage, before HiLo accurately classify
different domains through the domain head (thus leading to lower error in domain classification, ie., the second item).
D.2 A TIGHTER BOUND FOR THE TARGET ERROR
Lemma 2. For the hypothesis class G. Mc(Di,D)) S 2 D2llrv.
Proof. Recall that the total variation (TV) distance between two distributions D2 and D2 is defined as:
D D2llrv sup (A) D2(A)),
A
where the supremum is taken over all measurable sets A.
The A-distance can be seen as specific form of the TV distance where the measurable sets A are the subsets of the input space that can be defined by the hypotheses g E G. However, the TV distance considers all possible measurable sets A. For any measurable set A, we can consider the indicator function 1 (x) which takes value if x E A and O otherwise. The TV distance can be expressed in terms of these indicator functions:
/ (IA(E) - 11(()) /
P D2lTV sup (D (P)())))(AA) sup A
When considering the hypothesis class G, we look at the functions g(ø) that take the value I or similar to indicator functions for sets:
do D2) = sup Pr(g(c) = 1] Pr(g(() = 11|- get
D2
For given hypothesis g E G. let A, = for g(x)) = 1}. The difference in probabilities for this hypothesis is:
Pr(g(s)= =
Pr[g(>) -Pr-g[(g( Dr
1| = - -
We left off by noting that for any hypothesis g G. the difference in probabilities PPD, (g(x)
- Prp, (g(x) = 1] is bounded by the TV distance [ D2llrv:
(A,(A,) D2(A,) < ID: D2l
24

---

Published as a conference paper at ICLR 2025
where Ag {x ((()) = 1}.
The A-distance takes the supremum of this difference over all hypotheses g G:
ds(Di, D2) sup
Pr(g(x) 1 Pr(g(<) 11
gEC [2
D2
Because each individual difference (g()) 11-11PPP Pro, (x) = 1|| is bounded by 1 -22 P21|TV. the supremum over such differences must also be bounded by [ - D2llav:
sup gec
Pr(g(>) 1] Pr(o(>) 1| D,
$ [2: D2llrv
D
2 sup gEC
Pr(g(z) = 1] Pr(g(() =1] s 2 - D2llIV
D2
((1,)2) D D2llTV.
Lemma 3. Let random variable 2 be representation of the input features X. Fx(X) = with Fig E IF is a feature transformation and H E H operating in the representation space is prediction function. Hypotheses 9 E G are formed by compositions g = H. Fu and G = {H Fg H E F. E }. For all F& E R and H E H.
da(Z', S dD D
Proof. Let E G be a hypothesis such that g(x) = 1 if and only if g1 (x) * 92(x) for some 01.92 G, indicating disagreement between 91 (2x) and g2(z). Based on the definition of A- distance, we have
de(D', D") sup Pr(g(a) 1] p Pr(e(x) 1
geG Di
and similarly,
da(z" 2²) sup Hell
Pr[H(=) -))((()) 11
For Z (X). we know that:
Pr [H(=)= =
Pr [H[ (x)) = 1],
and
Pr [M(2)) =1 Pr (F))(())) = 1). D~D
2~
For each H E H., there a corresponding g E G such that = H o F However, not every g E G has corresponding H HH because F.p might not cover the entire space or map back uniquely. This leads
sup HEl
Pr [+(2) = Zi
Pr [+(2) = 1]
< sup geG
Pr [g(x) mnDi
1]
Pr [g(x) =
20
a~Do
2 sup Hell
Pr [H(2))
=
Pr [H(2) 1] 2 ssu gEG
Pr (0(x)=11 Pr (g(2) 11
z
IDDi
(z', z < ds (D' D
25

---

Published as a conference paper at ICLR 2025
Lemma 4. Let random variable Z be representation of the input features X. x22) 2d: 2s with sp E V is separator for the Nomain-specifet feature 2 and the semantic-specifcc feature 2, E 2 Let J E be a prediction function based on 2d 2 and [(z (x) be the mutual information between and 2s. for a hypothesis space J,
dr dz' S (D' D").
DD).
Proof. Hypotheses g E G can be formed by either compositions H Fip and G = F. H E F. E }. or compositions y = I o o and G = {] F. I E F ER). Then, similar to the proof for Lemma 3, we can easily get the inequality.
Theorem 2. Let G be symmetric hypothesis class defined on the space X, with VC dimension d. Let Q and Q be collections of samples under domains DD and Dz. 2d and 2, be drawn from 2d and Za, and ((za ze) be the mutual information between 2d and 2g. Define the optimal function functionn 9" ( as follows:
g" = arg min 9EG
Σ
[xx 22) +
Σ
1(
|.2n
==g(z)=0
mng(ar)=1
Then, (g) is more tightly bounded by epi (g) + (((za
Proof. Recall that A d-ditannce between two distributions 2d and 2a is defined as:
dz(2 2*) = sup geG
Pr(g(2)
= Pr Pr(g(2) = 1] 2
24
Recall that the Censen-Shamme (JS) divergence between two distributions 2d and 2a is defined as:
Dss(2"12*) 1
+ Dkx
where = (2 + 2*) the mixture distribution and Dx is the Kullback-Leiblrr (KL) divergence. Using Pinsker's Inequality Csiszár & Körner (2011), we have:
2zd 201 - (DKL(Z + DxL(2')
This implies:
/ + DxL DxI(2*I|M))
2 2 21 S
Thus, we can rewrite the TV distance bound in terms of the IS divergence:
- - 10 S DDs((zdl2))
From Lemma 2, we know that the A-distance is bounded by twice the TV distance:
dz(2 < 2| Z
Using the bound the TV distance in terms of the JS divergence, get:
dz(Z < Djs(Zal
26

---

Published as a conference paper at ICLR 2025
As Equation (4) is Donsker- Varadhan representations of KL divergence Donsker & Varadhan (1983) that approximates the mutual information using a neural network (MLP) the bound on mutual information in terms of the JS divergence is:
[zd: 2,) = Dxz(2")23
2 2 DKL
1 1 (x (Define M M =Dss(Zu12))
- 2/224 + 2*))
Substituting this into the A-distance bound, we get:
dz(2", 2*) S 2 DJs D)s(221122))) S 2 T(za5 2s).
Given that g" the optimal function Given that g" the optimal function from the of hypotheses G, we consider the bound of de (S2 (22) stated in Lemma Specifically, we have:
do(S2" 5
1
min gEG
1(x E 52") + |228
Σ
1(x ((EE
==g(=)=0
2:9(=)==
d dllog(2(22)) log s
(log(2 log 2
+ 4 max
|S2n|
|S2$|
The minimum of this bound cannot lower than Considering the mutual information [zai Ro). we know that:
0 $ [Zdi 2x) S 1.
Therefore, it follows that:
2 [(zd; 2.) 2.)55 22
This implies that 2 (Tzz 2x) can serve as a tighter upper bound for dS .2²).
Replacing the tighter bound of do(",, 22) in the proof steps of Theorem have
2(e (g) e (g)) < 1(2ss 2o) ep (g) S ezi (g) + ((Zdi
27

---

Published as a conference paper at ICLR 2025
E EFFECT OF MUTUAL INFORMATION MINIMIZATION ON DIFFERENT DATASETS
Our comprehensive analysis of HiLo's performance across diverse datasets, with and without Mutual Information (MJ) regularization, presented Figure The results demonstrate that regulariza- tion's efficacy is particularly pronounced in two distinct scenarios: datasets with markedly distinct low-level styles (e.g., Quickdraw, Infograph) and those with closely related semantic categories (e.g., SSB-C benchmark). In the case of Quickdraw, the dramatic style variations are readily captured by low-level features, allowing MI regularization to effectively disentangle semantic features from low-level information. Note that Quickdraw samples still too abstract to learn from constrastive learning, leading to poor performance the model. For the SSB-C benchmark, where image struc- ture remains largely unchanged across different noise types, MI regularization proves crucial in distinguishing subtle semantic differences from low-level information variations.
Quickdraw
wio MI
Clipart
Skelch
Infograph
Painting
CUB-C
FGVC-C
Scars:C
Figure 6: Comparison between HiLo with and without MI across different datasets. DomainNet is generic dataset with disparate styles (e.g., Painting, Quickdraw, Sketch, Clipart and Infograph). The labelled data are all from 'Real' domain. SSB-C (e.g.. CUB-C, Scars-C and FGVC-C) is created by adding several common noises in the real world to the fine-grained datasets. The labelled data are all from the original SSB. When low-level style is guite different (e.g.. Infograph, Quickdraw) semanties are close (e.g.. SSB-C benchmark). the improvement of MI regularization is pronounced.
28

---

Published as a conference paper at ICLR 2025
F NOVEL CATEGORY DISCOVERY IN THE PRESENCE OF DOMAIN SHIFTS
In this paper, we consider challenging problem of generalized category discovery with domain shifts, which, to our knowledge, has been studied the literature. However, the study of novel category discovery under domain shifts has been considered in Yu et al. (2022) from a domain adaptation perspective, which introduces a self-labeling framework, called NCDD, that can categorize unlabelled images from both source and target domains, by maximizing mutual information between labels and input images. The unlabelled images from the target domain may contain images from new categories that are not present in the source domain. Differently, in our study, we consider that unseen classes are also present in the unlabelled data from the source domain (i.e., the domain f2"), and new domains may appear at test time. Meanwhile, our HiLo framework also differs significantly from NCDD. Particularly, HiLo learns to disentangle Somaiicsemantic features by minimizing the mutual information between domain and semantic heads. It also incorporates novel PatchMix contrastive learning method and curriculum learning approach to facilitate the robustness of representation to domain shifts. To compare HiLo with NCDD, reimplement the NCDD method' and experiment on the experimental configuration following Yu et al. (2022) on the CUB-C. We present the results in Table 8. As can be seen, HiLo significantly outperforms NCDD and all other baselines, highlighting its effectiveness on domain-semann disentanglement.
Table 8: Evaluation on SSB-C datasets. We report results of baselines in the seen domain (i.e., Original) and the overall performance of different corruptions (i.e., Corrupted).
CUB-C
Original
Corrupled
Methods
All
Old
New
All
Old
New
RtakStats+
19.3
22.0
15.4
13.6
23.9
4.5
UNO+
25.9
40.1
21.3
21.5
33.4
8.6
ORCA
18.2
22.8
14.5
21.5
23.1
18.9
NCDD
37.0
50.7
28.7
30.2
53.0
11.7
GCD
26.6
27.5
25.7
25.1
28.7
22.0
SimGCD
31.9
28.8
31.6
25.0
(Ours)
52.0
53.6
'Our NCDD reimplementation's performance aligns with other efforts et al. (2023)
29

---

Published as a conference paper at ICLR 2025
G INVESTIGATION OF CLIP FOR GCD WITH DOMAIN SHIFTS
CLIP Radford et al. (2021) has demonstrated strong performance in various computer vision tasks. We thus investigate its potential for the challenging problem of GCD with domain shifts. We employ the pretrained vision transformer from CLIP as the backbone for HiLo. As illustrated in Table employing CLIP significantly improves the performance of HiLo on DomainNet, compared with the DINO-based HiLo and SimGCD.
Table 9: Effectiveness of employing CLIP the backbone for HiLo. We select the 'Real' and 'Painting' domains to train the DINO model with the techniques introduced above as the baseline.
Real
Painting
Methods
Old
New
All
Old
New
Bascline
SimGCD Wen al. (2023)
61.3
77.8
52.9
34.5
35.6
33.5
Pretrained CLIP
69.8
77.2
58.9
37.1
38.0
35.1
Bascline
HiL.o
64.4
77.6
57.5
42.1
42.9
41.3
Pretrained CLIP
74.5
78.1
47.1
49.5
45.4
Table 9, verifies that strong visual encoder can bring performance boost on both seen and novel domains. In Table 10, we further compare with another two CLIP baselines to better understand the potential of the visual language model, i.e., zero-shot CLIP with oracle class names (which are not expected to be unavailable in GCD) and with zero-shot CLIP a very large vocabulary (i.e., WordNet Miller (1995)), where we conduct zero-shot inference using the class names of both known and unknown classes, by comparing the visual feature of each image and the text features of class descriptions. The results in Table 10 demonstrate that CLIP models do not enhance robustness compared to our HiL.o model, a visual-only model, on CUB-C, despite that an extra vocabulary is provided for the CLIP model (which arguably reduces the difficulty of the GCD task, in which do not assume any extra textual or visual knowledge on the unlabelled data). This finding aligns with Taori et al. (2020), which indicates that robustness under natural distribution shifts does not necessarily translate robustness under synthetic distribution shifts, thereby suggesting the limited impact of CLIP models on covariate shifts. HiLo outperforms CLIPi and CLIPF, despite that they 'cheat' by using extra vocabulary, which further underscores the effectiveness and robustness of HiLo and the challenge of GCD with domain shifts.
Table 10: Zero-shot performance of CLIP on CUB. CLIPi is the CLIP with oracle class names, while CLIPÈ is the CLIP with large vocabulary (ie.. WordNet Miller (1995)).
CUB-C
Original
Corrupted
Methods
All
Old
New
All
Old
New
StatSt
19.3
22.0
15.4
13.6
23.9
4.5
UNO+
25.9
40.1
21.3
21.5
33.4
8.6
ORCA
18.2
22.8
14.5
21.5
23.1
18.9
GCD
26.6
27.5
25.7
25.1
28.7
22.0
SimGCD
31.9
33.9
29.0
28.8
31.6
25.0
HiLo (Ours)
56.8
54.0
60.3
52.0
53.6
50.5
CLIP
55.5
51.6
57.4
50.3
51.8
48.9
CLIP
55.1
51.0
57.1
49.6
51.4
47.8
30

---

Published as a conference paper at ICLR 2025
H DETAILED EVALUATION OF SSB-C DATASETS
In addition to the overall SSB-C results presented in the main paper. we provide detailed analysis of CUB-C, Scars-C, and FGVC-C against various corruptions in Table 12 and Table 13. Our proposed method consistently outperforms the baselines. Notably, while Gaussian, Speckle, Impulse, and Shot noise corruptions appear qualitatively similar, their performance impacts differ significantly. Specifically, Speckle noise has less detrimental effect performance compared to other noise types. As illustrated Figure 4, Speckle noise preserves more semantic information, whereas other noises pervade the images. This retention of semantic information is crucial for accurate object recognition fine-gr aineggrined settings, explaining the consistently better performance on Speckle noise compared other corruption types.
Table 11: A detailed evaluation of the CUB-C dataset. We assess the performance of each individual corruption.
Nelse
Sane
Find
Speciic
Spatic
Marhods
AE
New
XII
New
AE
03
Nex
Al
GM
Nev
AB
Oid
New
A3
Nev
AI
Cid
Nuw
A3
Ney
All
Cld
Now
Radddabe
12.
21,5
121
11.
355
363
115
38A
162
-1.3
16.
22.1
46.3
4.7
DBCA
21.5
23.1
1039
112
211
162
22
189
223
220
25.8
16.
162
214
224
26.
31.3
16.3
219
196
310
311
276
267
36.3
23.4
314
343
336
Table 12: A detailed evaluation of the Scars- C dataset. We assess the performance of each individual
corruption.
Guaviaan Neioo
Sky. Neise
Iegedie Nose
Zeees Bla
Sone
Fres
Pag
Speckle
Syatker
UNO-
13.0
24.5
140
60
84
312
28
26.9
43
280
91.
186
13.
21.5
35.4
4.0
OCD
11.4
168
131
112
118
211
312
80
25.
84
212
31.
41.
226
161
312
43.3
2.1
16.1
127
183
133
134
327
131
264
319
260
365
234
308
324
13.1
314
36.8
263
3L5
363
348
363
366
359
36A
651
3LS
368
380
365
63
24.
067
465
32.2
311
311
286
364
15.1
36.6
Table 13: A detailed evaluation of the FGVC-C dataset. We assess the performance of each individual corruption.
Nelaah
Al
Ol
New
ON
New
Al
Ol
New
XlI
Nev
AB
Nes
X1
DNI
Nov
MI
ON
New
AI
CH
Nev
All
Ol
Now
Rad-Sabe
7.3
13.6
3.0
6:3
167
58
5.0
103
51
13.1
159
43
63
125
38
5.
112
81
125
314
45
75
140
53
165
20.
4.9
OBCA
11.
17.3
11.1
11
11.
100
133
163
13.
145
85
13.
911
136
23.1
93
163
38.
91
13
18%
19.
150
25
9.3
SarOCD
16.3
11.
142
168
16.5
259
264
22.4
193
245
252
51.
978
168
180
173
203
X2.
28,6
31

---

Published as a conference paper at ICLR 2025
I ADDITIONAL EXPERIMENTAL RESULTS ON DOMAINNET
As discussed in Section 4.1 and Section 4.2 in the main paper, among the 6 domains in DomainNet, we utilize the 'Real' domain as Ω2" and each of the other domains serves as Ω in turn. Note that the model is fitted on the partially labelled data from domain S2, which contains labelled and novel classes, and the fully unlabelled data from domain Ω Therefore, though the model does not 'see' the novel classes in both domains S2" and Ω², it does 'sec' the unlabelled data from both domains, regardless of whether the images are from labelled or novel classes. To more comprehensively measure the model's capability, further evaluate the performance the unlabelled images from the remaining 4 domains aside from S?" and S2
In Table 14, we report the results by considering the 'Infograph' domain as Ω 'Others' denotes the results on the unlabelled data from the remaining 4 domains aside from 'Real' and 'Infograph'. Table 15 shows the evaluation on each domain in 'Others'. Similarly, Table 16 and Table 17 show the results by considering 'Quickdraw' domain as Ω2; Table 18 and Table 19 show the results by considering 'Sketch' domain as 52"; and Table 20 and Table 21 show the results by considering 'Clipart' domain as 52 Our HiLo framework consistently outperforms baseline methods in both 'All' and 'New' performance. Notably, the 'Quickdraw' domain presents greater challenges than other domains due to its highly abstract and lifficult-to-reooghil images, resulting in unsatisfactory performance for all methods.
Table 14: Evaluation the DomainNet dataset. The model is trained the 'Real' and 'Infograph' domains and we report the respective results on 'Real', 'Infograph' and the remaining four domains (i.e., OOthers)..
Real
Infograph
Others
Methods
All
Old
New
All
Old
New
All
Old
New
RankStats+
34.2
62.4
19.6
12.5
21.9
6.3
18.5
32.1
6.4
UNO+
42.8
69.4
29.0
10.9
15.2
8.0
18.2
28.0
9.6
ORCA
29.1
47.7
20.1
13.7
7.1
13.8
24.8
GCD
41.9
46.1
39.0
10.9
17.1
8.8
19.0
29.1
11.1
SimGCD
52.7
67.0
44.8
11.6
15.4
9.1
20.8
28.4
14.2
HiLo (Ours)
64.2
78.1
57.0
13.7
16.4
11.9
23.0
28.5
18.3
Table 15: Evaluation on the DomainNet dataset. Besides the overall performance given in Table 14, we show a detailed performance breakdown for each domain in 'Others'.
Painting
Quickdraw
Sketch
Clipart
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
RankStats+
29.6
49.2
10.0
2.5
1.6
3.4
17.4
32.2
6.5
24.4
45.5
5.8
UNO+
30.8
44.8
16.8
2.7
2.3
3.1
17.0
27.0
9.7
22.3
37.8
8.7
ORCA
20.0
40.2
8.1
1.6
1.8
1.2
13.2
21.1
8.0
20.5
36.0
4.1
GCD
30.8
45.1
18.4
3.6
4.7
2.5
18.8
26.4
11.2
22.9
40.0
12.3
SimGCD
35.9
45.6
26.3
1.7
2.5
20.8
29.3
14.5
24.5
36.9
13.6
Hil (Ours)
35.8
36.3
Table 16: Evaluation the DomainNet dataset. The model is trained the 'Real' and 'Quickdraw' domains and we report the respective results on 'Real', 'Quickdraw" and the remaining four domains (i.e., "Others').
Real
Quickdraw
Others
Methods
All
Old
New
All
Old
New
All
Old
New
RankStats+
34.1
62.5
19.5
4.1
4.4
3.9
21.0
37.4
7.2
UNO+
31.1
60.0
16.1
6.3
5.8
6.8
18.6
32.2
7.0
ORCA
19.2
39.1
15.3
3.4
3.5
3.2
15.6
28.4
8.1
GCD
37.6
41.0
35.2
5.7
4.2
6.9
21.9
34.3
12.2
SimGCD
47.4
64.5
37.4
6.6
5.8
7.5
22.9
33.8
13.8
HiLo (Ours)
58.6
76.4
52.5
7.4
6.9
8.0
25.9
32.5
20.4
32

---

Published as a conference paper at ICLR 2025
Table 17: Evaluation the DomainNet dataset. Besides the overall performance given in Table 16,
we show detailed performance breakdown for each domain in 'Others'.
Painting
Sketch
Clipart
Infograph
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
RankStatst
29.6
49.0
10.2
17.1
32.1
6.1
24.8
45.4
6.7
12.6
23.1
5.7
UNO+
26.8
43.7
9.9
14.7
25.6
6.6
20.7
38.4
5.1
12.2
21.0
6.4
ORCA
22.2
40.9
10.1
11.9
22.4
7.1
17.5
10.3
18.7
GCD
32.9
45.7
21.4
18.5
30.5
10.8
23.5
39.0
10.7
13.8
22.1
7.6
SimGCD
33.8
45.1
22.5
19.4
30.1
11.5
24.0
11.4
14.5
21.6
HiLo (Ours)
38.6
45.1
32.2
22.9
28.8
18.5
26.0
36.4
16.9
16.2
19.8
13.9
Table 18: Evaluation on the DomainNet dataset. The model is trained on the 'Real' and 'Sketch' domains and we report the respective results 'Real', 'Sketch' and the remaining four domains (i.e., 'Others').
Real
Sketch
Others
Methods
All
Old
New
All
Old
New
All
Old
New
RankStats+
34.2
62.0
19.8
17.1
31.1
6.8
17.3
30.0
6.1
UNO+
43.7
72.5
28.9
17.0
9.2
17.4
26.4
9.5
ORCA
32.5
50.0
23.9
11.4
14.5
7.2
13.3
23.1
GCD
48.0
53.8
45.3
16.6
22.4
11.1
20.7
25.8
15.8
SimGCD
62.4
77.6
54.6
16.4
20.2
13.6
20.4
25.4
16.1
HiLo (Ours)
63.3
77.9
55.9
19.4
22.4
17.1
21.3
25.8
17.4
Table 19: Evaluation on the DomainNet dataset. Besides the overall performance given in Table 18, we show a detailed performance breakdown for each domain in 'Others'.
Painting
Quickdraw
Clipart
Infograph
Methods
All
New
All
Old
New
Old
New
Old
New
RankStats+
29.7
10.2
2.1
2.4
45.9
22.6
UNO+
30.8
44.1)
17.6
2.4
2.4
2.3
23.1
38.0
10.1
13.2
21.2
7.9
ORCA
23.1
17.2
3.0
2.0
33.1
18.1
GCD
32.6
40.1
31.5
1.6
1.9
1.5
24.1
31.1
14.9
14.1
16.2
10.2
SinGCD
38.7
32.7
1.2
2.5
35.3
20.3
HiLo (Ours)
39.8
44.7
34,9
1.9
2.0
1.7
27.2
35.9
19.6
16.2
20.5
13.4
Table 20: Evaluation on the DomainNet dataset. The model is trained on the 'Real' and 'Clipart' 'Real", 'Clipart' and the remaining four domains (i.e.,
domains and we report the respective results 'Others').
Real
Clipart
Others
Methods
All
Old
New
All
Old
New
All
Old
New
RankStatst
34.0
62.4
19.4
24.1
45.1
6.2
15.8
27.0
6.4
UNO+
44.5
66.1
33.3
21.9
10.1
16.2
23.2
ORCA
32.0
49.7
23.9
19.1
31.8
4.3
13.7
GCD
47.7
53.8
44.3
22.4
34.4
16.0
18.0
24.1
12.1
SimGCD
61.6
77.2
53.6
23.9
31.5
17.3
19.2
23.6
15.6
HiLoo (Ours)
63.8
77.6
56.6
27.7
34.6
21.7
19.8
23.6
16.8
Table 21: Evaluation on the DomainNet dataset. Besides the overall performance given in Table 20, we show detailed performance breakdown for each domain in 'Others'.
Painting
Quickdraw
Sketch
Infograph
Methods
All
Old
New
All
Old
New
All
Old
New
All
Old
New
RankStats+
30.0
9.7
2.6
2.3
2.9
17.4
31.9
6.8
13.1
23,6
UNO+
31.5
43.3
19.6
2.8
2.1
3.6
17.3
26.8
10.2
13.3
20.6
8.5
9.2
21.9
GCD
33.4
40.4
22.2
3,6
5.7
2.2
19.5
27.7
12.7
15.5
22.7
11.1
SimGCD
Hiloo (Ours)
40.7
46.3
35.1
1.3
0.4
2.3
21.2
26.9
17.0
15.9
20.6
12.8
33

---

Published as a conference paper at ICLR 2025
J HILO ON THE VANILLA GCD SETTING
Although not explicitly designed for the vanilla GCD, we evaluate HiLo's effectiveness without domain shifts. We perform experiments on ImageNet-100 and SSB, Our HiLo framework outperforms the state-of-the-arl GCD method, as indicated in Table 22. We hypothesize that subtle covariate shifts may still be present within the same distribution (e.g.. varying 'Real' backgrounds with identical semantics), which can still be handled by HiLo effectively.
Table 22:
Evaluation of HiLo on ImageNet- 100 and SSB under the vanilla GCD setting. HiLo achieves better results than the SoTA GCD method.
ImageNet-100
SSB
Method
All
Old
New
All
Old
New
SimGCD
83.0
93.1
77.9
56.1
65.5
51.5
Hioo (Ours)
83,4
93.5
78.1
59.2
66,2
54.9
34

---

Published as a conference paper at ICLR 2025
K EFFECTS OF DIFFERENT OUTPUT DIMENSIONS FOR THE SEMANTIC AND DOMAIN HEADS
In the main paper, we assume access the ground-truth values of both the semantic class and domain (i.e,, kg and k,). However, real-world scenarios, these values are often unknown. Therefore, essential to assess the stability of our model's ' performance when assigning guesses to the varying output quantities of semantic class and domain type.
We employ different output dimensions for the semantic head and domain head. For the semantic head, we experiment with k, E {200, 1000, 2000, 5000, 10000) using all 5 severity levels. For the domain head, experiment with ky E (2, 10, 20, 50, 100} and corruptions with the highest severity level. Table 23 reports the accuracy with different k; and ku values, with the optimal number utilized to fix one output size while exploring the other. The highest performance is achieved when
k2 = y U V and kę = 10. Performance declines with increasing he or kg. As it tractable to roughly estimate the number of domains the model may handle, our method's 's insensitivity to the domain axis output size selection.
Table 23: Sensitivity analysis of the output size on CUB-C dataset. The inappropriate selection of kg and Ka would predispose poor performance for the semantic head while the domain head is relatively robust the output size.
Sem. Head
Dom. Head
Original
Comupted
Original
Corrupted
Size
Old
New
All
Old
New
Size
All
Old
New
All
Old
New
k. 56.8
60.3
52.0
53.6
50.5
43.5
45.8
40.2
35.1
32,9
K, 1000
47.5
51.0
35.1
38.1
48.3
30,9
ky 10
44.2
46.2
43.0
36.3
39.1
34.7
K.
31.4
43.0
44.6
32.3
K. 30.7
K. 5000
43.1
22.2
23.8
28.1
21.8
kg 500
37.5
41.4
34.9
30.3
33.1
28.7
12.1
13.9
35.0
38.3
27.3
35

---

Published as a conference paper at ICLR 2025
L UNKNOWN CATEGORY NUMBER
As the total number of semantic categories cannot be accessed in the weal-world setting, we evaluate HiLo with an estimated number of categories using an off-the-shelf method Vaze et al. (2022) on CUB (see Table 24). We find that our method consistently outperforms the strong baseline when the exact number of categories is unknown.
Table 24: Performance of HiLo and the baseline method SimGCD with an estimated number of categories on CUB. Bold values represent the best results. "GT" denotes the ground truth; 'Est,' denotes the estimation.
Original
Comuuted
Method
[C]
All
Old
New
All
Old
New
SimGCD Wen al. (2023)
GT
31.9
33.9
29.0
28.8
31.6
25.0
HiLeo (Ours)
GT (200)
56.8
54.0
60.3
52.0
53.6
50.5
SimGCD Wen al. (2023)
Est. (257)
29.5
32.4
28.0
27.6
29.7
24.1
HilLo (Ours)
Est. (257)
55.9
52.9
59.2
51.2
5288
49.5
36

---

Published as a conference paper at ICLR 2025
M HYPERPARAMETER CHOICES FOR HILO COMPONENTS
The hyperparameters of HiLo can be grouped via each component: (a) PatchMix (i.e. Ba); (b) representation learning and parametric classification losses (i.e., т, A, e); (c) curriculum learning (i.e., Tu. r' and t'). We follow Zhu et al. (2023); Wen et al. (2023) to choose values for the shared hyperparameters in (a) and (b) respectively.
As summarized in Table 25, choose the hyperparameters in (a) and (b) following Zhu al. (2023) and Wen al. (2023) respectively. For the hyperparameters in (c), we choose the values through the validation split of the labelled data in the 'Orignal' domain.
Table 25: Hyperparameter choices for HiLo components.
Hyperpuramctens
Value
Deseriptions
Suggested Following
(2023)
T2
Suggested following
(2023)
5
Suggested following
(2023)
7
Suggested volues (ollowing Wen (2023)
X
Suggested values following Wen (2023)
5
Beta(log[] lo( +
Suggested value following Zhu et (2023)
0.1
Choose through the validation split of the labelled data in the 'Orignall domain (see Figure 7(a))
0.05
Choose through validation split the labelled data the "Orignal" domain Figure 7b1)
76
validation through validation splir
labelled
"Orggall "Orignal"
Higure Figure 7(d))
In Figure 7, we report results on CUB-C with varying values of E, ro, r', E' that are specific to HiLo. We find that the order of samples (determined by To, r', .t') with different difficulties has a great influence performance both the source domain and target domains.
16.6
60.9
400
MOD
664
0.1
2.05
0.21
2.001
C.11
2.1
0.003
0,25
0.5
85
153
(6)
intttooono Depiat becppggeeeeeecccrrgg Figure The impact varying values of e, Tox and investigated on CUB-C dataset. Hyperparameters for curriculum sampling (i.e., ro, r have great influence on performance on both source domain and target domains.
37

---

Published as a conference paper at ICLR 2025
N EFFECTS OF LEARNING RATES
As learning rate is key hyperparameter for all methods, we present results using different learning rates for our method and the baselines the CUB-C datasets. We experiment with three different learning rates, 0.1, 0.05, and 0.01, for all the methods, using the SGD optimizer with the suggested weight decay and momentum in the original papers. 0.1 appears to be the best choice among the three values for RankStats+, UNO+, and SimGCD, while 0.05 better choice for our method. Among the compared methods, we can see that the performance variation is relatively large for GCD and SimGCD among these three values. The variation is relatively small for RankStat+, UNO+, and ORCA, while their performance notably inferior GCD and SimGCD. In contrast, our method has a very small performance variation while significantly outperforms all other methods.
Table 26: Performance comparison CUB-C with three different learning rates.
Original Old
Corrupted Old
Method
Learning Rate
All
New
All
New
0.1
19.3
22.0
15.4
13.6
23.9
4.5
RankStat+
0.05
17.1
24.9
12.7
11.9
16.7
8.5
0.01
15.0
17.1
10.7
9.1
15.5
3.8
0.1
25.9
40.1
21.3
21.5
33.4
8.6
UNO+
0.05
23.8
37.2
18.8
20.2
34.0
7.1
0.01
22.8
35.7
17.9
19.5
33.2
5.8
0.1
17.3
22.6
13.8
20.9
22.6
17.4
ORCA
0.05
18.2
22.8
14.5
21.5
23.1
18.9
0.01
17.4
22.1
13.2
20.8
23.6
15.8
0.1
26.6
27.5
25.7
25.1
28.7
22.0
GCD
0.05
24.7
25.4
23.8
24.0
28.2
20.8
0.01
48.1
53.1
47.0
33.1
37.2
29.9
0.1
31.9
33.9
29.0
28.8
31.6
25.0
SimGCD
0.05
29.2
30.7
27.1
25.0
26.5
24.0
0.01
26.3
27.0
25.9
21.8
21.4
23.5
0.1
56.0
54.1
58.9
50.8
52.4
48.1
Ours
0.05
56.8
54.0
60.3
52.0
53.6
50.5
0.01
54.7
60.1
56.4
48.1
49.0
47.6
38

---

Published as a conference paper at ICLR 2025
O STABILITY OF DIFFERENT METHODS
As the differences in the results of the GCD benchmark tests can be very large, we obtain the averaged results Table and Table 3 by conducting three independent runs for each method on both DomainNet and SSB-C. Here we visualize the bar chart 'All' classes ACC for each methods and list the corresponding error lines generated by the these independent runs. We notice that the error bars of ORCA and SimGCD exhibit significant oscillations.
Metiess
RaniStaits
UNO
ORCA
50
Accuracy (%)
HiLL (Dun)
40
30
20
10
Real
Painting
Sketch
Quickdraw
Clipart
Infograph
Datasets
(a) DomainNet
60
Methoc
RankStatts
UNO-
ORCA
Accuracy (%)
40
SimOCD
30
20
CUB-C (Original)
CUB-C (Corrupted)
Scars-C (Origina Datasets
Scars-C (oorupted)
FGVC-C (Original)
FGVC-C (Corrupted)
(b) SSB-C
Figure 8: The 'All' ACC results are averaged by conducting three independent runs for each method on both DomainNet and SSB-C.
39

---

Published as a conference paper at ICLR 2025
P QUALITATIVE RESULTS
We provide qualitative results on DomainNet and CUB-C. In Figure 9, we present the visualization by first applying PCA to the domain features and semantic features obtained through H. and then plotting the corresponding images. As can be seen, the images are naturally clustered according to their domains and semantics, demonstrating that HiLo successfully learns domain- domain-specific and ietantic-specifer features.
Dom head
Sem head
DomainNet
CUB-C
Figure 9: Visualization of domain and semantic features via projecting them through PCA. We randomly sample instances from the entire dataset and apply PCA project the semantic and domain features into 2-dimensional space. The domain branch tends to cluster images based on covariate features, while the semantic branch clusters images based on categories. Best viewed in PDF with zoom.
The attention map offers valuable insights into the focus of Transformer-basee based models on the input. We obtain the attention maps for the CLS token from multiple attention heads the final layer of the ViT backbone, highlighting the top 10% most attended patches in Figure 10. We observe that, compared with the baseline, Hil.o is much more effective in focusing on the foreground object even in the presence of significant domain shifts (e.g., painting style, foggy weather). This demonstrates that HiLo is robust to domain shifts and remains unaffected by potential spurious correlations between semantic features and levelevel statistics.
40

---

Published as a conference paper at ICLR 2025
DomainNet
CUB-C
SimGCD
HilL
Figure 10: Visualization of the attention map for different heads in the last layer on DomainNet and CUB-C. We highlight the attended regions with top 10% contribution in red. Compared with SimGCD, the attention maps of HiLo consistently focus on the foreground object without affecting by the strong domain shifts of painting style and foggy weather.
41

---

Published as a conference paper at ICLR 2025
Q BROADER IMPACTS AND LIMITATIONS
Our study aims to extend AI systems' capabiliiies from closed-world to open-world scenarios. particularly enhancing next-generation AI systems to categorize and organize open-world data autonomously. Despite promising results on public datasets, our method has limitations. First, interpretabilityy needs improvement, as the underlying decision-making principles remain unclear. Second, cross-domain robustness is inadequate. Although our method has achieved the best overall and new class discovery results in the GCD setting with domain shifts, performance still has significant room for improvement. Third, the novel domains we investigated in the paper are still limited. Domain and class imbalance present additional challenges in GCD scenarios. Our current method was specifically developed to handle these issues, which are important areas for future work.
42


Reviews:
[{"review_id": "wj1zyCvgGp", "reviewer": "Reviewer_6eyX", "summary": "This paper introduces a new challenge for Generalized Category Discovery, which requires model to categorize unlabeled data in the presence of domain shifts. Traditional GCD methods assume all images come from the same domain, which leads to a significant performance drop under domain shifts. The proposed HiLo framework explicitly disentangles semantics and domain, achieving domain adaptation in GCD through patchmix and curriculum learning. Experimental results show performance improvements, validating the effectiveness of the approach.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a new, practically meaningful, and challenging setting, and constructs corresponding datasets.\n2. The domain-semantic disentangled design is well-reasoned, clearly aligning the motivation.\n3. The proposed approach demonstrates significant performance improvement on SSB-C.\n4. The writing is clear and easy to follow.", "weaknesses": "1. The performance gain on DomainNet is considerably smaller than on SSB-C, and the improvement over methods like SimGCD, which does not account for domain shift, is modest. This indicates limited robustness on various domain shifts and fails to highlight the advantages of the proposed approach.\n2. The method is sensitive to certain hyperparameters, and $r^{'}$ does not exhibit consistent performance across the original and new domains.\n3. The approach of decoupling domain and semantics is derived from [1], and the use of patchmix for domain adaptation is adapted from [2]. The curriculum learning strategy is also straightforward. Overall, the method seems to be an assembly of prior works, lacking substantial novelty.\n4. There is no analysis of the disentangled domain and semantic features, such as distribution visualizations. This would help illustrate the effectiveness of the disentanglement.\n5. In line 287, same representation loss $L^{rep}_s$ on both domain and semantic features is confusing. This approach may lead domain features to capture information beyond true domain characteristics. It would be valuable to see t-SNE visualizations of domain features, semantic features, and their combination. The author does not provide a corresponding discussion.\n6. Line 313 mentions using pre-trained DINO to obtain $z_d$, but previously $z_d$ is associated with a projection head. If the projection head is discarded, then $z_d$ will always be identical in different time steps. If it is retained, the term “pretrained” is confusing. This needs clarification.\n7. The ablation study is somewhat unclear. For instance, in row (5) where only deep features are used, does this mean all other designs related to the shallow feature $z_d$ are also omitted? This also needs clarification.\n\nReference\n\n[1] Learning deep representations by mutual information estimation and maximization\n\n[2] Patch-mix transformer for unsupervised domain adaptation: A game perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KnY7vvQcN3", "reviewer": "Reviewer_RNZQ", "summary": "Generalized Category Discovery (GCD) is a challenging task where, given a partially labeled dataset, the model must classify all unlabeled instances. This paper introduces a new task and method to handle the GCD problem when the unlabeled data contains images from different domains. In terms of the method, the HiLo architecture and learning framework involves extracting \"low-level\" (early layers) and \"high-level\" (late layers) features from a vision Transformer and decoupling domain and semantic features by minimizing the mutual information between the two sets of features. The PatchMix contrastive learning method is introduced into the GCD task, with its objective function extended to enable the utilization of both labeled and unlabeled data for training. Curriculum learning is adopted, gradually increasing the sampling probability weight of samples predicted to be from unknown domains to enhance the model's robustness to domain shifts. Experiments are conducted on the DomainNet and the SSB-C benchmark datasets constructed based on the Semantic Shift Benchmark (SSB). The experimental results show that HiLo significantly outperforms existing category discovery models, validating the effectiveness of the method.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The HiLo architecture extracts features from different layers of the vision Transformer and decouples domain and semantic features by minimizing mutual information. This feature processing method, based on the neural network hierarchical structure and information theory, provides a more effective feature representation for category discovery in the presence of domain shifts and avoids the problem of feature confusion in traditional methods.\n\n2. The PatchMix method is introduced into the GCD task and innovatively extended. By adjusting its objective function, it can adaptively utilize labeled and unlabeled data for training. This extension not only combines the advantages of data augmentation but also flexibly adjusts the learning process according to the nature of different data, enhancing the model's ability to learn data from different domains and categories.\n\n3. The curriculum learning method is employed, which dynamically adjusts the sampling probability weights according to the difficulty of samples and the unknown degree of domains. This strategy of gradually introducing samples from easy to difficult conforms to the learning law, enabling the model to better adapt to the challenges brought by domain shifts and improving the model's convergence speed and robustness to complex data distributions.\n\n4. In terms of method design, innovative technical architectures and learning strategies are used, as well as theoretical analyses to verify their effectiveness. From the theoretical derivation of the target error to the analysis of the roles of different components, a solid theoretical foundation is provided for the innovation of the method, demonstrating the advantage of the close integration of theory and practice.", "weaknesses": "1. In HiLo, features are disentangled by assuming that features from different layers represent domain and semantic information, respectively and minimizing the mutual information based on this assumption. However, this assumption may oversimplify the complexity of feature representation in neural networks. In fact, features from different layers may be a mixture of multiple types of information. Simply defining the early layers as domain features and the late layers as semantic features may not be entirely accurate, which may lead to incomplete feature disentanglement in some complex data distributions and affect the performance and generalization ability of the model.\n\n2. The introduction and extension of PatchMix in the GCD task is an innovation, but it also brings some problems. The adjustment of its objective function and its application on different data increases the complexity of the model. When dealing with data with large domain differences, it is a challenge to determine the mixing proportion and application method accurately. If not handled properly, it may introduce too much noise or incorrect information, which may instead interfere with the learning process of the model and reduce the classification performance.\n\n3. In the curriculum learning method, the adjustment parameters of the sampling probability weights need to be selected through the validation set, which increases the dependence of the model on specific datasets. Moreover, for different datasets and tasks, the optimal values of these parameters may vary greatly, and the model cannot adaptively determine these parameters. If these parameters cannot be correctly selected in a new dataset or task, curriculum learning may not be able to play its intended role. It may even have a negative impact on the learning of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2eXFwg3Iak", "reviewer": "Reviewer_L4eA", "summary": "The paper introduces the HiLo framework, a learning method aimed at tackling Generalized Category Discovery (GCD) under domain shifts. HiLo addresses challenges in categorizing both seen and unseen categories across distinct domains within partially labeled datasets, leveraging a multi-faceted approach: mutual information minimization to separate domain and semantic features, PatchMix for augmented domain adaptation, and a curriculum learning strategy. The proposed method is evaluated on synthetic and real-world domain-shift datasets, showing substantial improvements over existing GCD models.", "questions": "1. For the experimental results of the ORCA method, what is the backbone used by the authors?\n2. Were any curriculum learning alternatives considered, such as adaptive weighting based on difficulty or dynamic sample weighting? A brief discussion on these choices would clarify why the current approach was favored.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents an innovative GCD approach by combining mutual information minimization with domain-specific data augmentation and curriculum learning to handle domain shifts effectively.\n2. Extensive evaluation on both synthetic (SSB-C) and real-world (DomainNet) benchmarks demonstrates the model's robustness and its superiority over baseline GCD models, especially under domain-shifted conditions.", "weaknesses": "1. In the \"Problem statement,\" the following sentence is unclear: \"The objective of GCD is ... with singleton cardinalities for the latter.\" The author needs to differentiate between the GCD task setting and the domain shift GCD task setting, so this statement should be revised for clarity and precision.\n2. The font sizes of the tables are not standardized, and the font in table 2 is too small.\n3. I'm curious as to how many runs each of the authors' experimental results were derived from, and given that the differences in the results of the GCD benchmark tests can be very large, the authors should have listed the error lines generated by the three independent runs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yf0ZnapsUW", "reviewer": "Reviewer_jy36", "summary": "This paper introduces a new problem setting: Generalized Category Discovery (GCD) with domain shift. The authors leverage techniques from domain adaptation and curriculum learning to propose a new method called HiLo. Comprehensive experiments on the proposed benchmark demonstrate substantial improvements.", "questions": "Please clarify the novelty of the proposed method, and include more comparisons with UinOT in the main results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a new problem setting and proposes a HiLo, which combines multiple techniques from domain adaption and achieves better results.", "weaknesses": "1. The novelty of the method appears limited, as it seems to combine various techniques from different domains.\n\n2. The comparison with UniOT should be included in the main results. Since the proposed setting is similar to universal domain adaptation, it is essential to compare methods from both domains in the main results.\n\nMinor： \n\nMissing citation for the following important paper\n\n[1] Rastegar et al. Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery. NeurIPS 2023.\n\n[2] Gu et al. Class-relation Knowledge Distillation for Novel Class Discovery. ICCV2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)