Novel Class Discovery for Long-tailed Recognition
Authors: Chuyu Zhang
Ruijie Xu
Xuming He
Venue: Accepted by TMLR ()

Abstract:     While the novel class discovery has recently made great progress, existing methods typically focus on improving algorithms on class-balanced benchmarks. However, in real-world recognition tasks, the class distributions of their corresponding datasets are often imbalanced, which leads to serious performance degeneration of those methods. In this paper, we consider a more realistic setting for novel class discovery where the distributions of novel and known classes are long-tailed. One main challenge of this new problem is to discover imbalanced novel classes with the help of long-tailed known classes. To tackle this problem, we propose an adaptive self-labeling strategy based on an equiangular prototype representation of classes. Our method infers high-quality pseudo-labels for the novel classes by solving a relaxed optimal transport problem and effectively mitigates the class biases in learning the known and novel classes. We perform extensive experiments on CIFAR100, ImageNet100, Herbarium19 and large-scale iNaturalist18 datasets, and the results demonstrate the superiority of our method. Our code is available at \url{https://github.com/kleinzcy/NCDLR}.

Published in Transactions on Machine Learning Research (08/2023)
Novel Class Discovery for Long-tailed Recognition
Chuyu Zhang"
ShanghaiTech University, Shanghai, China Lingong Laboratory, Shanghai, China
Ruijie Xu*
mwr2202288shanghaaccchh......
ShanghaiTech University, Shanghai, China
Xuming He
heam herm@@shanghaitcchccc...
ThanghaiTech University, Shanghai, China
Shanghai Engineering Research Center of Intelligent Vision and Imaging, Shanghai, China
Reviewed on OpenReview: https: // opwrreviee..weeeeee net/ bet/forunt forum? id= ey50Dk 7h0DuK
Abstract
While the novel class discovery has recently made great progress, cxisting methods typically locus on improving algorithms on class-balanced benchmarks. However, in real-world recog- nition tasks, the class distributions their corresponding datasets are often imbalanced, which leads to serious performance degeneration of those methods. In this paper, we con- sider a more realistic setting for novel class discovery where the distributions of novel and known classes are long-tailed. One main challenge of this new problem to discover imbal- anced novel classes with the help of long-tailed known classes. To tackle this problem, we propose an adaptive self-labeling strategy based an equiangular prototype representation of classes. Our method infers high-quality pseudo-labels for the novel classes by solving relaxed optimal transport problem and effectively mitigates the class biases in learning the known and novel classes. We perform extensive experiments on CIFARI00, ImageNet 100, Herbarium19 and large-scale iNaturalit118 datasets, and the results demonstrate the supe- riority of our method. Our code is available https: / //ggtttttttt////////////t cww/kleinzcy/////eeeecce
1 Introduction
Novel Class Discovery (NCD) has attracted increasing attention in recent years (Han et al., 2021; Fini et al., 2021; Vaze et al., 2022), which aims to learn novel classes from unlabeled data with the help of known classes. Despite the existing methods have achieved significant progress, they typically assume the class distribution is balanced, focusing on improving performance on datasets with largely equal-sized classes. This setting, however, is less practical for realistic recognition tasks, where the class distríbutions are often long-tailed (Zhang al., 2021b). To address this limitation, we advocate more realistic NCD sctting in this work, in which both known and novel-class data are long-tailed. Such NCD problem setting is important, as it bridges the gap between the typical novel class discovery problem and the real-world applications, and particularly challenging, as it is often difficult to learn long-tailed known classos, let alone discovering imbalanced novel classes jointly.
Most existing NCD methods have difficulty in coping with the imbalanced class scenario due their re- strictive assumptions. In particular, the pairwise learning strategy (Han al., 2021; Zhong al., 2021b) often learns a poor representation for the tail classes due to insufficient positive pairs from tail classes. The more recent self-labeling methods (Asano et al., 2020; Fini et al., 2021) typically assume that the unknown class sizes are evenly distributed, resulting in misclassifying the majority class samples into the minority ones. An alternative strategy is to combine the typical novel class discovery method with the supervisod
'ooth authors contributed equally.

---

Published in Transactions on Machine Learning Research (08/2023)
long-tailed learning method (Zhang et al., 2021b; Menon et al., 2020; Kang et al., 2020; Zhang et al., 2021a). They usually require est istttining the novel-class distribution for post-processing and/or retraining the classi- fier. However, as our preliminary study shows (c.f. Tab.2), such two-stage method often leads to inferior performance due to the noisy estimation of the distribution.
To address the aforementioned limitations, we propose a novel adaptive self-labeling learning framework to tackle novel class discovery for long-tailed recognition. Our main idea is to generate high-quality pseudo- labels for unseen classes, which enables us to alleviate biases in model learning under severe class imbalance. To this end, we first develop a new formulation for pseudo- label generation process based relaxed optimal transport problem, which assigns pscudo labels to the novel-class data in an adaptive manner and partially reduces the class bias by implicitly rebalancing the classes. Moreover, leveraging our adaptive self-labeling strategy, we extend the equiangular prototype-bas based classifier learning (Yang et al., 2022b) to the imbalanced novel class clustering problem, which facilitates unbiased representation learning of known and novel classes in unified manner.
Specifically, we instantiate our framework with deep neural network consisting of an encoder with unsu- pervised pre-training and an equiangular prototype- based classifier head. Given set of known-class and novel-class imput data, we first encode those input data into unified embedding space and then introduce two losses for known and novel classes, respectively. For the known classes, we minimize the distance be- tween each data embedding and the equiangular prototype of its corresponding class, which help reduce the supervision bias caused by the imbalanced labels. For the novel classes without labels, we develop a new adaptive self-labeling loss, which formulates the class discovery as a relaxed Optimal Transport problem. To perform network learning, we design an efficient bi-level optimization algorithm that jointly optimizes efficient iterative learning pro-
the two losses of the known and novel classes. In particular, we introduce cedure that alternates between generating soft pseudo-labels for the novel-class data and performing class representation learning. In such a strategy, the learning bias of novel classes can be significantly reduced by our cquiangular prototype design and soft adaptive self-labeling mochanism. Lastly, we also propose a novel strategy for estimating the number of novel classes under the imbalance scenario, which makes our method applicable to real-world scenarios with unknown numbers of novel classes.
We conduct extensive experiments two constructed long-tailed datasets, CIFAR100 and Imagenct as well as two challenging natural long-tailed datasels, Herbarium19 and iNaturalist18. The results show that our method achieves superior performance on the novel classes, especially on the natural datasets. In summary, the main contribution of our work is four-folds:
We present a more realistic novel class discovery setting, where the class distributions of known and novel categories are long-tailed.
We introduce novel adaptive self-labeling learning framework that generates pseudo labels of novel class in an adaptive manner and extends the cquiangular prototype-bascl classifier to address the challenge in imbalanced novel-class clustering.
We formulate imbalanced novel class discovery as a relaxed optimal transport problem and develop a bi-level optimization strategy for efficient. class learning.
Extensive experiments on several benchmarks with different imbalance settings show that our method achieves the state-of-the-art performance on the imbalanced novel class discovery.
2
Related Work
2.1 Novel Class Discovery
Novel Class Discovery (NCD) aims to automatically learn novel classes in the open-world setting with the knowledge of known classes. A typical assumption of NCD that certain relation (c.g., semantic) exists between novel and known classes so that the knowledge from known classes enables better learning of novel ones. The deep learning-based NCD problem was introduced in (Han et al., 2019), and the subsequent works
2

---

Published in Transactions on Machine Learning Research (08/2023)
can be grouped into two main categories based on their learning objective for discovering novel classes. One calegory of methods (Han et al., 2021; Zhong et al., 2021a;b; Hsu et al., 20;b); assume neighboring data samples in representation space belong to the same semantic category with high probability. Based on this assumption, they learn a representation by minimizing the distances between adjacent data and maximizing non-adjacent ones, which is then used to group unlabeled data into novel classes. The other calegory of methods (Fini et al., 2021; Yang et al., 2022a; Gu et al., 2023) adopt a self-labeling strategy in novel class learning. Under an equal-size assumption on novel classes, they utilize an optimal transport-based self- labeling (Asano et al., 2020) strategy to assign cluster labels to novel-class samples, and then self-train their networks with the generated psendo labels.
Despite the promising progress achieved by those methods, they typically adopt an assumption that the novel classes are largely uniformly distributed. This is rather restrictive for real-world recognition tasks as the class distributions are often long-tailed in realistic scenarios. However, both categories of methods have difficulty in adapting to the setting with imbalanced classes. In particular, for the first category of methods, the representation of tail classes could be poor due to insuffieient samples in learning. For the methods, the head classes often misclassified as the tail classes under the restrictive uniform distribution constraint. In addition, both types of methods tend to produce biased classifier head induced by the long-tailed distribution setting. While recent work (Weng et al., 2021) proposes an unsupervised discovery strategy for long-tailed visual recognition, they mainly focus on the task of instance segmentation. By contrast, we systematically consider the problem of imbalanced novel class discovery in this work. Morcover, we propose an adaptive self-labeling learning framework based on the equiangular prototype representation and novel optimal transport formulation, which enables us to mitigate the long-tail bias during class discovery in principled manner.
2.2 Supervised Long-tailed Learning
There has been a large body of literature on supervised long-tailed learning (Zhang et al., 2021b), which aims to learn from labelled data with a long-tailed class distribution. To perform well on both head and tail classes, existing methods typically design variety of strategies that improve the learning for the tail classes. One stream of strategics is to resample the imbalanced classes (Han ct al., 2005) or to re-weight loss functions (Cao et al., 2019), followed by learning input representation and classifier head simultaneously Along this line, Logit- Adjusmmejtt (LA) (Menon et al., 2020) is a simple and effective method that has been widely used, which mitigates the classifier bias by adjusting the logit based on class frequency during or after the learning. The other stream decouples the learning of input representation and classifier head (Kang et al., 2020; Zhang et al., 2021a). In particular, classifier ret.raining (cRT) (Kang et al., 2020) first learns representation by salaance-balanced resampling and then only retrains the classifier head with re-balancing techniques. Recently, inspired by the neural collapse (Papyan et al., 2020) phenomenon, Yang et al. (2022b) propose to initialize the classifier head as the vertices of simplex equiangular tight frame (ETF), and fix its parameter during the learning, which helps to mitigate the classifier bias towards majority classes.
While those methods have achieved certain success in supervised long-tailed recognition, it is rather difficult to adopt them in novel class discovery as the class distribution of novel classes is unknown. Moreover, the problem of classifier bias becomes even more severe in discovering imbalanced novel data, where both the representation and classifier head are learned without ground-truth label information. To address this problem, we develop novel representation learning strategy, which generalizes the ETF design and equal- sized optimal transport formulation, for modeling both long-tailed known and novel classes in a unified self-labeling framework.
3 Problem Setup and Method Overview
We consider the problem of Novel Class Discovery (NCD) for real-world visual recognition lasks, where the distributions of known and novel classes are typically long-tailed. In particular, we aim to learn a set of known classes y from an annotated dataset D= = {(ff,)))) and to discover a set of novel classes y from an unlabeled dataset D {2 M labels in D=. For the NCD task, those two class sets have no overlap, ie., ye n 0, and we denote
Rer- Here z;. X are the input images and yf are the known class
3

---

Published in Transactions on Machine Learning Research (08/2023)
D
frequency
Ls
Y
class
S Sample
Yu
frequency
fo
(x(²)
Adaptiive Self-labeling
class
Y
Vi
p
S sample noval classss
S Sample known clesses
O plimize gd
Optimize
Wo
Head Nedium Class
Figure The overview of our framework. Our method first samples data batch including known and novel classes from the long-tailed dataset and then encodes them into an embedding space. We adopt the equiangular prototypes representing known and novel classes, and propose an adaptive self-labeling strategy to generate pseudo-labels for the novel classes. Our learning procedure alternates between pseudo-label generation, where optimize Lu w.r.t Y", and minimizing an MSE loss, where we optimize L.. L. w.r.t w. This process repeated until convergence (See S.c.4.3 for details).
their sizes as K and K" respectively. For long-tailed recognition, the numbers of training examples different classes are severely imbalanced. For simplicity of nolation, we assume that the known and novel classes are sorted by the cardinality of their training set in descending order. Specifically, we denote the number of training data for the known class i and the novel class j as N; and Mj. accordingly, and we have N1 > N2 .....> Ng., M1 > M2...> Mg.. To measure the class imbalance, we define an imbalance ratio for Mi, N. and R = MMn respectively, and R R > 1.
the known and novel classes, denoted as R Nx=
To tackle the NCD task for long-tailed rocognition, we propose a novel adaptive self-labeling framework capable of better learning both known and novel visual classes under severe class imbalance. Our framework consists of three key ingredients that help alleviate the imbalance learning of known and novel classes: 1) We introduce classifier design based equiangular prototypes for both known and novel classes, which mitigates class bias due to its fixed parametrization; 2) For the novel classes, we develop a new adaptive self- labeling loss, which formulates the class discovery as a relaxed Optimal Transport problem and can be jointly optimized with the supervised loss of the known classes; 3) We design an efficient itcrative learning algorithm that alternates between generating soft pseudo-labels for the novel-class data and performing representation learning. An overview of our framework is illustrated in Fig.1. In addition, we propose a simple method to estimate the number of novel class in the imbalance scenario. The details of our method will be introduced in Sec. 4.
4 Our Method
In this section, we first. introduce our model architecture and class representations in Sec. 4.1, followed by the design of learning losses in Sec. 4.2 and Sec. 4.3 for the known and novel classes, respectively. Then we present our iterative self-labeling learning algorithm in Sec. 4.4. Finally, we provide the details of our strategy for estimating the mumber novel classes under class imbalance scenario in Sec. 4.5.
4.1 Model Architecture and Class Representation
We tackle the long-tailed NCD problem by learning a deep network classifier for both known and novel classes. To this end, we adopt generic design for the image classifier, consisting an image encoder and classification head for known and novel classes. Given an input æ, our encoder network, denoted as fa. computes a feature embedding fo(z) Dx1, which then fed into the classification head for class prediction. Here normalize the feature embedding such that |z|| 1. While any image encoder
4

---

Published in Transactions on Machine Learning Research (08/2023)
be potentially used in our framework, we adopt unsupervised pre-trained ViT model (Dosovitskly et al., 2021) as our initial encoder in this work, which can extract discriminative representation robust to the imbalanced learning (Liu et al., 2022). We also share the encoder of known and novel classes to encourage knowledge transfer between two class sets during model learning.
For the classification head, we consider a prototype-ban based class representation where each class repre- sented by a unit vector P: Dx1, More specifically, we denote the class prototypes of the known classes as P = (pt.... ... p and those of the novel classes P" .'".... - The entire class space is then represented as P = [P*, P'] DDx To perform classification, given feature embedding z, take the class of its nearest neighbor in the class prototypes P as follows,
arg min Pill,,
where P, is the i-th column of P, and c* is the predicted class of the input x.
In the imbalanced class scenario, bias toward the majority classes, especially for the novel classes, where the classifier and representation are learned without label information. While many calibration strategies have been developed for the long-tailes problems in supervised learning (c.f. Sec. 2), they not applicable to the imbalanced novel class discovery task as the label distribution of novel classes is unknown.
challenging to learn the prototypes from the data as they tend
To cope with imbalanced data in our NCD problem, our first design is to adopt. a fixed parametrization for the class prototypes. Specifically, we generalize the strategy proposed by Yang et al. (2022b) for imbalanced supervised learning, and utilize the vertices of a simplex oquiangular tight frame (ETF) as the prototype of both known and novel classes. By combining this prototypical representation with our adaptive self-labeling framework (Sec.4.3), our method is able to reduce the bias in learning for all the classes in unified manner. More concretely, we generate the prototype set P as follows:
K K K-1 MIIK - K [kxK),
1
P
(2)
where M is an arbitrary orthonormal matrix, I is a K K identity matrix, 1 denotes the all ones matrix and K= = K K" is the total number of class prototypes. The generated prototypes have unit l2 norm and same pair-wise angle, which treats all the classes equally and help debias the classifier learning.
4.2 Loss for Known Classes
For the known classes, we simply use the Mcan Square Error (MSE) loss, which minimizes the le distance between the feature embedding of an input x and the class prototype of its groundtruth label yf. Specifically, we adopt the average MSE loss on the subset of known classes D as follows,
N
N Σ
(0) = N
Σ 2
P
1|2 = N
22 PS + C,
(3)
where z", P"; is the feature embeddings and class prototypes, respectively, of is the ground-truth label of input z;, and C' is a constant. It is worth noting that our design copes with the class imbalance in the known classes by adopting the equiangular prototype and initializing the encoder based on an unsupervised pre-trained network. This strategy simple and effective (as shown our experimental study)", and can be easily extended to discovering novel classes.
4.3 Adaptive Self-Labeling Loss for Novel Classes
We now present the loss function for discovering the novel classes in Dd, Given an input xY" we introduce a pseudo-label variable y!" to indicate its (unknown) membership to the K classes and define a clustering loss
While possible integrate additional label balancing techniques, it is beyond the scope this work.
5

---

Published in Transactions on Machine Learning Research (08/2023)
based the Euclidean distance between its feature embedding z and the class prototypes Pa as follows,
M Σ
M Σ
1 M
())
M 2
2l2
2z
Pup
+ C,
(4)
where C' is a constant as the feature and prototype vectors are normalized. Our goal is to jointly infer an optimal membership assignment and learn a discriminative representation that better discovers novel classes.
Regularized Optimal Transport Formulation: Directly optimizing Cu is difficult, and a naive alter- nating optimization strategy often suffers from poor local minima (Caron et al., 2018), especially under the scenario of long-tailed class distribution. To tackle this, we reformulate the loss in Eq. into a regularized Optimal Transport (OT) problem, which enables to design an adaptive self-abbeling learning strategy that iteratively generates high-quality pseudo-labels (or class memberships) and optimizes the feature represen- tation jointly with the known classcs. To this end, we introduce two relaxation techniques to convert the Eq. to an OT problem as detailed below.
First, we consider a soft label y E _!*K" to encode the class membership of the datum at, where y 1. Ignoring the constants in Lu, we can re-write the loss function in vector form as follows,
1 M
min Ly = min Yu
Σ M
T P"), s.t. by
(5)
Yo
1=1
min Y", - 2'"") Fr s.t. Y"Ig= =
(6)
where (,)F represents the Frobenius product, Y" Y 1 lyf. yil E R Mx is the pseudo-label matrix, Z .z..... .z%r E DxM is the feature embedding matrix and μ= == Mls. This soft label formulation is more robust to the noisy learning (Lukasik et al., 2020) and hence will facilitate the model learning with inferred pseudo-labels.
Second, inspired by (Asano et al., 2020), we introduce a constraint on the sizes of clusters to prevent degenerale solutions. Formally, we denote the cluster size distribution as a probability vector v E RK" and define the pseudo-label matrix constraint as Y 1 = V. Previous methods typically take an equal- size assumption (Asano et al., 2020; Fini et al., 2021), where v is a uniform distribution. While such an assumption can partially alleviate the class bias by implicitly rebalancing the classes, it is often too restrictive for an unknown long-tailed class distribution. Tn particular, our preliminary empirical results show that it often forces the majority classes to be mis-clustered into minority classes, leading to noisy pseudo-label estimation. To remedy this, propose second relaxation mechanism on the above constraint. Specifically, we introduce an auxiliary variable w E RK" which is dynamically inferred during learning and encodes a proper constraint on the clustcr-size distribution. More specifically, we formulate the loss into a rogularized OT problem as follows:
min Lu (Y", 0) min (Y", -Z KKW,W)),
(7)
s.t. Y' E M R Y² μ, Y 1M w])
(8)
where y is the balance factor to adjust the strength of KL constraint. When y = inf, the KL constraint falls back to equality constraints. Intuitively, our relaxed optimal transport formulation allows us to generate better pseudo labels adaptively, which alleviates the learning bias of head classes by proper label smoothing.
Pseudo Label Generation: Based the regularized OT formulation of the clustering loss Cu, now present the pscudo label generation process when the encoder network fo is given. The generated pscudo labels will be used as the supervision for the novel classes, which is combined with the loss of known classes for updating the encoder network. We will defer the overall training strategy to Sec. 4.4 and first describe the pseudo label generation algorithm below.
Eq. (7) and (8) minimize Lu w.rtt (Y", w) with fixed cost matrix -Z TP (as 0 is given), which can be solved by convex optimization techniques (Dvurechensky et al., 2018; Luo et al., 2023). However, they
6

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 1: Sinkhorn-Knopp Based Pseudo Labeling Algorithm
Input: Matrix Y, marginal distribution N, hyper-parsmeters T
Output: Y
Function Pseuso-LabelinngXX,, H4, w):):
Y e/A( Y Y Y
a. 1.1
for C 1, do do
as p./(Y3), w./(YTa)
end
Y f- diag(a))diag(3)
return Y: End Function
are lypically computationally expensive in our scenario. Instead, we leverage the efficient Sinkhorn-Knopp algorithm (Cuturi, 2013) and propose bi-level optimization algorithm to solve the problem approximately. Our approximate strategy consists of three main components as detailed below.
A. Alternating Optimization with Gradient Truncation: We adopt an alternating optimization strategy with truncated back-propagation (Shaban et. al., 2019) to minimize the loss L. w) Specifically, we start from fixed w (initialized by v) and first minimize (Y",,, w) w.r.t Y". As the KT constraint term remains constant, the task turns into standard optimal transport problem, which can be efficiently solved by the Sinohorn-Knopp Algorithm (Cuturi, 2013) as shown in Alg. 1. We truncate the iteration with fixed T, which allows us to express the generated Y as a differentiable function of w, denoted as Y"(w). We then optimize L. (Y),,), w) w.r.t w with simple gradient descent. The alternating optimization of Y" and w lakes several iterations to produce high-quality pseudo labels for the novel classes.
B. Parametric Cluster Size Constraint: Instead of representing the cluster size constraint as a real-valued vector, we adopt parametric function form in this work, which significantly reduces the search space of the optimization and typically leads to more stable optimization with better empirical results. Specifically, we parametrize w as a function of parameter т in the follow form:
(9) where T can be viewed as the imbalance factor. As our class sizes decrease in our setting, we replace with function form of 1 + exp(() in practice, which is always larger than 1. Then we normalize w; by EK-1 wt to make it a proper probability vector.
W: T i 0, 1, K"
C. Mini-Batch Buffer: We typically generate pseudo labels in a mini-batch mode (c.f. Sec. 4.4), which however results in unstable optimization of the OT problem. This is mainly caused by poor estimation of the cost matrix due to insufficient data, especially in the long-tailed setting. To address this, we build a mini-batch buffer to store 2048 history predictions (i.c., Z P") and replay the buffer to augment the batch-wise optimal transport computation. Empirically, we found that this mini-batch buffer significantly improves the performance of the novel classes.
4.4 Joint Model Learning
Given the loss functions L. and Lu, now develop an iterative learning procedure for the entire model. As our classifier prototypes are fixed, our joint model learning focuses on the representation learning, param- eterized by the encoder network fo. Specifically, given the datasets of known and novel classes, we sample a mini-batch of known and novel classes data at each iteration, and perform the following two steps: 1) For novel-class data, we generate their pseudo labels by optimizing the regularized OT-based loss, as shown in Sec. 4.3; 2) Given the inferred pseudo labels for the novel-class data and the ground-truth labels for the known classes, we perform gradient descent on a combined loss function as follows,
((() = (+) + (L),
(10)
Note that we simplify Ca w) Lu (Y", w) as not optimize in pseudo label generation process.
7

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 2: Adaptive Self-labeling Algorithm
Input: D", 2", encoder for equiangular prototype P E (Xx(
initial mini-batch Buffer, w, f = 15x1, parammerrees B, L
for e 1,2... Epoch do
for C 1,2..... Step do
{(1.. vf h=
Sample(D") (=))E + Sample(D")
x¹ 1,(="), fo(=")
//MSE loss for labeled data
L. R E
P
y = pm E gI
Y : y Rel) "'
for IE 1, 2..,, L do
= P-ePeeedooaar LaciooLLLablinnna Laarddo-LabliinnnYP> Y": w)
w ~ arg min f. YY (w),
end
//MSE loss for unlabeled data
= Y / minimize C. aL w.r.
end
end
where Cg and Cu are the losses for the known and novel classes respectively, and is the weight to balance the learning of known and novel classes. The above learning process minimizes the overall loss function over the encoder parameters and pseudo labels in an alternative manner until converge. An overview of our entire learning algorithm is illustrated in Alg.2.
4.5 Estimating the Number of Novel Classes
In the sccnarios with unknown number of novel classes, we introduce simple and effective strategy for estimating the cardinality of the imbalanced novel class set, K To achieve this, our method utilizes the data clustering of the known classes to score and select the optimal K" (Vaze et al., 2022). Specifically, given a candidate K", we first perform hicrarchical clustering algorithm to cluster both known and novel classes (Ds, D") into K" clusters. Next, we employ the Hungarian algorithm to find the optimal mapping between the set of cluster indices and known class labels of data, and evaluate the clustering accuracy of the known classes. To determine the best value for K", we repeat this process for a set of candidate sizes and choose the setting with the highest accuracy of the known classes.
However, in imbalanced datasets, the average performance of known classes tends to be biased towards larger classes, which results in underestination of the number of unknown classes. To address this, consider the average accuracy over classes, which is less influenced by class imbalance, and design a mixed metric for selecting the optimal value of K" Concretely, the mixed metric is defined as weighted sum of the overall accuracy of the known classos, denoted by Acc,, and the average class accuracy of known classes, denoted by Acce, as follows:
Acc = BAcc, + (1 - ACccc
(11)
where 3 is weighting parameter and is set as 0.5 empirically. We employ the mixed metric to perform binary search for the optimal value of K", The detail algorithm is shown in Appendix A.
5
Experiments
5.1
Experimental Setup
Datasets We evaluate the performance of our method on four datasets, including long-tailed variants of two image classification datasets, CIFAR100 (Krizhevsky et al., 2009) and ImageNet100 (Deng ct al., 2009), and two real-world mediun/large-scalee long-tailed image classification datasets, Herbarium19 (Tan et al., 2019) and Naturalistt18 (Van Horn et al., 2018). For the NNaturalist18 dataset, we subsample 1000 and
8

---

Published in Transactions on Machine Learning Research (08/2023)
Table The details of all the datasets for evaluation. The imbalance ratio of the known classes Rt 50 for CIFAR100-50-50 and ImageNet100-50-50, and R" denotes the imbalance ratio of the novel classes.
Datasets
CIFAR100-50-50
ImageNet 100-50050
Herbarium19 UnKnown
iNaturalist I8
18
Rd
50
100
50
100
UnKnown
UnKnown
Known Classes
50
50
50
342
500
1000
Known Data
6.4k
6.4k
16.3k
16.5k
17.8k
26.3k
52.72
Novel Classes
50
50
50
50
342
500
1000
Novel Data
6.4k
5.5k
16.2k
14.0k
16.5k
26.4k
49.9k
Test set 10k
10k
5.0k
5.0k
2.8k
3.0k
6.0k
2000 classes from 1N to create (Naturalist18-1K and iNaturalist18-22 respectively, which allows us to test our algorithm on relatively large-scale benchmark with a practical computation budget, For each dataset, we randomly divide all classes into 50% known classes and 50% novel classes. For CIFAR100 and ImageNNe1000, we create "long-tailed" variants of the known and novel classes by downsampling data examples per class following the exponential profile in (Cui et al., 2019) with imbalance ratio R N: order to explore the performance of novel class discovery under different settings, we set the imbalance ratio of known classes R as 50 and that of novel classes R" as 50 and 100, which aim to mimic typical settings in long-tailed image recognition tasks. For testing, we report the NCD performance on the official validation sets of each dataset, except for CIFAR100,, where we use its official test set. We note that those test sets have uniform class distributions, with the exception of the Herbarium19 dataset, and encompass both known and novel classes. The details of our datasets are shown in Tab. 1.
In
Nx
Metric To evaluate the performance of our model on each dataset, we calculate the average accuracy classes on the corresponding test set. We measure the clustering accuracy by comparing the hidden ground-truth labels y: with the model predictions úi using the following strategy:
ClusterAcc
max permEP
N (v: p:rm(p:))))))..
(12)
where P represents the set of all possible permutations of test data. To compute this metric, we use the Hungarian algorithm (Kuhn, 1955) to find the optimal matching between the ground-truth and prediction. We note that we perform the Hungarian matching for the data from all categories, and then measure the classification accuracy both the known and novel subsets. We also sort the categories according to their sizes in descending order and partition them into 'Head', 'Medium', 'Tail' subgroups with ratio of 3: 4: 3 (in number of classes) for all datasets.
Implementation Details For a fair comparison, all the methods in evaluation use a ViT-B-16 backbone network as the image encoder, which is pre-trained with DINO (Caron et al., 2021) in an unsupervised manner. For our method, we train 50 epochs on CIFAR100 and ImageNet100, 70 epochs on Herbarium and NNturralis18. We use AdamW with momentum as the optimizer with linear warm-up and cosine annealing (Irbase = 1c-3, Immin Ic-4, and weight decay 5c-4). We set addition, analyze the sensitivity of y in Appendix C. For all the experiments, we set the batch size to 128 and the iteration step L to 10. For the Sinkhorn -noop algorithm, we adopt all the hyperparameters from (Caron et al., 2020), e.g. Niter = 3 and 0.05. Implementation details of other methods can be found in Appendix B.
1, and select 7 500 by validation. In
5.2 Comparison with NCD Baselines
We first report a comparison of our method with two major NCD baselines (Autonovel and UNO) and their variants on the benchmarks, CIFAR100 and ImageNet100 in Tab.2. In addition to the original NCD algorithms, we also apply two long-tailed learning techniques to the baselines, which include logits adjustment (LA), which adjusts the logits according to the estimated class distribution in the inference, and class reweighting (cRT), which retrains the classifier with a
dataset. For CIFAR100, when R
9

---

Published in Transactions on Machine Learning Research (08/2023)
Table 2: Long-tailed novel class discovery performance on CLFAR-100, ImageNet100. We report average class accuracy test datasets. R², R²" are the imbalance factors of known and novel classes respectively. "4LA" means post- processing with logiss-adjustment (Menon et al., 2020), "+cRT" means classifier retraining (Kang et al., 2020).
CIFAR MMPAR100-50-61
ImageNett Netne-Net
n²
50, I?"
50
R²
50, π
100
I
50, R?
50
Re
50. I
100
Method
All
Novel
Known
All
Novel
Known
All
Novel
Known
All
Novel
Known
Autonovel
44.42
22.28
66.56
44.04
22.12
65.96
67.50
47.96
87.04
63.86
40.88
86,81
Autonovel LA
45.32
20.76
69.88
42.20
18.00
66.40
67.74
46.72
88.76
61.08
39.32
88.84
Autel + cRTT
47.20
26.48
67.92
41.94
22.62
61.26
67.76
49.88
85.64
63.90
42.40
85.40
UNO
50.82
34.10
67.54
49.50
31.24
67.76
65.30
43.08
87.52
62.52
37.84
87.20
UNO LA
52.36
33.82
70.90
51.62
31.04
72.20
65.92
43.12
88.72
63.28
37.72
88.84
UNO cRT
54.26
40.42
68.10
47.62
31.02
64.22
68.38
50.80
85,96
63.10
30.96
86.24
Ours
53.75
40.60
66.90
51.90
36.80
67.00
73.94
61.48
86.40
69.38
51.96
86.80
Table 3: Long-tailed novel class discovery performance on medium//arge-scan Herbarium19 and iNaturalist18. Other
details are the same as Tab. 2.
Herbarium Novel
iNaturalist18-1
iNaturalist18-2K
Method
All
Known
All
Novel
Known
All
Novel
Known
Autonovel
34.58
9.96
59.30
42.33
11.67
73.00
39.08
8.57
69.60
Autonovel + LA
32.51
8.60
56.56
12.40
11.27
73.53
44.67
14.33
75.00
AutoNovel cRT
45.05
24.46
64.49
44.20
16.13
72.27
37.95
9.27
66.63
UNO
47.47
34.50
60.58
52.93
31.60
74.27
45.60
19.97
71:23
UNO + LA
46.76
27.96
65.69
46.63
24.33
74.60
46.63
20.33
72.93
UNO cRT
46.17
33.13
59.95
51.73
32.60
70.87
16.17
21.90
68.03
Ours
49.21
36.93
61.63
58.87
45.47
72.27
49.57
34.13
65.00
R = 50, our method achieves competitive results compared to the two-stage training methods. As the data become more imbalanced, i.c. R = 100, our method achieves 5. 787 improvement on the novel class accuracy. Here we note that our method does not exhibit significant advantage due to the limited quality of the representation computed from the Low-resolution images. For ImageNet100, our method achieves significant. improvements in two different R" settings, surpassing the previous SOTA method by 10.68% and 9.56% respectively.
Furthermore, in Tab. 3, we show the results on two medium//arge scale datasets. Specifically, on the challenging fino-grained imbalanced Herbarium19 dataset, which contains 341 known classes, our method achioves 2.43% improvement on the novel class accuracy compared to UNO. We also report the per-sample average class accuracy Appendix E, on which we achieve ~ 10% improvement. On the more challenging INaturalis118 datasets, we observe a significant improvement (>10%) in the performance of novel classes compared to the Herbarium19 dataset. In summary, our notable performance gains in multiple experimental setlings demonstrate that our method is effective for the challenging long-tailed NCD task.
5.3 NCD with Unknown Number of Novel Categories
To evaluate the effectiveness of our estimation method, we establish a bascline that uses the average accuracy as the indicator to search for the optimal K" value by hierarchical clustering. The details of the baseline algorithm are shown in Appendix A. As shown Tab.4, our proposed method significantly outperforms the baseline on three datasets and various scenarios, indicating the superiority of our proposed mixed metrie for estimating the number of novel classes in imbalanced scenarios.
Furthermore, we conduct experiments on three datasets with estimated K", As Tab.5 shown, our method achieves sizeable improvements on the novel and overall class accuracy, except in the case of CIFAR when Rd 50, which we achieve comparable results. On the CIFAR dataset, the existing methods outperform our method on the known classes, especially when the LA or cRT techniques are intograted. As a result, our method demonstrates only slight improvements or comparable results w.r.t the existing methods on the overall accuracy. However, it is worth noting that when our method is combined with the LA technique, we
10

---

Published in Transactions on Machine Learning Research (08/2023)
Table 4: Estimation the number of novel categories. Re 50 for CIFAR100 and ImageNet100 datasets.
CIFAR100-50-50
ImageNet100-50-300
Herbbarim
Method
R" = 50
R² = 100
R" 50
I²" = 100
Unknown
GT
50
50
50
50
341
Baseline
0
10
14
2
Ours
20
29
153
Table 5: Experiments on three datasets when K unknown.
CIFAR100-50--0
Imagee 00-50-50
Herbarium
R'
50, qh= 50
50, R
100
Rt
50, R = 50
R'
50, R
100
Uuknown Novel
Method
All
Novel
Known
All
Novel
Known
All
Novel
Known
All
Novel
Known
All
Known
AusoNowel
41.25
16.08
D6:42
43.74
17.64
63.84
63.992
06.68
44,96
58.40
37.84
15.64
60.16
AusaNovel+LA
41.82
16.18
67:06
13.82
18.08
09.56
61.74
32 08
90.80
62.26
35.52
89.00
12.15
19.06
65.37
AetoNooel+eeR
45.81
19.50
43.44
68.70
62.88
32.44
16.68
58:20
40:91
18.86
63.16
UNO
47.67
28.92
66.12
16.96
25.92
07:20
67. 96
18. 18
87.44
61.16
41.24
57.08
10.83
23.55
58.24
UNO+LA
49.51
28.18
70.81
25.70
70.31
07.91
17.
41.48
23.02
62.58
UNO
49.35
30.82
67.88
45.49
26.68
64.30
70.94
55.32
86.56
62.76
38.72
56.80
40.67
22.84
58.63
Ours
49.03
32.66
65,40
33.06
64.72
74.06
61.
68.94
50.84
87.04
29.01
50.34
Ours: LLA
49.64
32.46
06.82
49.87
33.12
06.02
74.38
611.48
87.28
71.08
54.92
87.24
45.74
29.55
61.90
achieve more favorable outcomes. On the ImageNet110 and Herbarium19 datasets, our method surpasses the existing methods by a large margin. For example, on ImageNet100 dataset when R" = 100, ours outperforms the best baseline (AutoNovel) by 3.92% in overall accuracy and 5.88% in novel-class accuracy. Moreover, when our method is equipped with LA, the performance is further improved, with an increase of 4.4% in overall accuracy and 9.96% novel-class accuracy.
It is important to note that the effect of estimated K" differs based on its relationship with the ground truth value. When the estimated K" is lower than the ground truth value, such as CIFAR100 and Herbarium 19, the performance deteriorates compared to using the true K This occurs because the estimated lower KM leads to the mixing of some classes, especially medium and tail classes, resulting in degraded performance. When the estimated Kn is highor than the ground truth value, such as ImageNet100, using the estimated K" leads to better results for UNO and comparable results for Ours. For UNO which assume equally sized distribution, larger K" tend to assign the head classes to additional empty classes, reducing the noise caused by the mixing of head classes with medium and tail classes, and thereby improving the accuracy of medium and tail classes. By contrast, our method dynamically adjusts the allocation ratio for novel classes, effectively suppressing the assignment of head classes to empty classes, which allows us to achieve comparable results. A more detailed analysis of this phenomenon can be found in Appendix F.
5.4 Ablation Study
Component Analysis: Table 6 presents ablation results of our method's components on ImageNet100.
The impact of each core component is analyzed individually, including equiangular prototype representation, adaptive self-labeling, and the mini-batch buffer. As shown in the first and second rows of the Tab.6, the addition of mini-batch buffer results in a 2% improvement compared to the baseline on novel class accuracy. By comparing the second and third rows, notable improvements emerge in subgroup class accuracy, especially for head classes. For instance, with R 100, our method achieves 16.67% improvement on the head, 7.8% improvement on medium, 7.33% improvement on tail classes. This demonstrates that the equiangular prototype representation helps alleviate the imbalance learning of novel classes. Comparing the third and last row, show that adopting adaptive self-labeling greatly improves the tail and head class accuracy, For example, our method achieves 11.47% improvement on head, and 7.34% improvement on tail classes for Rt = 50. Such results indicale that the uniform constraint on the distribution of clusters is not suitable for imbalance clustering, as it tends to misclassify head classes samples as tail classes. And the reason for the slightly worse performance for modium classes is that the uniform distribution constraint better approximates the true medium class distribution. However, this constraint harms the performance of the head and tail, resulting in nearly 4% decrease in the novel class accuracy. In conclusion, the overall results validate the
11

---

Published in Transactions on Machine Learning Research (08/2023)
Table Ablation on ImageNet100. "EP" and " "ASL" stands for equiangular prototype and adaptive self- labeling..
It 50,
50
It"
50, I2"
100
Method
Novel
Head
Medium
Tail
Novel
Head
Medium
Tail
Baseline
43.08
44.93
49.20
33.07
37.84
49.73
43.10
18.93
Buffer
46.36
46.93
51.90
38.40
39.88
52.00
46.00
19.60
Buffer + EP
57.40
66.00
59.70
45.73
47.24
68.67
49.20
23.20
Buffer EP + ASL
61.48
77.47
55.80
53.07
51.96
77.47
54.20
23.47
Table The effects of different combinations of loss function and classifier. 50, R = 50. cls is an abbreviation for classifier.
Results on ImageNet for
Method
Novel
Head
Medium
Tail
Learnable cls + loss EP + EP + MSE loss
46.36
46.93
51.90
38.40
52.40
53.47
66.10
33.07
57.40
59.70
45.73
Figure 2: t-SNE visualization of novel instances in ImageNet100 for features after the last transform block. The left is the feature space using learnable classifier, and the right is the feature space using equiangular prototype.
effectiveness of our proposed components, and especially the equiangular prototype and adaptive self-labeling produce notable improvements.
Effect of Equiangular Prototype: In Sec.4, we utilize the MSE loss to minimize the distance between sample and prototype. Here we delve into the impacts of various loss function and classifier combinations. As Tab.7 shown, a learnable classifier with CE loss performs worse than EP cls + CE loss. This discrepancy can be attributed to the tendency the learned prototypes exhibiting bias towards head classes, ultimately leading less discriminative representations. What's more, EP cls with MSE loss improve EP cls with CE loss by large margin, especially for head and tail classes. In the early learning stage, the representation is relatively poor, and massive head class samples are allocated to tail classes because of uniform distribution constraints. While the gradient of CE loss is larger than MSE loss, resulting in the EP + CE loss overfitting to the noise labudo-label quickly.
Moreover, to better understand the effect the equiangular prototype for novel class clustering, we visualize the feature space by t-SNE(Van der Maaten & Hinton, 2008) on the test set. As Fig.2 shown, the features learned by the equiangular prototype are more tightly grouped with more evenly distributed interclass distances. However, the learnable classifier setting results in several classes being entangled together.
Adaptive Self-labeling: In this part, we validate the effectiveness of our design on We set as the uniform and ground-truth distribution, and conduct corresponding experiments. Interestingly, as shown in the first two rows of Tab.8, setting w uniform prior achieves better performance, especially on medium and (ail class. We speculate that the uniform constraint smooths the pseudo label of head class samples, mitigating the bias learning of head classes, thus improving the results of medium and tail classes. addition, we also try two ways learn w with a prior constraint of uniform distribution. One way parameterize w as vallev vector, and the other is to use paranetric form as function of т. As shown in the last two rows of Tab.8, find that optimizing k-dimensional is unstable and prone to
12

---

Published in Transactions on Machine Learning Research (08/2023)
Head, Medium, Tail Acc
The adaptive imbalanced factor
Table 8: The impact of different param- eteric strategy of w. The first two rows of w are fixed, and the last two rows rep- resent the two parameterization ways of
80
1.10
1,08
[4pp))
w. Results on ImageNet100, for R
1.06
40-
50, Rt" 50.
head
KM Day n
Method
Novel
Head
Medium
Tall
medium
Maraa Wwl
tail
Uniform
57.40
66.00
59.70
45.73
1.00
w True Pricr
42:40
64.53
46.90
14.27
2000
4000
2000
4000
6000
55.64
69.73
59.80
31.87
w(Y)
61.48
77.47
65.80
53.07
Figure 3: Analysis of w during training
assign overly large cluster sizes for some clusters. Therefore, optimizing parametrized by function of parameter 7 (As shown in Eq.9) seems to be more effective.
To provide more analysis on w, we visualize the learned imbalance factor and the head/ peoud//edium//// t/aill class accuracy during the training process the model. As Fig.3 shows, in the early stage, the head class accuracy first increases quickly, indicating that the model bias to the head classes and their representations are better learned. Corrospondingly, the learned imbalance factor increases quickly, thus assigning more samples the head classes. Subsequently, the medium and tail class accuracy increase; meanwhile, the imbalance factor decreases, resulting in the pseudo label process biasing to the medium and tail classes. Although the imbalance factor changes only slightly during the learning, it improves novel class accuracy by a large margin compared to the fixed uniform prior (the first row and last row in Tab.8).
6 Conclusion
In this paper, we present real-world novel class discovery setting for visual recognition, where known and novel classes have long-tailed distributions. To mitigate the impact of imbalance on learning all classes, we have proposed a unified adaptive self-labeling framework, which introduces an equiangular prototype-based class representation and an iterative pseudo-label generation strategy for visual class learning. In particular, we formulate the pseudo-label generation step as a relaxed optimal transport problem and develop a bi-level optimization algorithm to efficiently solve the problem. Morcover, we propose an effective method to estimate the number of novel classes in the imbalanced scenario. Finally, we validate our method with extensive experiments two small-scale long- tailed datasets, CIFAR100 and ImageNet100, and two medium/larg scale real-world datasets, Herbarium19 and 1N Our framework achieves competitive or superior performances, demonstrating its efficacy.
Acknowledgement
This work was supported by Shanghai Science and Technology Program 21010502700, Shanghai Frontiers Sci- ence Center cumman-centerdd Artificial Intelligence and the MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech) University).
References
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Sclf-labelling via simultancous clustcring and representation learning. In International Conference on Learning Representations (ICLR), 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--waare margin loss. In Advances in Newral Information Processing Systems, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervisod learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp.
132 149, 2018.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912 9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 9650-9660, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effec- Proceedings of the IEER/CVF conference on computer vision and pattern
tive number of samples. recognition, pp. 9268 9277, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-244-400 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Pavel Dvurechensky, Alexander Gasnikov, and Alexey Kroshnin. Computational optimal transport: Com- plexity by accelerated gradient descent is better than by sinkhorn's 's algorithm. In International conference on machine learning, pp. 1367-1376. PMTR, 2018.
Farico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9284 9292, 2021.
Peiyan Gu, Chuyu Zhang, Ruijie Xu, and Xuming He. Class-relation knowledge distillation for novel class discovery. arXiv preprints, pp. arXiv-200 2307, 2023.
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in im- balanced data sets learning. In International conference on intelligent computing, pp. 878-887 887. Springer, 2005.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categorics via deep transfer clustering. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1401-8409,40 2019.
Kai Han, Sylvestre- Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Thansactions Pattern Analysis and Machine Intelligence, 2021.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVI conference on compuler vision and paltern recognition, pp- 9729-9733 2020.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt. Kira. Learning to cluster in order to transfer across domains and tasks. In International Conference on Learning Representations, 2018a.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In International Conference on Learning Representations, 2018b.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Confer- ence on Learning Representations (TCI.R), 2020.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83-97, 97, 1955.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more ro- bust to dataset imbalance. In International Conference on Learning Represenlations, 2022. URL attps://openrest http://ppwweeeeeeeeeeeoooooaaaaa set/fformm?d==A22bospra.a
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 6448-6458. PMLR, 2020.
Yiling Luo, Yiling Xie, and Xiaoming Huo. Improved rate of first order algorithms for entropic optimal transport. In International Conference Artificial Intelligence and Statistics, pp. 2723-2750. PMLR, 2023.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. orXiv preprint arXiv:2007.07914 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 246663, 2020.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1723-1732. PMLR, 2019.
Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. arXiv preprint ar Xix: 05337, 2019.
Laurens Van der Maaten and Geoffrey Hinton. research, 9(1),, 2008.
Visualizing data using t-sne. Journal of machine learning
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. $769-8778, 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492-7501, 2022.
Zhenzhen Weng, Mchmet Giray Ogut, Shai Limonchik, and Screna Yeung. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In Proceedings of the IERE/CVP Conference Computer Vision and Pattern Recognition, pp. 2603 2612, 2021.
Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 14268 14277, 2022a.
Yibo Yang, Liang Xic, Shixiang Chen, Xiangtai Li, Zhouchen Lin, and Dachong Tao. Do we really need a learnable classifier at the end of deep neural network? arXiv preprint arXiv:220s.0900 2022b.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution aligument: A unified framework for tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2361-2370, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. Xiv preprint orXiv:2110.04596, 2021b.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood con- trastive learning for novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10867-10875, 2021a.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in open world. In Proceedings of the IEEE/ IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 9462 9470, 2021b.
16

---

Published in Transactions on Machine Learning Research (08/2023)
A The algorithm of estimating the number of novel categories
We introduce a novel approach for estimating the number of unknown classes imbalanced scenarios (Section 4.5). Our method leverages the clustering performance of the known classes dataset D as a means of scarching for the optimal value of k The detailed algorithm for this estimation process is outlined in Algorithm 3. In our method, we evaluate the performance using the mixed metric (Eqn. 11). In contrast, the Baseline' method utilizes the average accuracy of each sample as its evaluation metric, which tends to bias to majority classes.
Algorithm 3: The algorithm of estimating the number of novel categories.
Input: D*, D", K", maximum Maa:: evaluation metric eval, hierarchical clustering algorithm HC
Output: K"
Right Kedr Koo +- Kmans /mase/ /2,0
AcChigh = =CVal DD", "D",Khigh high K²), D*) Accoed = eeal(HCCDD* DD, KC med + Ks), D*) Accyou = HHC(D, Du, K + K*),
while higgh Now do
if Acchigh > Acctow then
igw t- Ku
med high high + Kow 1//2
Acciow = Accmed
Accmed (HC DD, Kand K*), Dt)
end
else
Ky
high + k med
Ku med + (Kiigh + +Ky 1/2
AtChigh = Arcmee
Accmnd (D),, DD, K2 med + K2), Dt)
end
end
B Implementation details
As there are currently no existing baselines for novel class discovery in an imbalanced setting, we have implemented two typical NCD methods, AutoNovel (Han et al., 2021) and UNO (Fini et al., 2021). To handle imbalancod lcarning, we have combined these NCD methods with two common approaches for long- tailed problems: logit-adjusttent (Menon et al., 2020) and decoupling the learning of representation and classifier head (Kang al., 2020).
We have used the same unsupervised pretrained model and only modified the training setup of AutoNovel and UNO. Specifically, we trained AutoNovel for 200 epochs until convergence on all datasets, and the training strategy for UNO is identical to ours, as described in the main paper.
For our implementation itn of logit-adjustment, we have set π = following (Menon et al., 2020). If the estimated number pseudo-labels for a novel class is O, we do not make any corrections to its logits. When adding cRT (Kang al., 2020), we first estimate the class distribution and use the same number of epochs as in the first. stage.
C Sensitive analysis of 7
The optimal value for the hyperparameter ? is selected by partitioning a subset of known classes as the validation set. Additionally, to investigate the sensitivity of the hyperparameter gamma, we have presented the change novel class accuracy from gamma values of 100 to infinity in Figure 4. Our findings indicate that when gamma is larger than 300, our adaptive sele-lag method outperforms the naive baseline. However, the value selected using our validation set is not the optimal one.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Novel Accuracy
62
60
200
300
400
500
600
900
Figure 4: The sensitive analysis of
AutoNovel
UNO
40
40
14
14
16
16
18
18
20
-30
20
30
22
22
24
t
24
26
26
28
28
30
20
-10
10
42
44
46
46
48
48
23
R6
38
36
40
26
Prediction
Prediction
Figure 5: The confusion matrix novel classes for typical NCD methods.
Table The details analysis of typical NCD methods. Results on ImageNet100, for Rº 50, R" = 50.
Table 10: dataset.
The results UNO on the balance
Method Novel
Head
Medium
Tail
Dataset
Novel
Known
Autonovel
47.96
55.87
55.60
29.87
CIFARI00
74.55
65.40 76.96
UNO
43.08
44.93
49.50
32.13
ImageNet100
85.12
93.28
Ours
61.48
77.47
55.80
53.07
Analysis of NCD method
aragraphNCD methods on Read/Medium// Taii classes: Tab. 2 of the main manuscripts, we have presented the results of NCD mothods. To further analyze these methods, we have shown their performance on the Head, Medium, and Tail classes in the novel class in Tab. 9. Our proposed method shows an improvement of over 20% on both the Head and Tail classes, demonstrating its advantage.
Additionally, Autonovel performs worse on the Tail class due to the limited number of positive pair samples for lail classes. In contrast, UNO performs worse in the Head classes because the head classes are misclassified into Tail classes. This argument is supported by the confusion matrix shown in Fig. 5. Specifically, in the case of Autonovel, several tail classes are merged into a single class due poor representation. There are too many samples on the right side of the confusion matrix for UNO, which denotes that the head classes are being misclassified into tail classes.
18

---

Published in Transactions on Machine Learning Research (08/2023)
Distribution of herbariuml9 dataset
600
500
samples
400
300
100
100
200
300
400
500
600
700
Class Index
Figure 6: The distribution of herbarium dataset
NCD methods with class re-balancing techniques: We conclude that novel class discovery (NCD) for long-tailed data is challenging, and existing methods have not been able to solve this problem. As shown in Tab. 2 of the main paper and Tab. the novel class accuracy decreased by almost 30% on both CIFARI00 and ImageNet100 datasets.
To improve the performance NCD in long- tailed scenarios, we have combined NCD with long-tail methods (+LA, +cRT). We observe that the accuracy known and novel classes improves when the distribution estimation of novel classes more accurate. Specifically, poorly in estimating the distribution of novel classes due to the use of pairwise loss, which assigns similar features the same pseudo label, making it difficult to learn distinctive representations for tail classes in an imbalanced setting. This results in tail classos being mixed with head classes. When AutoNovel is combined with long-tail methods, the novel classes decrease while UNO improves. When R" = 100, the severe imbalance of novel classes makes learning novel classes more difficult, resulting in a worse estimated distribution. As a result, combining UNO with long-tail methods no longer has any effect.
CIFAR100, when R 50, AutoNovel performs
On ImageNet, the estimated distribution of novel classes is more accurate, and both AutoNovel and UNO have improved the accuracy of novel class. However, UNO's accuracy of known classes slightly decreased because novel classes and known classes are often confused. For Herbarium19, the actual distribution is difficult to predict, so the achievement of LA and cRT is limited.
In conclusion, due to the noisy estimated distribution, naively combining NCD and long-tail methods cannot effectively solve the long-tail tailed novel class discovery problem.
E More results on Herbarium19 dataset
Fig.6 presents the distribution of the Herbarium19 dataset. The dataset is composed of 683 classes, out of which 178 categories have less than 20 samples, which presents significant challenge when attempting to cluster novel classes. Tab.11 shows the average accuracy over both class and instance. Our proposed method outperforms the typical NCD methods by considerable margin in both metrics, demonstrating the effectiveness of our approach.
F More explanation about estimation the number of novel categories
In order better analyze the experiment of estimating the number of novel categorics under a higher K" we visualize the novel class confusion matrices of UNO and Ours for known and unknown K with R R" 50 in ImageNet 1000. The y-axis is groud-truth, and the x-axis is prediction.
19

---

Published in Transactions on Machine Learning Research (08/2023)
Table Long-tailed novel class discovery performance on Herbarium19, report average class and samples accuracy on test datasets. The top three lines is the accuracy average over classes. The bottom three lines is the accuracy average over samples.
Herbarium
Method
All
Novel
Known
Autonovel
31.58
9.96
59.30
UNO
47.47
34.50
60.58
Ous
49.21
36.93
61.63
Autonovel
40.83
14.15
65.98
UNO
50.20
31.40
67.51
Ours
55.66
41.15
69.33
UNO
UNO under unknown K,
10
10
18
0
28
10
14
10
42
46
Prediction
Prediction
Figure 7: The confusion matrix of UNO for known and unknown K
UNO utilizes the Sinkhorn algorithm to generate pseudo labels, which enforces an equal distribution for each class. This will result in splitting a head class into multiple subcategories, which may be mixed up with the medium and tail classes, as illustrated by the green-bordered boxes in the left side image in Figure Due to the misclassification of the head class, this will introduce noise that affects the quality of prediction for the medium and tail classes, as indicated by the red-bordered boxes in the left side image in Figure When T is larger than the true K", this problem can be alleviated because the head class is more likely to be assigned to categories that do not match the ground truth, shown by the purple-bordee bordered boxes in the right diagram. In this case, the noise in the pseudo labels for the medium class and tail class will be reduced, thus improving the accuracy of medium and tail classes, as shown in the the red-bordered boxes in the right side image in Figure! 7.
According the result analysis, we find that our method's novel classes accuracy will not be significantly affected when K" is greater than the true value. Our method learns a long-tailed distribution based on the training set data, dynamically adjusting the allocation ratio for novel classes. Compared to UNO, method can increase the weight of the head class, which can suppress the decomposition of head classes into multiple subcategories. As result, only a small number of categorios are allocated to categories that do not match the ground truth, as shown the purple-bordered boxes in the right side image in Figure 8.
G Without strong model on CIFAR and ImageNet
We conduct experiment CIFAR100 and ImageNet100 datasets using a non-strong model. We first perform unsupervised pre-training of the model on taildd dataset using MoCoHe et al. (2020) on known classes. Next, we conduct supervised training on the known classes data. Finally, we discover the novel classes by jointly training on the known and novel classes.
The results Tab.12 demonstrate that our method outperforms existing methods on the novel classes both CIFAR100 and ImageNet datasets, especially in the more challenging setting where R" 100. However, the CIFAR dataset, when equipped with the LA cRT technique, the baseline method surpasses
20

---

Published in Transactions on Machine Learning Research (08/2023)
Ours
under
Prediction
Prediction
Figure 8:
The confusion matrix of Ours for known and unknown K"
Table 12: The performance on CLEAR-100, ImageNet100 without strong model.
CIFAR100-50-60
R 50, R
50
R
50, R
100
R
50, R" = 50
R
50, R"
100
Method
All
Novel
Known
All
Novel
Known
All
Novel
Known
All
Novel
Known
AutoNovel
29.72
16.90
12.54
30.39
16.82
13.96 96
15.52
25.21
65.80
12.88
19.56
66.20
AutoNovel+LA
30.74
18.60
42.88
30.54
17.01
44.04
45.90
25.28
06.52
43.04
19.88
06.20
AutoNovvl++eR
30.72
18.38
43.06
29.35
16.20
42.50
27.20
42.78
21.20
64.36
UNO
33.80
27.04
40.56
33.05
24.88
59.12
UNO+LA
34.78
27.50
42.06
34.66
24.04
43.28
45.78
29.08
62.48
44,80
26.80
62.80
UNO+cRT
36.98
29.44
44.52
33.27
25.50
41.04
43.72
27.36
60.08
43.16
25.92
60.40
Ours
36.42
30.22
42.62
35.08
27.36
42.80
47.66
29.48
65.84
47.08
27.96
66.20
Ours+LA
37.51
30.72
44.30
35.84
27.50
44.18
48.90
29.28
68.52
48.04
27.80
68.28
method on the known classes, resulting in our method being slightly better or worse than existing methods in overall accuracy. While on TmageNeet100 datasets, our method surpasses the existing method by a sizeable margin. Furthermore, when applying the LA technique to our method, our results consistently outperform the existing methods terms of the 'All" and 'Novel' metrics.
Known data are balanced
We conduct the experiment when known classes are balanced on CIFAR100 and ImageNet100. The results Tab. 13 show we achieve consistent and signíficant improvement on novel classes. For CTFAR100, our method achieves large improvements in different R" settings, surpasses the previous SOTA method by 5.74% and 6.20% in novel accuracy. For ImageNet100, achieve significant improvement over the previous SOTA method by 13.68% and 14.48% in novel accuracy.
Table 13: Experiments balanced known classes
CIFAR100-50-60
ImageNet 100-50-50
R
1, I.
100
R
R.
R,
Method
All
Novel
Known
All
Novel
Known
All
Novel
Known
All
Novel
Known
Ausonovel
55.66
21.21
87.08
51.60
22.21
86.96
69.20
15.28
93.
66.60
39.60
93. 33.60
Autenovel + LA
56.20
21.80
87.70
55.11
22.46
87.76
69.00
11.72
93.28
66.51
39.18
93.60
AutaNovel cRT
57.11
27.78
86.44
56.01
25.88
86.41
69.71
45.74
93.68
67.35
41.23
93.48
UNO
27.16
68.68
93.32
UNO LA
59.87
32.92
86.82
58.07
28.90
87.24
68.82
44.28
93.36
65.00
41.60
94.40
UNO + eRT
60.35
34.38
86.32
37.74
28.46
87.02
69.74
45.88
93.60
67.66
40.88
94.44
Ours
63.19
10.12
86.26
60.55
35.10
$6.00
76.50
59.56
93.41
71.86
56.08
93.64
21


Reviews:
[{"review_id": "iIDJLk6R3d", "reviewer": "Reviewer_fTg4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Considering the novel class discovery under the long-tailed setting seems to be realistic and new.\n\nWeakness:\n1. The writing of the method section is unclear and confusing.\n - (5)(6) and (7)(8) describe two optimization problems. However, its not clear from the equations that (1) Are they minimization or maximization problems? Please use \"min\" or \"max\" instead of $\\mathcal{L}$ as the main objective. (2) What are the variables to be optimized? Please mark those variables under \"min\" or \"max\".\n- Why is the problem bi-level optimization? Can't the variables be jointly optimized?\n- The use of \"random variable\" and \"distribution\" is very messy: For example, (1) in section 4.3, \"Formally, we denote the cluster size distribution as $\\nu \\in \\mathbb{R}_{+}^{K^u}$\", I guess $\\nu$ here is a random variable, not distribution. (2) The authors then \"introduce an auxiliary variable $w$\", and again mention $\\nu$ is a prior distribution. Then how can the KL divergence be calculated between a random variable $w$ and a distribution $\\nu$\"? (3) Given the authors describe $w$ as an auxiliary variable $w$, the statement \"then we normalize $w_i$ ... to make it a valid probability distribution\" is contradictory. Is $w$ a random variable or distribution?\n- In C. Mini-Batch Buffer, \"min-batch\" appears twice. Is it typo or something else?\n- Section 4.5 is over-simplified with only textual description.\n\n2. The novelty and motivation of the proposed components are weak:\n- Section 4.1. The ETF seems to be a direct application of the original method proposed by Yang et al., 2022b\n- Section 4.3. It seems the problem formulation largely follows Asano et al., 2020. The only difference is the additional auxiliary variable. Moreover, the parametric form of $w$ is very heuristic.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper has the following contributions:\n1. A new setting for novel class discovery, where classes are long-tailed.\n2. A new adaptive self-labeling method, which assigns pseudo labels to novel classes adaptively. The method casts the pseudo label generation problem to an optimal transport problem.\n3. The authors conduct empirical experiments to demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please follow the weaknesses part and polish the method section in a more rigorous way. Also, please make it more clear which techniques are existing, which techniques are novel, and better motivate the novel parts.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y3ey6AMmJQ", "reviewer": "Reviewer_QARH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* the paper deals with a more realistic extension for NCD - the assumptions and datasets used for the task make it highly unrealistic in practice and more of a toy task; this paper takes a step towards a more realistic task to solve \n\n* The paper further partially studies the case of unknown number of unseen classes \n\n* the proposed Adaptive Self-labeling Algorithm contains technical novelty, at least in the NCD setting.\n\nWeaknesses:\n\n* The proposed baseline for estimating the number of novel categories is not adequately described. If I understood correctly it requires rerunning the algorithm for many values of K_u and keep the one that performs best based on Eq11. Is that correct? What is the computational cost of that? how do you sample possible values for K_u to evaluate? It seems infeasible in practice to me, but maybe I am missing something.\n\n* All experiments start from a very strong model, a DINO ViT trained on ImageNet1K, i.e. a dataset that contains a large number of classes including all \"unseen\" classes to discover for ImageNet100 and most of CIFAR100. Not only has the pretrained model seen those classes, which is generally fine cause it was trained without labels, however the model has however seen a much much higher number of images for all tail classes. So the premise now is that we first see many images per class, train for hundreds of epochs with a self-supervised loss and then retrain for a few epochs with LT labels. For the experiment to be fair, the pretrained model should not have seen the classes - a common protocol in LT papers is to start from a Places-pretrained model. This would be a much more fair setting I think\n\n* The \"new setup\" presented by the authors boils down to using LT datasets for evaluation. I would expect a more extensive and thorough SoTA reproduction and comparison. The authors only compare two 2 typical NCD methods. They cite more recent ones, eg Yang et al., 2022a, but do not include in the comparison. Adding more methods and maybe also some baselines would make the paper stronger.\n\n* The algorithm still is only evaluated on relatively small-sized dataset (CIFAR100, ImageNet100) that were artificially turned long-tail and a realistic, but still small/medium scaled dataset Herbarium19. No test on larger LT datasets like iNaturalist17/18 that have thousands of classes and are shown to be harder.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with an interesting and more realistic setup for novel class discovery (NCD), i.e. the case where discovery happens on an imbalanced dataset. It proposes a self labeling method that takes imbalance into account. They propose an algorithm for assigning pseudolabels to the unlabeled data by formulating it as an optimal transport problem.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* For the ImageNet and CIFAR datasets, new experiments should be run where the pretrained model hasnt explicitly seen all images from the truncated classes, eg starting from a Places pretrained model.\n\n* Experiments on all three datasets should be added for the cases where one further predicts the number of unknown classes, not just ImageNet100\n\n* Experiments on iNaturalist would make this paper's claims much stronger. Right now the only real LT dataset used is Herbarium, where the are generally small (less than 2% overall) versus simply forgetting about class imbalance and using UNO.\n\n* Would the proposed approach work if the known data were not LT? Known classes should not be expected to be LT as well - we have a large number of datasets that are balanced that would still be nice to use for LT discovery. It would be nice to have an experiment on the smaller datasets (ImageNet and CIFAR) where the known classes are balanced.\n\n\nOther notes and questions:\n- Missing refs on class discovery for long tail instance segmentation: Wang et al CVPR 2021: Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision\n- Were the hyperparameters tuned per dataset and per method? E.g. why is the UNO baseline worse after adapting it to LT for Herbarium? Were the hyperparameters properly tuned for that dataset?\n- \"Superivsed\" type in Sec 2\n- how much does using Eq9 offer? How much better is it than using a real-valued vector for w?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7iDZCBcAKF", "reviewer": "Reviewer_2okT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper proposes a realistic and challenging setting for novel class discovery, where the distribution of novel and known classes is long-tailed.\n2. The proposed method is effective and the details about the algorithm are provided.\n3. The experiments are convincing and the ablation study analysis the effect of each component. \n\nWeakness:\n1. Compared with the results under known $K_u$, the proposed method and UNO perform better when the $K_u$ is unknown. Please give more explanation about the experiment of estimation the number of novel categories;\n2. There are several confusing equations, such as \\mathbf{Y}^u\\mathbf{1}_{K_u} = \\mathbf{\\mu}  in equation (6), Y_u\\in \\mathbb{R}_+^{K_{u}\\times M}, and \\mathbf{1}_{K^u}\\in \\mathbb{R}_+^{K_{u}\\times 1};\n3. There are several typos, such as \"We typically generate pseudo labels in a min-batch mode\" in page 7. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "For the novel class discovery, this paper presents a new setting that the distribution of novel class and known class is long-tailed. Then they propose a iterative algorithm consisting of equiangular prototype, adaptive self-labeling to alleviate the biases result from imbalance in known classes and novel classes.  The results on three datasets demonstrate the proposed method outperforms AutoNovel and UNO under such a new setting. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some parts of the paper is not clear enough. Please see the weakness part. There also miss some detailed discussions of the experimental results.", "broader_impact_concerns": "There is no concern on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores novel class discovery in a long-tailed distribution scenario, which is more realistic than previous relevant research, proposing an iterative self-labeling algorithm to address the class-imbalance issue. The extensive experimental results support the proposed algorithm, and the authors provided informative responses to the reviewers, leading to acceptance by two reviewers. One reviewer expressed a concern about the novelty but will feel OK if the paper is accepted. The AE weighed the pros and cons and recommends acceptance with minor revision. The authors are mandated to make vital revision to include the technical clarifications as well as the supplemented experimental results in their responses. 
Decision: Accept (poster)